pandas read_csv() and python iterator as input

Question

I want to use pandas read_csv() func where the input is a python iterator, where each next() will bring to the next line of my text file. What would you suggest me to do? I want best performance.
As I understood, StringIO works in that case, but i would rather to not use that.

BTW, after that I'm using as_matrix() function in order to create a numpy array.
Doing so because it's much faster than np.loadtxt() func which is horribly slow :(

@MaxNoe because i can get access to the content of the txt file only by a generator — member555
– member555, Commented Nov 30, 2015 at 11:44
Just because i'm curious: Where does this strange limitation come from? — MaxNoe
– MaxNoe, Commented Nov 30, 2015 at 19:24
@MaxNoe I'm working with Apache Spark and my big text file is distributed across the nodes in my cluster. I want to do compute some function on each part of the text file at each node. The way I get access to that part is only by a python's iterator. — member555
– member555, Commented Nov 30, 2015 at 22:32

ojdo · Accepted Answer · 2022-07-07 16:55:56Z

2

You should use :

 from io import StringIO
 pd.read_csv(StringIO("\n".join(iter)))

where iter is your iterator / generator variable.
This is still faster than using np.loadtxt(iter)

edited Jul 7, 2022 at 16:55

ojdo

9,1258 gold badges43 silver badges66 bronze badges

answered Nov 29, 2015 at 19:18

member555

8071 gold badge15 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

pandas read_csv() and python iterator as input

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related