Reading until specific line of CSV in Python

Question

I am opening a big CSV file and during inserting the data into the array I got MemoryError. So how can i read specific lines of that CSV file (for example from line 1 till 10000)

here is the code:

datafile=open('test.csv','r')
datareader=csv.reader(datafile,delimiter=';')

for row in datareader:
    MyArray.append(row)

Why are you putting the lines into an array (list? np.array?), rather than just iterating over them? What are you trying to achieve? — jonrsharpe
– jonrsharpe, Commented Sep 23, 2014 at 9:30
The answer by Tim works well, it is more that 1GB, around 9 millions of line. I need the content of data in line (it is network packet) — pafpaf
– pafpaf, Commented Sep 23, 2014 at 9:38

Tim Pietzcker · Accepted Answer · 2014-09-23 09:49:00Z

2

I'd use islice instead of enumerate:

from itertools import islice

# First 10000
MyArray.extend(islice(datareader, 10000))

# Or, specify start/stop ranges (10000-20000 (non inclusive))
MyArray.extend(islice(datareader, 10000, 20000))

# Or read in chunks of 10k
for chunk in iter(lambda: list(islice(datareader, 10000)), []):
    # do something with 10k rows

edited Sep 23, 2014 at 9:49

Tim Pietzcker

337k59 gold badges520 silver badges572 bronze badges

answered Sep 23, 2014 at 9:41

Jon Clements

143k34 gold badges254 silver badges288 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Tim Pietzcker Over a year ago

Oh, much nicer. I've never had use for islice before, but this is certainly a perfect case. @user3636424, you might want to accept this answer instead.

Jon Clements Over a year ago

@Tim I'm not fussed about accept/reps - just happy it's useful for someone :)

Tim Pietzcker Over a year ago

@JonClements: Me neither, but this is the better answer and should be on top.

Jon Clements Over a year ago

@Tim well - it appears the OP took your advice... thanks for the edit.

Tim Pietzcker · Accepted Answer · 2014-09-23 09:37:46Z

2

Use enumerate():

for i, row in enumerate(datareader):
    MyArray.append(row)
    if i == 10000:
        break

or, for any range:

start = 1000
stop = 2000
for i, row in enumerate(datareader):
    if i < start: 
        continue     # skip this row
    elif i > stop:
        break        # abort the loop
    else:            # ("else" is not strictly necessary here, but more explicit)
        MyArray.append(row)

edited Sep 23, 2014 at 9:37

answered Sep 23, 2014 at 9:30

Tim Pietzcker

337k59 gold badges520 silver badges572 bronze badges

Collectives™ on Stack Overflow

Reading until specific line of CSV in Python

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related