memory error in python while processing large file

Question

Input: large file around 12GB with sequence file, with ~ delimiter and I want to break after every 10th occurrence with new line.

I tried with

with open ("file.txt") as f:
    for line in f:
        x = line.count("~")
        y = line.split("~")
        s = ['Ç'.join(x) for x in [y[i:i + 10] for i in xrange(0, len(y), 10)]]
with open ("output.txt","w") as outfile:
    outfile.write("~\n".join(s))

While line.split('~') I'am getting memory error.

I tried with y = [line.split('~') for line in f] but no use same error. Please assist me how to handle this issue.

Yevgeniy Shchemelev · Accepted Answer · 2017-07-10 07:32:57Z

1

for line in f:

will try to load all file into your RAM

Use xreadline iterator to load file line by line:

for line in f.xreadlines():

answered Jul 10, 2017 at 7:32

Yevgeniy Shchemelev

3,6612 gold badges34 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

akhil Over a year ago

after changing to for line in f.xreadlines(): still am facing memory error for this line y = line.split("~").

Yevgeniy Shchemelev Over a year ago

Check the size of a line. If you don't have proper end of line symbols in your file then all file will be read into one line. In that case you have to read your file as binary using a buffer. See example in following question: stackoverflow.com/questions/1035340/…

akhil Over a year ago

i tried with binary as well but no use again same memory error

Yevgeniy Shchemelev Over a year ago

outfile.write("~\n".join(s)) will also load everything into the memory try to use: for item in s: outfile.write(item+'\n')

Yevgeniy Shchemelev Over a year ago

diveintopython.net/file_handling/file_objects.html could be useful for understanding of file opertions

|

Collectives™ on Stack Overflow

memory error in python while processing large file

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related