Error reading and writing csv files into python

Question

System info:
Python 2.7.2
MAC OSX 10.7.2

Problem (+background):
I have a large '.csv' file (~1 gig) which needs some minor editing. Every value in the 5th column needs to be 5 characters long (some are 4 characters long, and need a '0' placed in front of them). The code (shown below) reports no errors when run, but stops writing with approximately 100 lines in the file left (thereby losing some crucial data!). Anyone know why this is happening?

I've re-created the 'read_file.csv' and inspected it, but I don't see anything out of place. The code always aborts in the same location, but I don't understand why?

import csv

path = '/Volumes/.../'

r = csv.reader(open(path + 'read_file.csv','rU'))
f =  open(path + 'write_file.csv', 'wb')

writer = csv.writer(f)

for line in r:

    if len(line[5]) == 4:
        line[5] = '0' + line[5]

    writer.writerow((line[0],line[1],line[2],line[3],line[4],line[5],line[6],line[7]))

why opening file in binary mode?

Andrey Sboev
– Andrey Sboev

2011-11-02 22:18:58 +00:00
Commented Nov 2, 2011 at 22:18 — Andrey Sboev
– Andrey Sboev, Commented Nov 2, 2011 at 22:18
Can you give an example of what the file looks like?

chown
– chown

2011-11-02 22:19:24 +00:00
Commented Nov 2, 2011 at 22:19 — chown
– chown, Commented Nov 2, 2011 at 22:19

Dave · Accepted Answer · 2011-11-03 01:26:25Z

1

Either close the output file after writing it, or write the output in a with context which will always close the file even if an error occurs:

with open('path + 'write_file.csv', 'wb') as f:
    writer = csv.writer(f)
    for line in r:
        ...

answered Nov 3, 2011 at 1:26

Dave

3,9672 gold badges36 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

retracile · Accepted Answer · 2011-11-03 01:24:17Z

0

Things to check:

Are you examining this after your code has exited so you know the file has been .close() or .flush()ed?
Is it possible you have something odd in your data on that line that makes it think the rest of the file is data in a field?
You're only saving a set number of columns of your line; you might try writer.writerow(line) instead...

answered Nov 3, 2011 at 1:24

retracile

12.4k4 gold badges38 silver badges42 bronze badges

Comments

Austin Marshall · Accepted Answer · 2011-11-03 04:05:10Z

0

Ensure that the file is properly closed, with makes it easy.

with open('test.csv', 'rU') as inp:
    csvin=csv.reader(inp)
    with open('output.csv', 'wb') as outp:
        csvout=csv.writer(outp)
        for line in csvin:
            csvout.writerow(line[:4] + [line[4].rjust(5, '0')] + line[5:])

answered Nov 3, 2011 at 4:05

Austin Marshall

3,10519 silver badges14 bronze badges

Collectives™ on Stack Overflow

Error reading and writing csv files into python

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related