Reading csv file in python

Question

With the following data, using the code snippet, I am getting the following error. Can you please help me with this. I am a beginner in python. Data :

"Id","Title","Body","Tags"
"Id1","Tit,le1","Body1","Ta,gs1"
"Id","Title","Body","Ta,2gs"

Code:

#!/usr/bin/python 
import csv,sys
if len(sys.argv) <> 3:
print >>sys.stderr, 'Wrong number of arguments. This tool will print first n records from a comma separated CSV file.' 
print >>sys.stderr, 'Usage:' 
print >>sys.stderr, '       python', sys.argv[0], '<file> <number-of-lines>'
sys.exit(1)

fileName = sys.argv[1]
n = int(sys.argv[2])

i = 0 
out = csv.writer(sys.stdout, delimiter=',', quotechar='"', quoting=csv.QUOTE_NONNUMERIC)

ret = []


def read_csv(file_path, has_header = True):
    with open(file_path) as f:
        if has_header: f.readline()
        data = []
        for line in f:
            line = line.strip().split("\",\"")
            data.append([x for x in line])
    return data


ret = read_csv(fileName)
target = []
train = []
target = [x[2] for x in ret]
train = [x[1] for x in ret]

Error:

    target = [x[2] for x in ret]
IndexError: list index out of range

Sorry, for the mistake in the data. I have editted the question now. Thanks @PauloBu — novieq
– novieq, Commented Dec 18, 2013 at 14:10
@novieq Are there extra newlines (blank lines) after your data? When it tries to index the line [''] it gives you this error? Adding an if statement to check for that before appending your data would help? — MakeCents
– MakeCents, Commented Dec 18, 2013 at 14:38

Martijn Pieters · Accepted Answer · 2013-12-18 15:09:06Z

3

You are mixing file.readline() and using the file object as an iterable. Don't do that. Use next() instead.

You also should use the csv.reader() module to read your data, don't reinvent this wheel. The csv module can handle quoted CSV values with delimiters embedded in thevalues much better in any case:

import csv

def read_csv(file_path, has_header=True):
    with open(file_path, 'rb') as f:
        reader = csv.reader(f)
        if has_header: next(reader, None)
        return list(reader)

Last but not least, you can use zip() to transpose rows and columns:

ret = read_csv(fileName)
target, train = zip(*ret)[1:3]  # just the 2nd and 3rd columns

Here zip() will stop at the first row where there are not enough columns, at the very least avoiding the exception you see.

If there are columns missing in some of the rows, use itertools.izip_longest() instead (itertools.zip_longest() in Python 3):

from itertools import izip_longest

ret = read_csv(fileName)
target, train = izip_longest(*ret)[1:3]  # just the 2nd and 3rd columns

The default is to replace missing columns with None; if you need to use a different value, pass a fillvalue argument to izip_longest():

target, train = izip_longest(*ret, fillvalue=0)[1:3]  # just the 2nd and 3rd columns

edited Dec 18, 2013 at 15:09

answered Dec 18, 2013 at 14:13

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

novieq Over a year ago

Thanks @Martijin for this. I get the following error : target, train = zip(*ret)[1:2] ValueError: need more than 0 values to unpack

Martijn Pieters Over a year ago

@novieq: Your CSV file was empty in that case. With no columns to work with, zip() returned an empty list.

novieq Over a year ago

I used ret = read_csv(fileName) print(ret[0][2]) print(ret[1][2]) target, train = zip(*ret)[1:2] and I can see the output. So the csv is parsed correctly. Thanks @Martijn in advance.

Martijn Pieters Over a year ago

You need to slice with [1:3] to get two columns; the end value is not included.

Martijn Pieters Over a year ago

Are you still seeing any other problems?

|

Collectives™ on Stack Overflow

Reading csv file in python

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related