I am downloading a CSV file directly from a URL using requests.
How can I parse the resulting string with csv.DictReader?
Right now I have this:
r = requests.get(url)
reader_list = csv.DictReader(r.text)
print reader_list.fieldnames
for row in reader_list:
print row
But I just get ['r'] as the result of fieldnames, and then all kinds of weird things from print row.
From the documentation of csv, the first argument to csv.reader or csv.DictReader is csvfile -
csvfile can be any object which supports the iterator protocol and returns a string each time its
__next__()method is called — file objects and list objects are both suitable.
In your case when you give the string as the direct input for the csv.DictReader() , the __next__() call on that string only provides a single character, and hence that becomes the header, and then __next__() is continuously called to get each row.
Hence, you need to either provide an in-memory stream of strings using io.StringIO:
>>> import csv
>>> s = """a,b,c
... 1,2,3
... 4,5,6
... 7,8,9"""
>>> import io
>>> reader_list = csv.DictReader(io.StringIO(s))
>>> print(reader_list.fieldnames)
['a', 'b', 'c']
>>> for row in reader_list:
... print(row)
...
{'a': '1', 'b': '2', 'c': '3'}
{'a': '4', 'b': '5', 'c': '6'}
{'a': '7', 'b': '8', 'c': '9'}
or a list of lines using str.splitlines:
>>> reader_list = csv.DictReader(s.splitlines())
>>> print(reader_list.fieldnames)
['a', 'b', 'c']
>>> for row in reader_list:
... print(row)
...
{'a': '1', 'b': '2', 'c': '3'}
{'a': '4', 'b': '5', 'c': '6'}
{'a': '7', 'b': '8', 'c': '9'}
splitlines can break the record data line, since it can be inserted into a field by using quotes. If this is a problem, it should be stated. If the csv library can handle this, it should be also explained.