How to read csv data into django models?

Question

I keep getting this error message after trying to import csv file into Django models.

TICKERS has two columns: column[0] = name, column[1] = ticker

populate_symbol.py

def populate():
   with open(TICKERS, 'rU') as csvfile:
    file = csv.reader(csvfile, delimiter=',')
    for row in file:
        add_ticker(str(row[0]), str(row[1]))

def add_ticker(name, ticker):
  c = Symbol.objects.get_or_create(name=name, ticker=ticker)
  return c

Error message:

(result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8e in position 555: invalid start byte

Is there a way to flag csv reader to read all kind of data (utf-8 or unicode)?

P.S: Python 3.4.3, Django 1.7

Ryan Serra · Accepted Answer · 2015-05-19 19:54:26Z

1

The official documentation for Python 2.7 (assumption) notes the following..

Note: This version of the csv module doesn’t support Unicode input. Also, there are currently some issues regarding ASCII NUL characters. Accordingly, all input should be UTF-8 or printable ASCII to be safe; see the examples in section Examples.

Python 2.7 CSV Examples

The last two examples on the page should give you what you need.

** Edit

As per your edit...

Python 3.4 CSV Examples

with open('some.csv', newline='', encoding='utf-8') as f:

If you are not aware of the encoding of the data you're trying to import, you're going to run into issues. There are ways to determine the type of encoding but as far as I'm aware, that is beyond the scope of the CSV module.

edited May 19, 2015 at 19:54

answered May 19, 2015 at 19:37

Ryan Serra

1442 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Bun · Accepted Answer · 2015-05-19 20:19:09Z

1

This line seem to work for me:

with open(TICKERS, encoding='utf-8', errors='ignore') as f:

answered May 19, 2015 at 20:19

Bun

3,1273 gold badges21 silver badges30 bronze badges

Collectives™ on Stack Overflow

How to read csv data into django models?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related