import CSV file into postgres using django

Question

I am trying to import csv file into postgres DB using Django .

I try following function:

import os
from django.db import models
import psycopg2
from postgres_copy import CopyMapping

host = 'localhost'
port = '5432'
dbname = 'sellerhub'
username = 'postgres'
password = 'postgres'


class Reports:
    def __init__(self):
        global host, port, dbname, username, password
        try:
            self.db_conn = psycopg2.connect("host=%s port=%s dbname=%s user=%s password=%s" %(host, port, dbname, username, password))
        except psycopg2.OperationalError:
            print "Database Not Found Or the Credentials are wrong."
        self.cur = self.db_conn.cursor()
def saveUploadedInventory(self, inventory_file):
        #print "Inventory File"
        with open('uploaded_inventory_sheet.csv','wb+') as destination:
            for chunk in inventory_file.chunks():
                destination.write(chunk)
        #print "Inventory Saved."
        copy_sql = """copy fk_invent_temp from stdin WITH CSV HEADER DELIMITER as ',' """
        #print "query created"
        with open('uploaded_inventory_sheet.csv','r') as pmt_file:
            self.cur.copy_expert(sql=copy_sql, file=pmt_file)
        #print "file uploades"
        os.system('rm uploaded_inventory_sheet.csv')
        #print "removes file"

and setting.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'sellerhub',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': 'localhost',
        'PORT': '5432',
        }
    }

this function is executed completly with no error,

but there is no data in fk_invent table .

If i directly import that file using PGAdmin3 UI that is uploaded successfully . Pls any body can tell what i am doing wrong ?

There's too much missing information. What class is this a method of? What is self.cur and self.cur.copy_expert? — Daniel Roseman
– Daniel Roseman, Commented Nov 9, 2015 at 12:53
cur is the Connection from database and copy expert is for copying database — Shubham Batra
– Shubham Batra, Commented Nov 9, 2015 at 13:01
But without seeing the code for copy_expert, how can we know where it is going wrong? — Daniel Roseman
– Daniel Roseman, Commented Nov 9, 2015 at 13:10
Does your Postgres user (specified in settings.py) have superuser rights? Must be superuser to COPY to or from a file... — AtlasStrategic
– AtlasStrategic, Commented Nov 9, 2015 at 13:11

Shubham Batra · Accepted Answer · 2015-11-27 06:02:22Z

I get temporary solution by inserting line by line:

with open('uploaded_inventory_sheet.csv','wb+') as destination:
            for chunk in inventory_file.chunks():
                destination.write(chunk)
        print "Inventory Saved."
        reader = csv.reader(open('uploaded_inventory_sheet.csv','rb'))
        count = 0 
        for row in reader:
            if count == 0:
                count=1
                continue
            # print row[0],"\n"
            count = count +1
            try:
                self.cur.execute("""INSERT into  fk_payment_temp values(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",(row[0],row[1],row[2],row[3],row[4],row[5],row[6],row[7],row[8],row[9],row[10],row[11],row[12],row[13],row[14],row[15],row[16],row[17],row[18],row[19],row[20],row[21],row[22],row[23],row[24],row[25],row[26],row[27],row[28],row[29],row[30],row[31],row[32],row[33],row[34],row[35],row[36],row[37],row[38],row[39],row[40],row[41],row[42],row[43],row[44]))
                print "INSERT into  fk_payment_temp values('",row[0],"','",row[1],"''",row[2],"','",row[3],"''",row[4],"','",row[5],"''",row[6],"','",row[7],"''",row[8],"','",row[9],"''",row[10],"','",row[11],"''",row[12],"','",row[13],"''",row[14],"','",row[15],"''",row[16],"','",row[17],"''",row[18],"','",row[19],"''",row[20],"','",row[21],"''",row[22],"','",row[23],"''",row[24],"','",row[25],"''",row[26],"','",row[27],"''",row[28],"','",row[29],"''",row[30],"','",row[31],"''",row[32],"','",row[33],"''",row[34],"','",row[35],"''",row[36],"','",row[37],"''",row[38],"','",row[39],"''",row[40],"','",row[41],"''",row[42],"','",row[43],"''",row[44],"','",row[45],"')"
            except:
                pass
        print count
        self.cur.callproc("flip_payment_test")
        self.cur
        self.db_conn.commit()

        print "file uploades"
        os.system('rm uploaded_inventory_sheet.csv')
        print "removes file"

Nhor · Accepted Answer · 2015-11-09 12:49:15Z

0

The simplest and cleanest solution is to import a json fixture file with Django manage.py command. So I'd recommend you converting your CSV file to json and load it to database with manage.py loaddata command:

python manage.py loaddata yourfile.json

You can find CSV to json converters all over the Internet.

answered Nov 9, 2015 at 12:49

Nhor

3,9806 gold badges31 silver badges41 bronze badges

1 Comment

Shubham Batra Over a year ago

csv file is selected by user that is not specific

Collectives™ on Stack Overflow

import CSV file into postgres using django

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related