3

I am trying to execute a sql query from a file using sqlalchemy.

When I run the queries, I get a result saying that it affected x amount of rows, but when I check the DB it doesn't actually insert anything to the tables.

Here is my current code:

def import_to_db(df, table_name):
    df.to_sql(
        table_name, 
        con=engine, 
        schema='staging', 
        if_exists='replace', 
        index= False, 
        method= 'multi'
    )

    print('imported data to staging.{}'.format(table_name))

    with open('/home/kyle/projects/data_pipelines/ahc/sql/etl_{}.sql'.format(table_name)) as fp:
        etl = fp.read()

    result = engine.execute(etl)
    print('moved {} rows to public.{}'.format(result.rowcount, table_name))

When I run the .sql scripts manually, they work fine. I even tried making stored procedures but that didn't work either. Here is an example of one of the sql files im executing:

--Delete Id's in prod table that are in current staging table
DELETE
FROM public.table
WHERE key IN
    (SELECT key FROM staging.table);

--Insert new/old id's into prod table and do any cleaning
INSERT INTO 
public.table
SELECT columna, columnb, columnc
FROM staging.table;
5
  • Perhaps try cnxn = engine.raw_connection(); cnxn.autocommit = True; crsr = cnxn.cursor(); crsr.execute(etl) Commented Oct 17, 2019 at 19:01
  • nope. did not work. Commented Oct 17, 2019 at 19:05
  • Also related: stackoverflow.com/questions/51561894/… Commented Oct 18, 2019 at 5:53
  • 2
    TL;DR from the dupe target: your SQL file does not begin with something that SQLA recognizes as DDL / DML (instead it begins with a comment), so it does not autocommit. Commented Oct 18, 2019 at 6:02
  • Thanks that did it. If you want to type it up as answer ill mark it. Commented Oct 18, 2019 at 21:08

1 Answer 1

3

Found a solution, although I don't fully understand it.

I added BEGIN; at the top of my script, and COMMIT; at the bottom.

This works, but my row count now say -1 so it doesn't help me much for logging.

Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.