I am trying to execute a sql query from a file using sqlalchemy.
When I run the queries, I get a result saying that it affected x amount of rows, but when I check the DB it doesn't actually insert anything to the tables.
Here is my current code:
def import_to_db(df, table_name):
df.to_sql(
table_name,
con=engine,
schema='staging',
if_exists='replace',
index= False,
method= 'multi'
)
print('imported data to staging.{}'.format(table_name))
with open('/home/kyle/projects/data_pipelines/ahc/sql/etl_{}.sql'.format(table_name)) as fp:
etl = fp.read()
result = engine.execute(etl)
print('moved {} rows to public.{}'.format(result.rowcount, table_name))
When I run the .sql scripts manually, they work fine. I even tried making stored procedures but that didn't work either. Here is an example of one of the sql files im executing:
--Delete Id's in prod table that are in current staging table
DELETE
FROM public.table
WHERE key IN
(SELECT key FROM staging.table);
--Insert new/old id's into prod table and do any cleaning
INSERT INTO
public.table
SELECT columna, columnb, columnc
FROM staging.table;
cnxn = engine.raw_connection(); cnxn.autocommit = True; crsr = cnxn.cursor(); crsr.execute(etl)