0

I need to analyze data from sql server. The process is memory heavy which requires me to use my university's computing resources as opposed my personal computer. Due to university policy, I cannot access the sqlserver directly and the only workaround is to mysqldump everything into a .sql file, and then copy it to the university's supercomputer.

The problem I have now is opening the .sql file in pandas. I referred to Read External SQL File into Pandas Dataframe

and got this code:

# Read the sql file and execute the query
with open('/filepath/mydata.sql', 'r') as query:
    connection = 'database'
    sql = pd.read_sql_query(query.read(),connection, encoding='unicode_escape')
sql

I am getting this error, and I am not sure how to fix it.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 6848294369: invalid start byte

I included encoding='unicode_escape' which seems to fix the error in read_csv, but I have not been successful when using read_sql_query.

Also, when I do get read_sql_query, how would I download data from a certain table? I use a sqlalchemy connection and this code: pd.read_sql(f"SELECT * FROM table", con=db_connection, chunksize = 100) or pd.read_sql_table(f"table", con=db_connection, chunksize = 100) to achieve that, but how would I do that using a .sql file?

Thank you for the help

3
  • 1
    So you are using mysqldump to create the mydata.sql file, uploading that to the school computer, and then trying to run read_sql_query() to load the data into a remote MySQL instance? Is that correct? Also please specify whether you are running mysqldump on a Windows machine or a non-Windows machine. Commented Jun 6, 2024 at 21:41
  • @GordThompson That is correct. I ran this terminal command on Linux Mint:mysqldump --single-transaction --host=hostname--user=username --password=pwd database > filepath/mydata.sql. I secure copied that to the school's computer which use Rocky Linux. Commented Jun 7, 2024 at 2:06
  • Okay, just so we're clear: That .sql file you created cannot be queried like a database. It just contains a bunch of CREATE TABLE and INSERT statements that can be used to populate a database. You could then query the database that you populated on the school computer, just like you query a database on your personal computer. Commented Jun 7, 2024 at 16:21

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.