1

I am writing a web app in Flask, and I am using pandas to retrieve data from a MySQL DB. The following used to work, using what I read from another post:

db = SQLAlchemy()
app = Flask(__name__)
app.config.from_object(config['default'])
db.init_app(app)
conn = db.engine.connect().connection

Then in the view function (in the same .py file):

@app.route('/report', methods=['GET', 'POST'])
def report():
    form = ReportForm()
...
    sql = '''SELECT * FROM availability ...'''
    df = psql.read_sql(sql, conn)
...

    return render_template('report.html', form=form, df=df)

This webpage displays a table based on the date interval chosen (from the form). I added inline editing of this table on the webpage via x-editable, and I see that the changes have been posted to the database, but if I submit the form again the table only loads up the old data before the change was made. I only see the changes after I reopen the webpage, which is very strange.

Moving the 'conn' line right before read_sql works:

sql = '''SELECT * FROM availability ...'''
conn = db.engine.connect().connection
df = psql.read_sql(sql, conn)

But this is not nice as I have to repeat this for every query I make (and there are a few for each view). Is there a way where I just declare this connection object once at the beginning and get it over with? SQLAlchemy's ORM works, but I prefer writing raw SQL in this case.

Edit:

joris' suggestion worked for one query, but now I ran into another error:

File "C:\Users\KF\flask-test\hello.py", line 107, in report
df = pd.read_sql_query(sql.format(vd='20140727', sd=sd, ed=ed), db.engine)
File "C:\Anaconda\envs\lightson\lib\site-packages\pandas\io\sql.py", line 363, in read_sql_query
parse_dates=parse_dates)
File "C:\Anaconda\envs\lightson\lib\site-packages\pandas\io\sql.py", line 823, in read_sql
result = self.execute(*args)
File "C:\Anaconda\envs\lightson\lib\site-packages\pandas\io\sql.py", line 810, in execute
return self.engine.execute(*args, **kwargs)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 1752, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 721, in execute
return self._execute_text(object, multiparams, params)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 870, in _execute_text
statement, parameters
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 958, in _execute_context
context)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 1163, in _handle_dbapi_exception
util.reraise(*exc_info)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\base.py", line 951, in _execute_context
context)
File "C:\Anaconda\envs\lightson\lib\site-packages\sqlalchemy\engine\default.py", line 436, in do_execute
cursor.execute(statement, parameters)
File "c:\users\kf\appdata\local\temp\easy_install-_444w8\MySQL_python-1.2.5-py2.7-win-amd64.egg.tmp\MySQLdb\cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not enough arguments for format string

But the raw sql string is fine - I ran the query myself and is valid. Actual SQL:

sql = '''SELECT * FROM availability WHERE view_date = str_to_date('{vd}', '%Y%m%d') and book_date >= str_to_date('{sd}','%Y%m%d') and book_date <= str_to_date('{ed}', '%Y%m%d')'''

Here 'sd' and 'ed' are strings that look like '20140801'. Looks like I have to escape those '%'s for MySQL but I can't seem to find a way to do it.

Edit:

Got it to work. Add another '%' in front of '%Y','%m' and '%d' to escape the percent signs. Basically, as joris said, use pd.read_sql_query and you should be fine.

9
  • What version of pandas are you using? Starting from 0.14, you should just give the engine instead of a connection to read_sql. Does changing that solve the problem? Commented Aug 7, 2014 at 14:51
  • Using df = psql.read_sql(sql, db.engine) gives me this error: OperationalError: (OperationalError) (1017, "Can't find file: '... Commented Aug 7, 2014 at 15:04
  • Which version of pandas? Commented Aug 7, 2014 at 15:10
  • 0.14.1, so that shouldn't be a problem. not many examples of using read_sql floating around though... Commented Aug 7, 2014 at 15:15
  • Can you post the full error? Also, you don't need to import the io.sql module, it is now available as a top level funtion. And if you do a query, better to use the pd.read_sql_query() function (read_sql tries to delegate to that function if it gets a query, so using it directly is a step less). Commented Aug 7, 2014 at 15:28

1 Answer 1

5

Figured it out. For those who might need this in the future: 1.Add another '%' in front of '%Y','%m' and '%d' to escape the percent signs. 2.Like joris said, use pd.read_sql_query instead of read_sql.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.