Python database WITHOUT using Django (for Heroku)

Question

To my surprise, I haven't found this question asked elsewhere. Short version, I'm writing an app that I plan to deploy to the cloud (probably using Heroku), which will do various web scraping and data collection. The reason it'll be in the cloud is so that I can have it be set to run on its own every day and pull the data to its database without my computer being on, as well as so the rest of the team can access the data.

I used to use AWS's SimpleDB and DynamoDB, but I found SDB's storage limitations to be to small and DDB's poor querying ability to be a problem, so I'm looking for a database system (SQL or NoSQL) that can store arbitrary-length values (and ideally arbitrary data structures) and that can be queried on any field.

I've found many database solutions for Heroku, such as ClearDB, but all of the information I've seen has shown how to set up Django to access the database. Since this is intended to be script and not a site, I'd really prefer not to dive into Django if I don't have to.

Is there any kind of database that I can hook up to in Heroku with Python without using Django?

CraigKerstiens · Accepted Answer · 2012-05-17 21:59:50Z

21

You can get a database provided from Heroku without requiring your app to use Django. To do so:

heroku addons:add heroku-postgresql:dev

If you need a larger more dedicated database, you can examine the plans at Heroku Postgres

Within your requirements.txt you'll want to add:

psycopg2

Then you can connect/interact with it similar to the following:

import psycopg2
import os
import urlparse

urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ['DATABASE_URL'])

conn = psycopg2.connect("dbname=%s user=%s password=%s host=%s " % (url.path[1:], url.username, url.password, url.hostname))
cur = conn.cursor()

query = "SELECT ...."
cur.execute(query)

answered May 17, 2012 at 21:59

CraigKerstiens

5,9891 gold badge27 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Conrad.Dean Over a year ago

thank you SO much! I've been trying to get a bare-bones postgres flask app running for a while, and all anyone wants to write about is sqlalchemy these days.

Kyle Carlson Over a year ago

I get a <type 'exceptions.KeyError'> at the line with DATABASE_URL whenever I try to run this. Any suggestions?

Jökull Over a year ago

If you are using Flask, Flask-SQLAlchemy will spin up a connection pool and give you a nice db.Model class to define declarative models. SQLAlchemy gives you many ways to interact with your database.

santiagobasulto · Accepted Answer · 2012-05-17 19:05:29Z

3

I'd use MongoDB. Heroku has support for it, so I think it will be really easy to start and scale out: https://addons.heroku.com/mongohq

About Python: MongoDB is a really easy database. The schema is flexible and fits really well with Python dictionaries. That's something really good.

You can use PyMongo

from pymongo import Connection
connection = Connection()

# Get your DB
db = connection.my_database

# Get your collection
cars = db.cars

# Create some objects
import datetime
car = {"brand": "Ford",
       "model": "Mustang",
       "date": datetime.datetime.utcnow()}

# Insert it
cars.insert(car)

Pretty simple, uh?

Hope it helps.

EDIT:

As Endophage mentioned, another good option for interfacing with Mongo is mongoengine. If you have lots of data to store, you should take a look at that.

edited May 17, 2012 at 19:05

answered May 17, 2012 at 18:53

santiagobasulto

11.8k13 gold badges70 silver badges90 bronze badges

5 Comments

Endophage Over a year ago

+1 for mongo but I'd also recommend using mongoengine which provides an ORM layer on top of pymongo and GridFS.

santiagobasulto Over a year ago

@Endophage that's true. As he wants simplicity, i recommended Pymongo. But, i should mentioned that. Thx!

jdotjdot Over a year ago

What's the benefit of mongoengine over pymongo? If anything, it just looks like mongoengine requires schemas to be written, whereas pymongo doesn't.

santiagobasulto Over a year ago

Yes, you're right. But for that same reason, it makes it simpler to handle complex schemas.

Liam Dawson Over a year ago

I find that for the current project I'm building, a document structure is ideal, but yet a schema saves me a few headaches and just helps in general.

chromy · Accepted Answer · 2012-05-17 18:44:43Z

3

I did this recently with Flask. (https://github.com/HexIce/flask-heroku-sqlalchemy).

There are a couple of gotchas:

1. If you don't use Django you may have to set up your database yourself by doing:

heroku addons:add shared-database

(Or whichever database you want to use, the others cost money.)

2. The database URL is stored in Heroku in the "DATABASE_URL" environment variable. In python you can get it by doing.

dburl = os.environ['DATABASE_URL']

What you do to connect to the database from there is up to you, one option is SQLAlchemy.

answered May 17, 2012 at 18:44

chromy

1422 silver badges8 bronze badges

2 Comments

chromy Over a year ago

No rep, can't edit but: - Heroku Database Page & SQLAlchemy in Flask

Will Over a year ago

Instead of shared-database, use heroku-postgresql:dev which is also free, and lets you connect from outside heroku, is postgres 9.1, etc: postgres.heroku.com/blog/past/2012/4/26/…

Will · Accepted Answer · 2012-05-17 18:37:05Z

0

Create a standalone Heroku Postgres database. http://postgres.heroku.com

answered May 17, 2012 at 18:37

Will

2,96119 silver badges12 bronze badges

1 Comment

jdotjdot Over a year ago

Okay, but the question remains there--how would I hook up to it with python? psycopg2?

Collectives™ on Stack Overflow

Python database WITHOUT using Django (for Heroku)

4 Answers 4

3 Comments

5 Comments

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

5 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related