Migrate Postgres to MongoDB row-by-row

Question

I'm working on a project where I need to migrate from Postgres to MongoDB. The issue is, I cannot save the entire Postgres database to memory. Also, unlike approaches taken here or here, this is not a one-time migration, so I can't copy the entire contents of the Postgres database as text on disk every time data is transferred between the databases.

I was wondering if there was some way to "stream" data, row by row, from Postgres into MongoDB--realy, frompsycopg2 to pymongo.

edit: I suppose I should make the question more specific; I'm asking whether there exists a python library that can stream data between SQL/NoSQL databases, preferably one that works well with psycopg2 and pymongo.

edit2 For those who plan on down-voting, could you leave a comment explaining why? I thought it was a pretty legitimate question.

Have you considered using the MongoDB FDW to push data from PostgreSQL to MongoDB? — mu is too short
– mu is too short, Commented Dec 21, 2018 at 20:33
First time I've heard of it, thanks! Although I don't believe I have access to install new packages on the mongo server. I'll definitely look into that though. — David
– David, Commented Dec 21, 2018 at 20:54
You'd install it on the PostgreSQL server, hook up MongoDB collections as foreign tables, and then you could say things like insert into some_mongo_collection (...) select ... from postgresql_table where .... I haven't used it but worth a look. — mu is too short
– mu is too short, Commented Dec 21, 2018 at 20:56

David · Accepted Answer · 2018-12-22 07:38:28Z

3

To anyone else looking for a solution, I found a python library that might help. This Stack Overflow post asked a similar question to mine, but the author was looking for node.js or java libraries for migration from Postgres to Mongo. However, one of the respondents linked to this python library: pg2mongo: A hassle-free postresql to mongo migration framework.

answered Dec 22, 2018 at 7:38

David

66310 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Migrate Postgres to MongoDB row-by-row

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related