Collecting Relational Data and Adding to a Database Periodically with Python

Question

I have a project that :

fetches data from active directory
fetches data from different services based on active directory data
aggregates data
about 50000 row have to be added to database in every 15 min

I'm using Postgresql as database and django as ORM tool. But I'm not sure that django is the right tools for such projects. I have to drop and add 50000 rows data and I'm worry about performance. Is there another way to do such process?

Tometzky · Accepted Answer · 2017-07-30 22:06:44Z

1

50k rows/15m is nothing to worry about.

But I'd make sure to use bulk_create to avoid 50k of round trips to the database, which might be a problem depending on your database networking setup.

answered Jul 30, 2017 at 22:06

Tometzky

24.2k5 gold badges64 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Phil Krylov · Accepted Answer · 2017-07-30 20:53:19Z

0

For sure there are other ways, if that's what you're asking. But Django ORM is quite flexible overall, and if you write your queries carefully there will be no significant overhead. 50000 rows in 15 minutes is not really big enough. I am using Django ORM with PostgreSQL to process millions of records a day.

answered Jul 30, 2017 at 20:53

Phil Krylov

5404 silver badges8 bronze badges

2 Comments

jojo Over a year ago

could you suggest another way please? Thank you

Phil Krylov Over a year ago

You can use SQLAlchemy, or use another ORM, or use psycopg2 directly, which allows you to do low-level COPY functionality.

Jose Cherian · Accepted Answer · 2017-07-30 20:17:35Z

0

You can write a custom Django management command for this purpose, Then call it like

python manage.py collectdata

Here is the documentation link

answered Jul 30, 2017 at 20:17

Jose Cherian

7,7773 gold badges41 silver badges40 bronze badges

2 Comments

jojo Over a year ago

Is Django the right tool for such projects? I have to drop and add 50000 rows data and I'm worry about performance.

Jose Cherian Over a year ago

@jojo, Django is just a framework. Dropping and adding rows happens at database level. Postgresql is definitely capable of doing that.

Collectives™ on Stack Overflow

Collecting Relational Data and Adding to a Database Periodically with Python

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related