4

My software runs a cronjob every 30 minutes, which pulls data from Google Analytics / Social networks and inserts the results into a Postgres DB.

The data looks like this:

url text NOT NULL,    
rangeStart timestamp NOT NULL,
rangeEnd timestamp NOT NULL,
createdAt timestamp DEFAULT now() NOT NULL,
...
(various integer columns)

Since one query returns 10 000+ items, it's obviously not a good idea to store this data in a single table. At this rate, the cronjob will generate about 480 000 records a day and about 14.5 million a month.

I think the solution would be using several tables, for example I could use a specific table to store data generated in a given month: stats_2015_09, stats_2015_10, stats_2015_11 etc.

I know Postgres supports table partitioning. However, I'm new to this concept, so I'm not sure what's the best way to do this. Do I need partitioning in this case, or should I just create these tables manually? Or maybe there is a better solution?

The data will be queried later in various ways, and those queries are expected to run fast.

EDIT:

If I end up with 12-14 tables, each storing 10-20 millions rows, Postgres should be still able to run select statements quickly, right? Inserts don't have to be super fast.

Improve this question
2
  • 1
    The official documentation on this subject is pithy and should be sufficient. Commented Sep 26, 2015 at 12:18
  • See also pg_partman, which is an easy-to-use extension providing a lot of the niceties the built-in plumbing lacks. Commented Sep 26, 2015 at 12:45

2 Answers 2

Reset to default
9

Partitioning is a good idea under various circumstances. Two that come to mind are:

  • Your queries have a WHERE clause that can be readily mapped onto one or a handful of partitions.
  • You want a speedy way to delete historical data (dropping a partition is faster than deleting records).

Without knowledge of the types of queries that you want to run, it is difficult to say if partitioning is a good idea.

I think I can say that splitting the data into different tables is a bad idea because it is a maintenance nightmare:

  • You can't have foreign key references into the table.
  • Queries spanning multiple tables are cumbersome, so simple questions are hard to answer.
  • Maintaining tables becomes a nightmare (adding/removing a column).
  • Permissions have to be carefully maintained, if you have users with different roles.

In any case, the place to start is with Postgres's documentation on partitioning, which is here. I should note that Postgres's implementation is a bit more awkward than in other databases, so you might want to review the documentation for MySQL or SQL Server to get an idea of what it is doing.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for the response. I can't really tell exactly which kinda queries we want, since the historical data will be queried by a different app. But I think the most important key will be the date range (from - to), so maybe it's a good idea to create partitions based on that.
2

Firstly, I would like to challenge the premise of your question:

Since one query returns 10 000+ items, it's obviously not a good idea to store this data in a single table.

As far as I know, there is no fundamental reason why the database would not cope fine with a single table of many millions of rows. At the extreme, if you created a table with no indexes, and simply appended rows to it, Postgres could simply carry on writing these rows to disk until you ran out of storage space. (There may be other limits internally, I'm not sure; but if so, they're big.)

The problems only come when you try to do something with that data, and the exact problems - and therefore exact solutions - depend on what you do.

If you want to regularly delete all rows which were inserted more than a fixed timescale ago, you could partition the data on the createdAt column. The DELETE would then become a very efficient DROP TABLE, and all INSERTs would be routed through a trigger to the "current" partition (or could even by-pass it if your import script was aware of the partition naming scheme). SELECTs, however, would probably not be able to specify a range of createAt values in their WHERE clause, and would thus need to query all partitions and combine the results. The more partitions you keep around at a time, the less efficient this would be.

Alternatively, you might examine the workload on the table and see that all queries either already do, or easily can, explicitly state a rangeStart value. In that case, you could partition on rangeStart, and the query planner would be able to eliminate all but one or a few partitions when planning each SELECT query. INSERTs would need to be routed through a trigger to the appropriate table, and maintenance operations (such as deleting old data that is no longer needed) would be much less efficient.

Or perhaps you know that once rangeEnd becomes "too old" you will no longer need the data, and can get both benefits: partition by rangeEnd, ensure all your SELECT queries explicitly mention rangeEnd, and drop partitions containing data you are no longer interested in.

To borrow Linus Torvald's terminology from git, the "plumbing" for partitioning is built into Postgres in the form of table inheritance, as documented here, but there is little in the way of "porcelain" other than examples in the manual. However, there is a very good extension called pg_partman which provides functions for managing partition sets based on either IDs or date ranges; it's well worth reading through the documentation to understand the different modes of operation. In my case, none quite matched, but forking that extension was significantly easier than writing everything from scratch.

Remember that partitioning does not come free, and if there is no obvious candidate for a column to partition by based on the kind of considerations above, you may actually be better off leaving the data in one table, and considering other optimisation strategies. For instance, partial indexes (CREATE INDEX ... WHERE) might be able to handle the most commonly queried subset of rows; perhaps combined with "covering indexes", where Postgres can return the query results directly from the index without reference to the main table structure ("index-only scans").

Improve this answer

3 Comments

Thank you for the detailed explanation, it's very useful! I already checked the docs, and will check pg_partman now. So far, createdAt seems to a good column to partition on. I looked into the query code and the most important queries can explicitly mention it (as a range).
About keeping the data in one table: that's an option. But even if that table is indexed properly, if there are 500 million rows there, SELECTs will be relatively slow, unless the index somehow guarantees that only a small part of the dataset has to be examined on each query. So splitting the data using partitioning or other technique, seems to be a better approach. Am I wrong about this?
@user2297996 "unless the index somehow guarantees that only a small part of the dataset has to be examined on each query" - that is pretty much exactly what an index is for, efficiently jumping to the right part of the dataset. I don't know the details of how btree indexes work, but my rule of thumb with databases is that if I don't know how it works, the DBMS is probably going to make better decisions out of the box than I would trying to optimise low-level structures by hand.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.