Skip to content

Commit 7300a69

Browse files
tvondratvondra
committed
Add support for multivariate MCV lists
Introduce a third extended statistic type, supported by the CREATE STATISTICS command - MCV lists, a generalization of the statistic already built and used for individual columns. Compared to the already supported types (n-distinct coefficients and functional dependencies), MCV lists are more complex, include column values and allow estimation of much wider range of common clauses (equality and inequality conditions, IS NULL, IS NOT NULL etc.). Similarly to the other types, a new pseudo-type (pg_mcv_list) is used. Author: Tomas Vondra Reviewed-by: Dean Rasheed, David Rowley, Mark Dilger, Alvaro Herrera Discussion: https://postgr.es/m/dfdac334-9cf2-2597-fb27-f0fb3753f435@2ndquadrant.com
1 parent 333ed24 commit 7300a69

32 files changed

+3597
-134
lines changed

‎doc/src/sgml/catalogs.sgml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6562,7 +6562,8 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
65626562
An array containing codes for the enabled statistic kinds;
65636563
valid values are:
65646564
<literal>d</literal> for n-distinct statistics,
6565-
<literal>f</literal> for functional dependency statistics
6565+
<literal>f</literal> for functional dependency statistics, and
6566+
<literal>m</literal> for most common values (MCV) list statistics
65666567
</entry>
65676568
</row>
65686569

@@ -6585,6 +6586,16 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
65856586
</entry>
65866587
</row>
65876588

6589+
<row>
6590+
<entry><structfield>stxmcv</structfield></entry>
6591+
<entry><type>pg_mcv_list</type></entry>
6592+
<entry></entry>
6593+
<entry>
6594+
MCV (most-common values) list statistics, serialized as
6595+
<structname>pg_mcv_list</structname> type.
6596+
</entry>
6597+
</row>
6598+
65886599
</tbody>
65896600
</tgroup>
65906601
</table>

‎doc/src/sgml/func.sgml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22174,4 +22174,86 @@ CREATE EVENT TRIGGER test_table_rewrite_oid
2217422174
</sect2>
2217522175
</sect1>
2217622176

22177+
<sect1 id="functions-statistics">
22178+
<title>Statistics Information Functions</title>
22179+
22180+
<indexterm zone="functions-statistics">
22181+
<primary>function</primary>
22182+
<secondary>statistics</secondary>
22183+
</indexterm>
22184+
22185+
<para>
22186+
<productname>PostgreSQL</productname> provides a function to inspect complex
22187+
statistics defined using the <command>CREATE STATISTICS</command> command.
22188+
</para>
22189+
22190+
<sect2 id="functions-statistics-mcv">
22191+
<title>Inspecting MCV lists</title>
22192+
22193+
<indexterm>
22194+
<primary>pg_mcv_list_items</primary>
22195+
<secondary>pg_mcv_list</secondary>
22196+
</indexterm>
22197+
22198+
<para>
22199+
<function>pg_mcv_list_items</function> returns a list of all items
22200+
stored in a multi-column <acronym>MCV</acronym> list, and returns the
22201+
following columns:
22202+
22203+
<informaltable>
22204+
<tgroup cols="3">
22205+
<thead>
22206+
<row>
22207+
<entry>Name</entry>
22208+
<entry>Type</entry>
22209+
<entry>Description</entry>
22210+
</row>
22211+
</thead>
22212+
22213+
<tbody>
22214+
<row>
22215+
<entry><literal>index</literal></entry>
22216+
<entry><type>int</type></entry>
22217+
<entry>index of the item in the <acronym>MCV</acronym> list</entry>
22218+
</row>
22219+
<row>
22220+
<entry><literal>values</literal></entry>
22221+
<entry><type>text[]</type></entry>
22222+
<entry>values stored in the MCV item</entry>
22223+
</row>
22224+
<row>
22225+
<entry><literal>nulls</literal></entry>
22226+
<entry><type>boolean[]</type></entry>
22227+
<entry>flags identifying <literal>NULL</literal> values</entry>
22228+
</row>
22229+
<row>
22230+
<entry><literal>frequency</literal></entry>
22231+
<entry><type>double precision</type></entry>
22232+
<entry>frequency of this <acronym>MCV</acronym> item</entry>
22233+
</row>
22234+
<row>
22235+
<entry><literal>base_frequency</literal></entry>
22236+
<entry><type>double precision</type></entry>
22237+
<entry>base frequency of this <acronym>MCV</acronym> item</entry>
22238+
</row>
22239+
</tbody>
22240+
</tgroup>
22241+
</informaltable>
22242+
</para>
22243+
22244+
<para>
22245+
The <function>pg_mcv_list_items</function> function can be used like this:
22246+
22247+
<programlisting>
22248+
SELECT m.* FROM pg_statistic_ext,
22249+
pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts';
22250+
</programlisting>
22251+
22252+
Values of the <type>pg_mcv_list</type> can be obtained only from the
22253+
<literal>pg_statistic_ext.stxmcv</literal> column.
22254+
</para>
22255+
</sect2>
22256+
22257+
</sect1>
22258+
2217722259
</chapter>

‎doc/src/sgml/perform.sgml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1285,6 +1285,72 @@ nd | {"1, 2": 33178, "1, 5": 33178, "2, 5": 27435, "1, 2, 5": 33178}
12851285
plans. Otherwise, the <command>ANALYZE</command> cycles are just wasted.
12861286
</para>
12871287
</sect3>
1288+
1289+
<sect3>
1290+
<title>Multivariate MCV lists</title>
1291+
1292+
<para>
1293+
Another type of statistics stored for each column are most-common value
1294+
lists. This allows very accurate estimates for individual columns, but
1295+
may result in significant misestimates for queries with conditions on
1296+
multiple columns.
1297+
</para>
1298+
1299+
<para>
1300+
To improve such estimates, <command>ANALYZE</command> can collect MCV
1301+
lists on combinations of columns. Similarly to functional dependencies
1302+
and n-distinct coefficients, it's impractical to do this for every
1303+
possible column grouping. Even more so in this case, as the MCV list
1304+
(unlike functional dependencies and n-distinct coefficients) does store
1305+
the common column values. So data is collected only for those groups
1306+
of columns appearing together in a statistics object defined with the
1307+
<literal>mcv</literal> option.
1308+
</para>
1309+
1310+
<para>
1311+
Continuing the previous example, the MCV list for a table of ZIP codes
1312+
might look like the following (unlike for simpler types of statistics,
1313+
a function is required for inspection of MCV contents):
1314+
1315+
<programlisting>
1316+
CREATE STATISTICS stts3 (mcv) ON state, city FROM zipcodes;
1317+
1318+
ANALYZE zipcodes;
1319+
1320+
SELECT m.* FROM pg_statistic_ext,
1321+
pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts3';
1322+
1323+
index | values | nulls | frequency | base_frequency
1324+
-------+------------------------+-------+-----------+----------------
1325+
0 | {Washington, DC} | {f,f} | 0.003467 | 2.7e-05
1326+
1 | {Apo, AE} | {f,f} | 0.003067 | 1.9e-05
1327+
2 | {Houston, TX} | {f,f} | 0.002167 | 0.000133
1328+
3 | {El Paso, TX} | {f,f} | 0.002 | 0.000113
1329+
4 | {New York, NY} | {f,f} | 0.001967 | 0.000114
1330+
5 | {Atlanta, GA} | {f,f} | 0.001633 | 3.3e-05
1331+
6 | {Sacramento, CA} | {f,f} | 0.001433 | 7.8e-05
1332+
7 | {Miami, FL} | {f,f} | 0.0014 | 6e-05
1333+
8 | {Dallas, TX} | {f,f} | 0.001367 | 8.8e-05
1334+
9 | {Chicago, IL} | {f,f} | 0.001333 | 5.1e-05
1335+
...
1336+
(99 rows)
1337+
</programlisting>
1338+
This indicates that the most common combination of city and state is
1339+
Washington in DC, with actual frequency (in the sample) about 0.35%.
1340+
The base frequency of the combination (as computed from the simple
1341+
per-column frequencies) is only 0.0027%, resulting in two orders of
1342+
magnitude under-estimates.
1343+
</para>
1344+
1345+
<para>
1346+
It's advisable to create <acronym>MCV</acronym> statistics objects only
1347+
on combinations of columns that are actually used in conditions together,
1348+
and for which misestimation of the number of groups is resulting in bad
1349+
plans. Otherwise, the <command>ANALYZE</command> and planning cycles
1350+
are just wasted.
1351+
</para>
1352+
</sect3>
1353+
12881354
</sect2>
12891355
</sect1>
12901356

‎doc/src/sgml/planstats.sgml‎

Lines changed: 114 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -455,7 +455,7 @@ rows = (outer_cardinality * inner_cardinality) * selectivity
455455
<secondary>multivariate</secondary>
456456
</indexterm>
457457

458-
<sect2>
458+
<sect2 id="functional-dependencies">
459459
<title>Functional Dependencies</title>
460460

461461
<para>
@@ -540,7 +540,7 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
540540
</para>
541541
</sect2>
542542

543-
<sect2>
543+
<sect2 id="multivariate-ndistinct-counts">
544544
<title>Multivariate N-Distinct Counts</title>
545545

546546
<para>
@@ -585,6 +585,118 @@ EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b;
585585
</para>
586586

587587
</sect2>
588+
589+
<sect2 id="mcv-lists">
590+
<title>MCV lists</title>
591+
592+
<para>
593+
As explained in <xref linkend="functional-dependencies"/>, functional
594+
dependencies are very cheap and efficient type of statistics, but their
595+
main limitation is their global nature (only tracking dependencies at
596+
the column level, not between individual column values).
597+
</para>
598+
599+
<para>
600+
This section introduces multivariate variant of <acronym>MCV</acronym>
601+
(most-common values) lists, a straightforward extension of the per-column
602+
statistics described in <xref linkend="row-estimation-examples"/>. These
603+
statistics address the limitation by storing individual values, but it is
604+
naturally more expensive, both in terms of building the statistics in
605+
<command>ANALYZE</command>, storage and planning time.
606+
</para>
607+
608+
<para>
609+
Let's look at the query from <xref linkend="functional-dependencies"/>
610+
again, but this time with a <acronym>MCV</acronym> list created on the
611+
same set of columns (be sure to drop the functional dependencies, to
612+
make sure the planner uses the newly created statistics).
613+
614+
<programlisting>
615+
DROP STATISTICS stts;
616+
CREATE STATISTICS stts2 (mcv) ON a, b FROM t;
617+
ANALYZE t;
618+
EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
619+
QUERY PLAN
620+
-------------------------------------------------------------------------------
621+
Seq Scan on t (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1)
622+
Filter: ((a = 1) AND (b = 1))
623+
Rows Removed by Filter: 9900
624+
</programlisting>
625+
626+
The estimate is as accurate as with the functional dependencies, mostly
627+
thanks to the table being fairly small and having a simple distribution
628+
with a low number of distinct values. Before looking at the second query,
629+
which was not handled by functional dependencies particularly well,
630+
let's inspect the <acronym>MCV</acronym> list a bit.
631+
</para>
632+
633+
<para>
634+
Inspecting the <acronym>MCV</acronym> list is possible using
635+
<function>pg_mcv_list_items</function> set-returning function.
636+
637+
<programlisting>
638+
SELECT m.* FROM pg_statistic_ext,
639+
pg_mcv_list_items(stxmcv) m WHERE stxname = 'stts2';
640+
index | values | nulls | frequency | base_frequency
641+
-------+----------+-------+-----------+----------------
642+
0 | {0, 0} | {f,f} | 0.01 | 0.0001
643+
1 | {1, 1} | {f,f} | 0.01 | 0.0001
644+
...
645+
49 | {49, 49} | {f,f} | 0.01 | 0.0001
646+
50 | {50, 50} | {f,f} | 0.01 | 0.0001
647+
...
648+
97 | {97, 97} | {f,f} | 0.01 | 0.0001
649+
98 | {98, 98} | {f,f} | 0.01 | 0.0001
650+
99 | {99, 99} | {f,f} | 0.01 | 0.0001
651+
(100 rows)
652+
</programlisting>
653+
654+
This confirms there are 100 distinct combinations in the two columns, and
655+
all of them are about equally likely (1% frequency for each one). The
656+
base frequency is the frequency computed from per-column statistics, as if
657+
there were no multi-column statistics. Had there been any null values in
658+
either of the columns, this would be identified in the
659+
<structfield>nulls</structfield> column.
660+
</para>
661+
662+
<para>
663+
When estimating the selectivity, the planner applies all the conditions
664+
on items in the <acronym>MCV</acronym> list, and then sums the frequencies
665+
of the matching ones. See <function>mcv_clauselist_selectivity</function>
666+
in <filename>src/backend/statistics/mcv.c</filename> for details.
667+
</para>
668+
669+
<para>
670+
Compared to functional dependencies, <acronym>MCV</acronym> lists have two
671+
major advantages. Firstly, the list stores actual values, making it possible
672+
to decide which combinations are compatible.
673+
674+
<programlisting>
675+
EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10;
676+
QUERY PLAN
677+
---------------------------------------------------------------------------
678+
Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
679+
Filter: ((a = 1) AND (b = 10))
680+
Rows Removed by Filter: 10000
681+
</programlisting>
682+
683+
Secondly, <acronym>MCV</acronym> lists handle a wider range of clause types,
684+
not just equality clauses like functional dependencies. See for example the
685+
example range query, presented earlier:
686+
687+
<programlisting>
688+
EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a &lt;= 49 AND b &gt; 49;
689+
QUERY PLAN
690+
---------------------------------------------------------------------------
691+
Seq Scan on t (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
692+
Filter: ((a &lt;= 49) AND (b &gt; 49))
693+
Rows Removed by Filter: 10000
694+
</programlisting>
695+
696+
</para>
697+
698+
</sect2>
699+
588700
</sect1>
589701

590702
<sect1 id="planner-stats-security">

‎doc/src/sgml/ref/create_statistics.sgml‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,10 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na
8181
<para>
8282
A statistics kind to be computed in this statistics object.
8383
Currently supported kinds are
84-
<literal>ndistinct</literal>, which enables n-distinct statistics, and
84+
<literal>ndistinct</literal>, which enables n-distinct statistics,
8585
<literal>dependencies</literal>, which enables functional
86-
dependency statistics.
86+
dependency statistics, and <literal>mcv</literal> which enables
87+
most-common values lists.
8788
If this clause is omitted, all supported statistics kinds are
8889
included in the statistics object.
8990
For more information, see <xref linkend="planner-stats-extended"/>
@@ -164,6 +165,36 @@ EXPLAIN ANALYZE SELECT * FROM t1 WHERE (a = 1) AND (b = 0);
164165
conditions are redundant and does not underestimate the row count.
165166
</para>
166167

168+
<para>
169+
Create table <structname>t2</structname> with two perfectly correlated columns
170+
(containing identical data), and a MCV list on those columns:
171+
172+
<programlisting>
173+
CREATE TABLE t2 (
174+
a int,
175+
b int
176+
);
177+
178+
INSERT INTO t2 SELECT mod(i,100), mod(i,100)
179+
FROM generate_series(1,1000000) s(i);
180+
181+
CREATE STATISTICS s2 (mcv) ON (a, b) FROM t2;
182+
183+
ANALYZE t2;
184+
185+
-- valid combination (found in MCV)
186+
EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 1);
187+
188+
-- invalid combination (not found in MCV)
189+
EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2);
190+
</programlisting>
191+
192+
The MCV list gives the planner more detailed information about the
193+
specific values that commonly appear in the table, as well as an upper
194+
bound on the selectivities of combinations of values that do not appear
195+
in the table, allowing it to generate better estimates in both cases.
196+
</para>
197+
167198
</refsect1>
168199

169200
<refsect1>

0 commit comments

Comments
 (0)