Parallel tuplesort (for parallel B-Tree index creation)

Edit
ID 690
Title Parallel tuplesort (for parallel B-Tree index creation)
CI (CFBot) Not processed
Stats (from CFBot) Unknown
Topic Performance
Tags
Created 2016-08-01 22:19:22
Last modified 2018-02-03 04:29:57 (7 years ago)
Latest email 2018-02-08 12:45:52 (7 years ago)
Status
2018-01 (2018-01-01 – 2018-01-31): Committed
2017-03 (2017-03-01 – 2017-03-31): Moved to different CF
2017-01 (2017-01-01 – 2017-01-31): Moved to different CF
2016-11 (2016-11-01 – 2016-11-30): Moved to different CF
2016-09 (2016-09-01 – 2016-09-30): Moved to different CF
Target version
Authors Peter Geoghegan (pgeoghegan), Rushabh Lathia (rlathia)
Reviewers Thomas Munro (macdice)Become reviewer
Committer Robert Haas (rhaas)
Links Wiki
Emails
Parallel tuplesort (for parallel B-Tree index creation)
First at 2016-08-01 22:18:22 by Peter Geoghegan <pg at heroku.com>
Latest at 2018-02-08 12:45:52 by Robert Haas <robertmhaas at gmail.com>
Latest attachment (0001-Mark-logtape.c-buffer-s-tail-as-defined-to-Valgrind.patch) at 2018-02-06 18:33:45 from Peter Geoghegan <pg at bowt.ie>
    Attachment (0001-Mark-logtape.c-buffer-s-tail-as-defined-to-Valgrind.patch) at 2018-02-06 18:33:45 from Peter Geoghegan <pg at bowt.ie> (Patch: Yes)
    Attachment (0001-Add-logtape.c-Valgrind-suppression.patch) at 2018-02-03 03:26:59 from Peter Geoghegan <pg at bowt.ie> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-02-02 16:16:50 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Chaos-monkey-fork-failure.patch) at 2018-01-25 01:31:19 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (pessimistic-fork-failure-detector-v2.patch) at 2018-01-24 20:13:10 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-20 01:33:28 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (propagate-reindex-state-v1.patch) at 2018-01-17 18:40:14 from Robert Haas <robertmhaas at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-16 00:54:40 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch) at 2018-01-05 22:17:51 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0004-Calculate-parallel-worker-inside-index_create.patch) at 2018-01-02 09:38:14 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0003-Adjust-planner-code-to-consider-parallel_leader_part.patch) at 2017-12-12 10:09:29 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0002-Adjust-planner-code-to-consider-parallel_leader_part.patch) at 2017-12-08 07:28:59 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (barrier-forget-participants-v1.patch) at 2017-12-08 02:03:50 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v14.patch) at 2017-12-07 08:25:06 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v13.patch) at 2017-11-14 09:41:39 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v12_rebase.patch) at 2017-10-26 11:22:16 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v12.patch) at 2017-10-10 09:23:41 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting_v11.patch) at 2017-09-19 10:21:29 from Rushabh Lathia <rushabh.lathia at gmail.com> (Patch: Yes)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-20 01:03:51 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-03-12 22:05:40 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup-100.txt) at 2017-02-16 02:05:14 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0001-Add-parallel-B-tree-index-build-sorting.patch.gz) at 2017-02-10 00:10:14 from Peter Geoghegan <pg at bowt.ie> (Patch: No)
    Attachment (speedup.txt) at 2017-02-03 13:04:55 from Thomas Munro <thomas.munro at enterprisedb.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2017-01-03 23:53:59 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0002-Add-temporary-testing-tools.patch.gz) at 2016-12-04 01:29:01 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (max-sort-tapes.patch) at 2016-11-10 00:01:30 from Robert Haas <robertmhaas at gmail.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-11-08 04:28:35 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0003-Add-force_btree_randomaccess-GUC-for-testing.patch.gz) at 2016-10-25 01:17:41 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-10-08 00:47:13 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (0001-Refactor-LogicalTapeSet-LogicalTape-interface.patch) at 2016-09-21 16:52:00 from Heikki Linnakangas <hlinnaka at iki.fi> (Patch: Yes)
    Attachment (0003-Rearrange-header-file-include-directives.patch.gz) at 2016-09-11 18:05:07 from Peter Geoghegan <pg at heroku.com> (Patch: No)
    Attachment (root_displace_patched_timings.log) at 2016-09-10 00:22:01 from Claudio Freire <klaussfreire at gmail.com> (Patch: No)
    Attachment (0003-Displace-heap-s-root-during-tuplesort-merge.patch) at 2016-08-16 00:33:38 from Peter Geoghegan <pg at heroku.com> (Patch: Yes)
    Attachment (0001-Cap-the-number-of-tapes-used-by-external-sorts.patch.gz) at 2016-08-01 22:18:22 from Peter Geoghegan <pg at heroku.com> (Patch: No)

Annotations

When Who Mail Annotation
2016-09-11 18:07:19 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-09-11 18:05:07		 V2 -- uses maintenance_work_mem as budget for entire CREATE INDEX, regardless of number of workers
2016-10-08 00:49:11 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-08 00:47:13		 V3 -- rebased on top of Heikki's logtape.c preload memory patch
2016-10-25 02:06:25 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-25 01:17:41		 V4 -- mechanical rebase on top of refactoring commit b75f467b6eec0678452fd8d7f8d306e6df3a1076
2016-10-26 16:42:26 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-10-26 16:02:21		 Do-over of original benchmark from August 1, showing how preloading work from Heikki (and other improvements) have increased scalability of parallel CREATE INDEX
2016-11-08 04:35:03 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-11-08 04:28:35		 V5 -- Overhaul to cost model, new testing tool
2016-12-04 01:30:15 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-12-04 01:29:01		 V6 -- tuplesort.c now uses condition variables
2017-01-03 23:57:44 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2017-01-03 23:53:59		 V7 -- rebased on top of logtape.c simplifications, pg_restore avoids using parallel CREATE INDEX unless storage parameter was used, refactoring
2017-02-10 00:12:17 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-02-10 00:10:14		 V8 -- rebased on top of logtape.c bugfix, implements BufFile refcount mechanism
2017-03-12 22:11:30 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-03-12 22:05:40		 V9 -- Bulletproof resource management, significantly simplified cost model
2017-03-20 01:06:55 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2017-03-20 01:03:51		 V10 -- Fixes bug caused by commit 2609e91fc, bitrot
2017-11-14 22:55:27 Peter Geoghegan (pgeoghegan) From Rushabh Lathia <rushabh.lathia@gmail.com>
at 2017-11-14 09:41:39		 v13 -- Contains fixes requested by Peter following adopting patch to parallel hash join BufFile infrastructure
2018-01-06 00:16:32 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@bowt.ie>
at 2018-01-05 22:17:51		 Revision of Rushabh's most recent patch -- mostly just polishing.
Parallel tuplesort, partitioning, merging, and the future
First at 2016-08-08 19:44:02 by Peter Geoghegan <pg at heroku.com>
Latest at 2016-08-12 19:22:24 by Robert Haas <robertmhaas at gmail.com>

Annotations

When Who Mail Annotation
2016-08-08 20:54:30 Peter Geoghegan (pgeoghegan) From Peter Geoghegan <pg@heroku.com>
at 2016-08-08 19:44:02		 Related design discussion -- how tuplesort.c may in the future support partitioning
Cost model for parallel CREATE INDEX
First at 2017-02-28 19:28:15 by Peter Geoghegan <pg at bowt.ie>
Latest at 2017-03-09 21:11:21 by Peter Geoghegan <pg at bowt.ie>
History
When Who What
2018-02-03 04:29:57 Peter Geoghegan (pgeoghegan) Closed in commitfest 2018-01 with status: Committed
2018-02-03 04:29:57 Peter Geoghegan (pgeoghegan) Changed committer to rhaas
2018-01-10 03:13:06 Peter Geoghegan (pgeoghegan) New status: Ready for Committer
2018-01-06 00:16:32 Peter Geoghegan (pgeoghegan) Added annotation "Revision of Rushabh's most recent patch -- mostly just polishing." to CAH2-Wzk66gCxRjs8tQWQwBc8kg1VG9+KNitutjg3K=jsYL+euA@mail.gmail.com
2017-11-14 22:56:18 Peter Geoghegan (pgeoghegan) Changed authors to Peter Geoghegan (pgeoghegan), Rushabh Lathia (rlathia)
2017-11-14 22:55:27 Peter Geoghegan (pgeoghegan) Added annotation "v13 -- Contains fixes requested by Peter following adopting patch to parallel hash join BufFile infrastructure" to CAGPqQf17vxDXFi=u9yf=aDF_0LJ+NGmRKJ0fUJYVUtzseQm6Zw@mail.gmail.com
2017-11-14 22:51:38 Peter Geoghegan (pgeoghegan) Closed in commitfest 2017-03 with status: Moved to next CF
2017-11-14 22:51:32 Peter Geoghegan (pgeoghegan) New status: Needs review
2017-03-23 02:32:23 Peter Geoghegan (pgeoghegan) Closed in commitfest 2017-03 with status: Returned with feedback
2017-03-20 01:06:55 Peter Geoghegan (pgeoghegan) Added annotation "V10 -- Fixes bug caused by commit 2609e91fc, bitrot" to CAH2-WznqgFJVb0OpbrS445TnyS8ib8kSxAASvLqLCaSnQeyd0Q@mail.gmail.com
2017-03-12 22:11:30 Peter Geoghegan (pgeoghegan) Added annotation "V9 -- Bulletproof resource management, significantly simplified cost model" to CAH2-Wznuf2nwsr6=4n8aRQtqCu3PaFYuvDEoYeEenYK-bXYX4w@mail.gmail.com
2017-03-12 22:06:03 Peter Geoghegan (pgeoghegan) New status: Needs review
2017-02-28 19:32:44 Peter Geoghegan (pgeoghegan) Attached mail thread CAH2-WzmjVMCUviDnUmrJnvhfPpzODtCM71NEHx7_QYCtz+=8ng@mail.gmail.com
2017-02-27 22:35:14 Peter Geoghegan (pgeoghegan) New status: Waiting on Author
2017-02-10 00:12:17 Peter Geoghegan (pgeoghegan) Added annotation "V8 -- rebased on top of logtape.c bugfix, implements BufFile refcount mechanism" to CAH2-WzmWtorLU0qi63dTgNbBJPds1wRLDtoZSDRwkRWdvBnMww@mail.gmail.com
2017-02-01 04:38:04 Michael Paquier (michael-kun) Closed in commitfest 2017-01 with status: Moved to next CF
2017-01-03 23:57:44 Peter Geoghegan (pgeoghegan) Added annotation "V7 -- rebased on top of logtape.c simplifications, pg_restore avoids using parallel CREATE INDEX unless storage parameter was used, refactoring" to CAM3SWZRkQ64nhFiWeGTx4CUuvCaL8m+GSg8PhssHY=E-_wNpBA@mail.gmail.com
2016-12-07 00:25:00 Thomas Munro (macdice) Added macdice as reviewer
2016-12-05 04:26:29 Haribabu Kommi (haribabu) Closed in commitfest 2016-11 with status: Moved to next CF
2016-12-04 01:30:15 Peter Geoghegan (pgeoghegan) Added annotation "V6 -- tuplesort.c now uses condition variables" to CAM3SWZQ0EgCxs2NoEUwB6BACi-pp5qVmp432i+Wh_XyUrg1FbQ@mail.gmail.com
2016-11-08 05:52:54 Peter Geoghegan (pgeoghegan) Deleted annotation ""shift down" merge heap patch committed" from 19f05353-9514-9139-619a-dbaa7a0e5e78@iki.fi
2016-11-08 05:52:37 Peter Geoghegan (pgeoghegan) Deleted annotation "This is a spin-off of the parallel CREATE INDEX stuff, intended to represent 3 patches that are helpful with merging. Merge performance is particularly important for parallel tuplesort, but is independently useful." from CAM3SWZRhBhiknTF_=NjDSnNZ11hx=U_SEYwbc5vd=x7M4mMiCw@mail.gmail.com
2016-11-08 05:52:12 Peter Geoghegan (pgeoghegan) Deleted annotation "Patch has bitrot. New version pending." from CAM3SWZTjULhhHPMYb9KYKD48rs_RgnMQPMZMdf0dqkRpZoY_ZA@mail.gmail.com
2016-11-08 04:35:03 Peter Geoghegan (pgeoghegan) Added annotation "V5 -- Overhaul to cost model, new testing tool" to CAM3SWZSA_+CBsaYyb22cdqwkfNgD9vQL74NVYnbu62ZqhLD4Og@mail.gmail.com
2016-10-27 20:21:59 Peter Geoghegan (pgeoghegan) Changed wikilink to https://wiki.postgresql.org/wiki/Parallel_External_Sort
2016-10-26 20:16:04 Peter Geoghegan (pgeoghegan) Detached mail thread f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-10-26 16:42:26 Peter Geoghegan (pgeoghegan) Added annotation "Do-over of original benchmark from August 1, showing how preloading work from Heikki (and other improvements) have increased scalability of parallel CREATE INDEX" to CAM3SWZSsvig8eZtxCG1L9bZZw+-uWdJ48yxSM41QmNLtJeQSyA@mail.gmail.com
2016-10-25 02:06:25 Peter Geoghegan (pgeoghegan) Added annotation "V4 -- mechanical rebase on top of refactoring commit b75f467b6eec0678452fd8d7f8d306e6df3a1076" to CAM3SWZStcX5DNxRvjGTj4KRvjg+sV2CdxJvL9S4cksbRumVmhA@mail.gmail.com
2016-10-08 00:49:11 Peter Geoghegan (pgeoghegan) Added annotation "V3 -- rebased on top of Heikki's logtape.c preload memory patch" to CAM3SWZTmkOFEiCDpUNaO4n9-1xcmWP-1NXmT7h0Pu3gM2YuHvg@mail.gmail.com
2016-10-03 03:37:07 Michael Paquier (michael-kun) Closed in commitfest 2016-09 with status: Moved to next CF
2016-09-12 04:17:42 Peter Geoghegan (pgeoghegan) Added annotation ""shift down" merge heap patch committed" to 19f05353-9514-9139-619a-dbaa7a0e5e78@iki.fi
2016-09-12 04:10:02 Peter Geoghegan (pgeoghegan) Added annotation "Review" to CAM3SWZQrU12mog=TG9jvNOpeUL5Krk9N1Y8GmsLGTGRBJui-aQ@mail.gmail.com
2016-09-12 04:09:30 Peter Geoghegan (pgeoghegan) Added annotation "Latest version from Heikki -- applies on top master branch on 2016-09-11 (with Peter's 0003-* merge heap patch committed)" to 49817174-c2b6-1378-3d3e-285d270f8fcc@iki.fi
2016-09-12 04:07:11 Peter Geoghegan (pgeoghegan) Added annotation "Heikki's proposed alternative to patch 0002-* from original patch set (batch memory patch)" to f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-09-12 04:01:13 Peter Geoghegan (pgeoghegan) Attached mail thread f298f77a-cf06-e70c-d5a4-a20b472b4627@iki.fi
2016-09-11 18:07:19 Peter Geoghegan (pgeoghegan) Added annotation "V2 -- uses maintenance_work_mem as budget for entire CREATE INDEX, regardless of number of workers" to CAM3SWZTW+3vmZugxhy=_jdm5YxNXGdAEHv5Sz6jBpdyfsJVkRg@mail.gmail.com
2016-08-22 21:50:52 Peter Geoghegan (pgeoghegan) Added annotation "Patch has bitrot. New version pending." to CAM3SWZTjULhhHPMYb9KYKD48rs_RgnMQPMZMdf0dqkRpZoY_ZA@mail.gmail.com
2016-08-16 00:47:06 Peter Geoghegan (pgeoghegan) Added annotation "This is a spin-off of the parallel CREATE INDEX stuff, intended to represent 3 patches that are helpful with merging. Merge performance is particularly important for parallel tuplesort, but is independently useful." to CAM3SWZRhBhiknTF_=NjDSnNZ11hx=U_SEYwbc5vd=x7M4mMiCw@mail.gmail.com
2016-08-08 20:54:30 Peter Geoghegan (pgeoghegan) Added annotation "Related design discussion -- how tuplesort.c may in the future support partitioning" to CAM3SWZR+ATYAzyMT+hm-Bo=1L1smtJbNDtibwBTKtYqS0dYZVg@mail.gmail.com
2016-08-08 20:53:40 Peter Geoghegan (pgeoghegan) Attached mail thread CAM3SWZR+ATYAzyMT+hm-Bo=1L1smtJbNDtibwBTKtYqS0dYZVg@mail.gmail.com
2016-08-01 22:19:31 Peter Geoghegan (pgeoghegan) Changed authors to Peter Geoghegan (pgeoghegan)
2016-08-01 22:19:22 Peter Geoghegan (pgeoghegan) Attached mail thread CAM3SWZQKM=Pzc=CAHzRixKjp2eO5Q0Jg1SoFQqeXFQ647JiwqQ@mail.gmail.com
2016-08-01 22:19:22 Peter Geoghegan (pgeoghegan) Created patch record
Edit