From 4ad11ff6cbcd818f5dcd5b34b3267f0475e938f1 Mon Sep 17 00:00:00 2001
From: Nikolay Samokhvalov <nik@postgres.ai>
Date: Tue, 18 Mar 2025 20:45:07 -0700
Subject: [PATCH 1/5] Single-session query regression performance testing

---
 ...on_query_regression_performance_testing.md | 471 ++++++++++++++++++
 1 file changed, 471 insertions(+)
 create mode 100644 0095_single_session_query_regression_performance_testing.md

diff --git a/0095_single_session_query_regression_performance_testing.md b/0095_single_session_query_regression_performance_testing.md
new file mode 100644
index 0000000..22514cd
--- /dev/null
+++ b/0095_single_session_query_regression_performance_testing.md
@@ -0,0 +1,471 @@
+# Single-session query regression performance testing
+## Definition and goals
+Goal: compare execution plans for 2 or more different situations (example: two different Postgres major versions), using just single connection. This type of benchmark is designed to study execution plans using `EXPLAIN (ANALYZE, BUFFERS)`. 
+
+This type of benchmark is not applicable for research of Postgres components behavior (e.g., buffer pool and contention in it, WAL writer or checkpointer activities).
+
+## When to use
+This approach can be used in varous cases. Some examples:
+- Major Postgres upgrades: compare behavior of planner and executor on old and new version
+- Migration to partitioned schema: compare execution plans and the key characteristics (first of all, `BUFFERS`) before and after partitioning
+- Reconfiguration of Postgres server: e.g., change of planner settings such as `random_page_cost`, or `work_mem`
+
+## Shared environments and hardware differences
+Single-session benchmark can be executed on machines that:
+- have weaker resources compared to production (e.g. less RAM or weaker disks)
+- different filesystem (e.g. ZFS with CoW support vs. ext4)
+- shared environments with significant risks of saturation
+
+## Principles
+However, to allow such differences, we must follow these rules:
+1. **Planner settings** (`select * from pg_settings where category ~ '^Query Tuning'`) and `work_mem` should the the same as on production machines – this is needed to make Postgres planner behave the same as on production.
+    - For example, we can use a very low `shared_buffers` and the very small RAM, but we do need to set `effective_cache_size` as on production – interesting that:
+        - Postgres planner doesn't know anything about actual resources the machine has – CPU, RAM, disk
+        - it doesn't take into account `shared_buffers`, this parameter doesn't affect planner's decisions
+        - but we do need to use the proper `effective_cache_size`, and we can use any value, even much bigger than actual RAM – this value is going to define the planner's behavior (as well as, for example, `work_mem`)
+
+Note: SQL [query](https://postgres.ai/docs/how-to-guides/administration/postgresql-configuration#postgresql-configuration-in-clones) to get all non-default values of the parameters affecting the planner's behavior.
+2. **A clone of production database** has to be used.
+    - Physical level (a clone of PGDATA) is more preferrable than logical one (dump/restore) since it not only gives the same row numbers for tables, but also physical distribution, bloat, `pg_class.relpages`, the changes to have identical plans as on production are very high.
+3. **BUFFERS first, timing second**: understanding that we have hardware differences, different (compared to production) state of caches (Postgres buffer pool and page cache), different filesystem (e.g. ZFS in case of DBLab), and risks of saturated resources since we workin a shared environment and other people or CI/CD pipelines might use a lot of resources, we need to pay less attention to timing metrics, which are very volatile, and prefer using IO and data volume metrics – buffer numbers, rows – as well as plan structure and costs.
+
+## "pgss_pgsa" – single-session benchmarks for Postgres major upgrades
+For Postgres major upgrades, we can follow this methodology to perform query plan regression analysis:
+1. **pgss samples**: get query groups (normalized queries) from `pg_stat_statements` from production
+    - Top-N by `total_exec_time` (most important in terms of resource utilization)
+    - Top-N by `mean_exec_time` (most important in terms of UX)
+    - Top-N by `calls` (degradation of high-frequent queries can be very noticeable)
+    - optionally, more – e.g., `total_plan_time`
+    - optionally, not only from primary but also from standbys
+1. **pgsa samples**:  sample `pg_stat_activity` during substantial period of time to collect samples of queries
+    - `track_activity_query_size` should be increased (default: 1024, which is usually not enough; unfortunately to change it, a restart is required)
+1. Join samples:
+    - move samples to some machine with Postgres and load them for analysis
+    - for newer Postgres versions (14+), use `query_id` (identifier of this backend's most recent query. [Details](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)) to JOIN
+    - for older, use [libpg_query](https://github.com/pganalyze/libpg_query) to calculate fingerprint for both pgss and pgsa samples, and then JOIN
+    - optionally, leave only N samples for each pgss group
+1. Using DBLab, get two clones – older Postgres version and newer one (e.g. pg_14 and pg_16)
+    - clones must be created based on the same snapshot, so the data in clones is identical
+1. Collect query execution plans using `EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT JSON)`
+    - 2-3 executions are recommended to collect plans with warmed-up caches
+1. Perform regression analysis of plans with focus on:
+    - plan structure
+    - data volumes (`BUFFERS`)
+    - costs
+    - and, finally, timing – as the last metric
+
+### Collect pgss and pgsa samples
+On production primary (and optionally, on standbys), collect pgss and pgsa samples:
+
+```shell
+export PGSS_LIMIT=20
+export PGSA_SAMPLING_SECONDS=600
+export _PSQL="psql"
+
+rm -rf /tmp/pgss_sampling.error.log /tmp/pgsa_sampling.csv
+rm -rf /tmp/pgsa_sampling.error.log /tmp/pgsa_sampling.csv
+
+# Note: For Postgres 12 and earlier versions: replace total_exec_time and mean_exec_time with total_time and mean_time
+$_PSQL -XAtc "
+  copy (
+    select
+      now() as date_captured,
+      queryid,
+      regexp_replace(query, E'[ \\t\\n\\r]+', ' ', 'g') as query,
+      calls as pgss_calls,
+      total_exec_time as pgss_total_time,
+      mean_exec_time as pgss_mean_time
+    from (select * from pg_stat_statements order by total_exec_time desc limit ${PGSS_LIMIT}) a
+    union
+    select
+      now() as date_captured,
+      queryid,
+      regexp_replace(query, E'[ \\t\\n\\r]+', ' ', 'g') as query,
+      calls as pgss_calls,
+      total_exec_time as pgss_total_time,
+      mean_exec_time as pgss_mean_time
+    from (select * from pg_stat_statements order by calls desc limit ${PGSS_LIMIT}) a
+    union
+    select
+      now() as date_captured,
+      queryid,
+      regexp_replace(query, E'[ \\t\\n\\r]+', ' ', 'g') as query,
+      calls as pgss_calls,
+      total_exec_time as pgss_total_time,
+      mean_exec_time as pgss_mean_time
+    from (select * from pg_stat_statements order by mean_exec_time desc limit ${PGSS_LIMIT}) a
+  ) to stdout csv delimiter ',' \
+" 2>/tmp/pgss_sampling.error.log >/tmp/pgss_sampling.csv
+
+# Note: For Postgres 13 and earlier versions: remove query_id
+for i in $(seq 1 "${PGSA_SAMPLING_SECONDS}"); do
+  PGOPTIONS="-c statement_timeout=3s" $_PSQL -XAtc "
+    copy(
+      select now() as date_captured,
+      query_id,
+      regexp_replace(query, E'[ \\t\\n\\r]+', ' ', 'g') as query
+      from pg_stat_activity
+      where
+        clock_timestamp() - query_start < '1 sec'::interval
+        and query not in ('', ';', 'select;', 'select 1;')
+        and query not ilike 'SET %'
+        and length(query) < (select (setting::int - 1) from pg_settings where name = 'track_activity_query_size')
+        and backend_type = 'client backend'
+        and datname = current_database()
+    ) to stdout csv delimiter ','" \
+    2>>/tmp/pgsa_sampling.error.log >>/tmp/pgsa_sampling.csv
+
+  sleep 1
+done
+```
+
+Copy `/tmp/pgss_sampling.csv` and `/tmp/pgsa_sampling.csv` files.
+
+### Load samples and JOIN
+
+On some machine with Postgres (can be a laptop, or DBLab clone), load the files collected in previous step, and JOIN them.
+
+```sql
+drop schema pgss_pgsa cascade;
+create schema pgss_pgsa;
+
+drop table if exists pgss_pgsa.pgss_sampling;
+create table pgss_pgsa.pgss_sampling (
+  date_captured timestamptz,
+  queryid text,
+  query text,
+  pgss_calls int8,
+  pgss_total_time double precision,
+  pgss_mean_time double precision
+);
+
+\copy pgss_pgsa.pgss_sampling from '/tmp/pgss_sampling.csv' csv delimiter ',';
+
+drop table if exists pgss_pgsa.pgsa_sampling;
+create table pgss_pgsa.pgsa_sampling (
+  date_captured timestamptz,
+  query_id text,
+  query text
+);
+
+\copy pgss_pgsa.pgsa_sampling from '/tmp/pgsa_sampling.csv' csv delimiter ',';
+```
+
+For Postgres 13 and earlier versions only:
+
+```sql
+drop table if exists pgss_pgsa.pgss_fingerprint;
+create table pgss_pgsa.pgss_fingerprint (queryid text, query text, fingerprint text);
+
+drop table if exists pgss_pgsa.pgsa_fingerprint;
+create table pgss_pgsa.pgsa_fingerprint (query text, fingerprint text);
+```
+
+If production is running on Postgres 14 or higher, use `pgsa.query_id` / `pgss.queryid` to match samples:
+
+```shell
+psql -XAtc "
+  copy (
+    with samples as (
+      select
+        ss.queryid,
+        sa.query,
+        ss.pgss_calls,
+        ss.pgss_total_time,
+        ss.pgss_mean_time,
+        row_number() over (partition by ss.queryid order by random()) as rn
+      from 
+        pgss_pgsa.pgss_sampling ss
+      join 
+        pgss_pgsa.pgsa_sampling sa
+      on 
+        ss.queryid = sa.query_id
+      order by 
+        ss.queryid, ss.pgss_total_time desc
+    )
+    select *
+    from samples
+    where query is not null
+    and rn < 4
+    ) to stdout csv delimiter ',' \
+" > /tmp/query_samples.csv
+```
+
+Otherwise, use `./query_fingerprint.rb` to calculate fingerprints:
+
+<details><summary>Click here to expand...</summary><p>
+
+```shell
+ruby query_fingerprint.rb > query_fingerprint.log 2>&1
+```
+
+Check for errors (inspecting `query_fingerprint.log`) and result:
+```sql
+select distinct sa.query
+from pgss_pgsa.pgss_fingerprint ss
+join pgss_pgsa.pgsa_fingerprint sa on ss.fingerprint = sa.fingerprint;
+
+select count(distinct ss.queryid), count(ss.query), count(sa.query)
+from pgss_pgsa.pgss_fingerprint ss
+left join pgss_pgsa.pgsa_fingerprint sa on ss.fingerprint = sa.fingerprint;
+```
+
+Export samples to further use them on production clones (to gather execution plans):
+```shell
+psql -XAtc "
+  copy (
+    with samples as (
+      select
+        ss.queryid,
+        sa.query,
+        pgss_calls,
+        pgss_total_time,
+        pgss_mean_time,
+        row_number() over (partition by ss.queryid order by random())
+      from pgss_fingerprint ss
+      join pgsa_fingerprint sa using (fingerprint)
+      join pgss_sampling pgss using (queryid)
+      order by ss.queryid, pgss.total_time desc
+    )
+    select *
+    from samples
+    where query is not null
+    and row_number < 4
+    ) to stdout csv delimiter ',' \
+" > /tmp/query_samples.csv
+```
+</p></details>
+
+### Prepare two clones (for two major Postgres versions)
+Using DBLab, create two clones. Use the same snapshot for them.
+
+Upgrade one of them to newer Postgres version. See [How to upgrade Postgres to a new major version in the DBLab clone](https://postgres.ai/docs/how-to-guides/cloning/clone-upgrade)
+
+:warning: Do not forget to run DB-wide ANALYZE after upgrade - for plan analysis we do need stats to be collected.
+
+### Collect plans
+Prepare schema in both clones:
+```sql
+select version();
+
+begin;
+drop schema if exists pgss_pgsa cascade;  -- ! careful – ensure that no data loss here
+create schema pgss_pgsa;
+
+create table pgss_pgsa.samples(
+  queryid int8,
+  query text,
+  pgss_calls int8,
+  pgss_total_time double precision,
+  pgss_mean_time double precision,
+  row_number int8
+);
+alter table pgss_pgsa.samples add primary key (queryid, row_number);
+
+create extension if not exists "uuid-ossp";
+create table pgss_pgsa.plans (
+  id uuid primary key default uuid_generate_v4(),
+  queryid int8 not null,
+  row_number int8 not null,
+  created_at timestamptz not null default now(),
+  plan json,
+  version_major int not null default current_setting('server_version_num')::int / 10000,
+  version_full text not null default version()
+);
+alter table pgss_pgsa.plans
+  add constraint f_1
+  foreign key (queryid, row_number)
+  references pgss_pgsa.samples (queryid, row_number);
+
+create or replace function pgss_pgsa.explain_analyze(
+  in query text,
+  out plan json
+) as $$
+declare
+  sql text;
+begin
+  -- Attempt to perform a detailed EXPLAIN ANALYZE with all options
+  sql := format('explain (analyze, buffers, verbose, format json) %s', query);
+  begin
+    execute sql into plan;
+  exception when others then
+    -- If an error occurs, fallback to a simple EXPLAIN
+    sql := format('explain (format json) %s', query);
+    begin
+      execute sql into plan;
+      raise notice 'Fallback to plain EXPLAIN due to error in EXPLAIN ANALYZE.';
+    exception when others then
+      -- Log the error and set plan to null
+      raise notice 'Error executing EXPLAIN: %', sqlerrm;
+      plan := null;
+    end;
+  end;
+  return;
+end;
+$$ language plpgsql;
+
+create or replace procedure pgss_pgsa.analyze_samples(
+  queryid int8 default null,
+  row_number int default null
+) as $$
+declare
+  r record;
+  plan json;
+begin
+  for r in
+    select *
+    from pgss_pgsa.samples
+    where
+      (samples.queryid = analyze_samples.queryid and samples.row_number = analyze_samples.row_number)
+      or (analyze_samples.queryid is null and analyze_samples.row_number is null) -- get all records
+    order by queryid, row_number
+  loop
+    raise info '[%] Collect plans for query %:%...', now(), r.queryid, r.row_number;
+
+    -- Collect plan for the first time
+    plan := pgss_pgsa.explain_analyze(r.query);
+    if plan is not null then
+      insert into pgss_pgsa.plans(queryid, row_number, plan)
+      values (r.queryid, r.row_number, plan);
+      raise info '[%] 1st – done.', now();
+      commit;
+
+      -- Collect plan for the second time
+      plan := pgss_pgsa.explain_analyze(r.query);
+      if plan is not null then
+        insert into pgss_pgsa.plans(queryid, row_number, plan)
+        values (r.queryid, r.row_number, plan);
+        raise info '[%] 2nd – done.', now();
+        commit;
+
+        -- Collect plan for the third time
+        plan := pgss_pgsa.explain_analyze(r.query);
+        if plan is not null then
+          insert into pgss_pgsa.plans(queryid, row_number, plan)
+          values (r.queryid, r.row_number, plan);
+          raise info '[%] 3rd – done.', now();
+          commit;
+        else
+          raise notice '[%] 3rd pass failed for query %:%.', now(), r.queryid, r.row_number;
+        end if;
+      else
+        raise notice '[%] 2nd pass failed for query %:%.', now(), r.queryid, r.row_number;
+      end if;
+    else
+      raise notice '[%] 1st pass failed for query %:%.', now(), r.queryid, r.row_number;
+    end if;
+  end loop;
+end
+$$ language plpgsql;
+
+commit;
+```
+```sql
+\copy pgss_pgsa.samples from '/tmp/query_samples.csv' csv delimiter ',';
+```
+
+Now, collect plans:
+
+- on the clone with older Postgres:
+
+    ```sql
+    call pgss_pgsa.analyze_samples();
+    \copy pgss_pgsa.plans to '/tmp/plans_pg_old.csv' csv delimiter ',';
+    ```
+
+- similarly, on the clone with newer Postgres:
+
+    ```sql
+    call pgss_pgsa.analyze_samples();
+    \copy pgss_pgsa.plans to '/tmp/plans_pg_new.csv' csv delimiter ',';
+    ```
+
+### Analyze results and draw conclusions
+
+Import plans for analysis
+
+- on the clone with older Postgres:
+
+    ```sql
+    \copy pgss_pgsa.plans from '/tmp/plans_pg_new.csv' csv delimiter ',';
+    ```
+
+Note: Replace `version_major` value with versions for old and new Postgres
+
+Dump result it as CSV:
+```shell
+export PLANS_SQL="
+  with plans_pg_old as (
+    select
+      *,
+      row_number() over (partition by queryid, row_number order by created_at) as seq_num
+    from pgss_pgsa.plans
+    where version_major = 14
+  ), plans_pg_new as (
+    select
+      *,
+      row_number() over (partition by queryid, row_number order by created_at) as seq_num
+    from pgss_pgsa.plans
+    where version_major = 16
+  )
+  select
+    old.queryid,
+    old.row_number,
+
+    (old.plan::jsonb->0->'Plan'->>'Node Type' = new.plan::jsonb->0->'Plan'->>'Node Type') as diff_root_node,
+    (
+      ((new.plan::jsonb->0->'Plan'->>'Shared Hit Blocks')::int8 + (new.plan::jsonb->0->'Plan'->>'Shared Read Blocks')::int8)
+      - ((old.plan::jsonb->0->'Plan'->>'Shared Hit Blocks')::int8 + (old.plan::jsonb->0->'Plan'->>'Shared Read Blocks')::int8)
+    ) as diff_shared_blks,
+    (
+      (new.plan::jsonb->0->'Plan'->>'Total Cost')::numeric
+      - (old.plan::jsonb->0->'Plan'->>'Total Cost')::numeric
+    ) as diff_total_cost,
+    (
+      (new.plan::jsonb->0->>'Planning Time')::numeric
+      - (old.plan::jsonb->0->>'Planning Time')::numeric
+    ) as diff_plan_time,
+    (
+      (new.plan::jsonb->0->>'Execution Time')::numeric
+      - (old.plan::jsonb->0->>'Execution Time')::numeric
+    ) as diff_exec_time,
+    --
+    old.plan::jsonb->0->'Plan'->>'Node Type' as pg_old_root_node,
+    old.plan::jsonb->0->>'Planning Time'      as pg_old_plan_time,
+    old.plan::jsonb->0->>'Execution Time'     as pg_old_exec_time,
+    old.plan::jsonb->0->'Plan'->>'Total Cost'         as pg_old_total_cost,
+    old.plan::jsonb->0->'Plan'->>'Shared Hit Blocks'  as pg_old_shared_blks_hit,
+    old.plan::jsonb->0->'Plan'->>'Shared Read Blocks' as pg_old_shared_blks_read,
+    old.plan::jsonb->0->'Plan'->>'Actual Total Time'  as pg_old_actual_total_time,
+    --
+    new.plan::jsonb->0->'Plan'->>'Node Type' as pg_new_root_node,
+    new.plan::jsonb->0->>'Planning Time'      as pg_new_plan_time,
+    new.plan::jsonb->0->>'Execution Time'     as pg_new_exec_time,
+    new.plan::jsonb->0->'Plan'->>'Total Cost'         as pg_new_total_cost,
+    new.plan::jsonb->0->'Plan'->>'Shared Hit Blocks'  as pg_new_shared_blks_hit,
+    new.plan::jsonb->0->'Plan'->>'Shared Read Blocks' as pg_new_shared_blks_read,
+    new.plan::jsonb->0->'Plan'->>'Actual Total Time'  as pg_new_actual_total_time,
+
+    old.plan as pg_old_plan,
+    new.plan as pg_new_plan,
+
+    s.query
+
+  from plans_pg_old as old
+  join plans_pg_new as new using (queryid, row_number, seq_num)
+  join pgss_pgsa.samples s on old.queryid = s.queryid and old.row_number = s.row_number
+  where old.seq_num = 3
+  order by 1, 2
+"
+```
+```shell
+psql -Xc "copy ($PLANS_SQL) to stdout with csv header delimiter ','" > /tmp/plans_pg_old_vs_pg_new.csv
+```
+
+Upload to a spreadsheet, use conditional formatting to highliad improved and degraded queries.
+
+TODO:
+- analyze diffs
+- conclude if there is noticeable regression, for which queries
+- visualize
+
-- 
GitLab


From 6c2351391ffcaeb7ca079beb4e0821bf2ff36fd5 Mon Sep 17 00:00:00 2001
From: Nikolay Samokhvalov <nik@postgres.ai>
Date: Tue, 18 Mar 2025 20:49:57 -0700
Subject: [PATCH 2/5] Edit
 0095_single_session_query_regression_performance_testing.md

---
 ...ingle_session_query_regression_performance_testing.md | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/0095_single_session_query_regression_performance_testing.md b/0095_single_session_query_regression_performance_testing.md
index 22514cd..78dd8fc 100644
--- a/0095_single_session_query_regression_performance_testing.md
+++ b/0095_single_session_query_regression_performance_testing.md
@@ -42,7 +42,7 @@ For Postgres major upgrades, we can follow this methodology to perform query pla
 1. Join samples:
     - move samples to some machine with Postgres and load them for analysis
     - for newer Postgres versions (14+), use `query_id` (identifier of this backend's most recent query. [Details](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)) to JOIN
-    - for older, use [libpg_query](https://github.com/pganalyze/libpg_query) to calculate fingerprint for both pgss and pgsa samples, and then JOIN
+    - for older, use [libpg_query](https://github.com/pganalyze/libpg_query) to calculate fingerprint for both pgss and pgsa samples, and then JOIN // in newer PG14+, we can use `queryid`, if [`compute_query_id`](https://postgresqlco.nf/doc/en/param/compute_query_id/) is configured so `pg_stat_activity` also has it
     - optionally, leave only N samples for each pgss group
 1. Using DBLab, get two clones – older Postgres version and newer one (e.g. pg_14 and pg_16)
     - clones must be created based on the same snapshot, so the data in clones is identical
@@ -152,7 +152,6 @@ create table pgss_pgsa.pgsa_sampling (
 ```
 
 For Postgres 13 and earlier versions only:
-
 ```sql
 drop table if exists pgss_pgsa.pgss_fingerprint;
 create table pgss_pgsa.pgss_fingerprint (queryid text, query text, fingerprint text);
@@ -161,8 +160,7 @@ drop table if exists pgss_pgsa.pgsa_fingerprint;
 create table pgss_pgsa.pgsa_fingerprint (query text, fingerprint text);
 ```
 
-If production is running on Postgres 14 or higher, use `pgsa.query_id` / `pgss.queryid` to match samples:
-
+If production is running on Postgres 14 or higher, use `pgss.queryid` / `pgsa.query_id` (requires [`compute_query_id`](https://postgresqlco.nf/doc/en/param/compute_query_id/) configuration) to match samples:
 ```shell
 psql -XAtc "
   copy (
@@ -193,8 +191,6 @@ psql -XAtc "
 
 Otherwise, use `./query_fingerprint.rb` to calculate fingerprints:
 
-<details><summary>Click here to expand...</summary><p>
-
 ```shell
 ruby query_fingerprint.rb > query_fingerprint.log 2>&1
 ```
@@ -234,7 +230,6 @@ psql -XAtc "
     ) to stdout csv delimiter ',' \
 " > /tmp/query_samples.csv
 ```
-</p></details>
 
 ### Prepare two clones (for two major Postgres versions)
 Using DBLab, create two clones. Use the same snapshot for them.
-- 
GitLab


From cb0927b697a9d0962748f3902317f839cf9c1a6f Mon Sep 17 00:00:00 2001
From: Nikolay Samokhvalov <nik@postgres.ai>
Date: Tue, 18 Mar 2025 21:12:11 -0700
Subject: [PATCH 3/5] Update file
 0095_single_session_query_regression_performance_testing.md

---
 ...on_query_regression_performance_testing.md | 62 ++++++++++---------
 1 file changed, 33 insertions(+), 29 deletions(-)

diff --git a/0095_single_session_query_regression_performance_testing.md b/0095_single_session_query_regression_performance_testing.md
index 78dd8fc..778af16 100644
--- a/0095_single_session_query_regression_performance_testing.md
+++ b/0095_single_session_query_regression_performance_testing.md
@@ -1,61 +1,65 @@
-# Single-session query regression performance testing
+# Single-session query regression performance testing (a.k.a. "Query execution plan A/B testing")
 ## Definition and goals
-Goal: compare execution plans for 2 or more different situations (example: two different Postgres major versions), using just single connection. This type of benchmark is designed to study execution plans using `EXPLAIN (ANALYZE, BUFFERS)`. 
+Goal: compare execution plans for 2 or more different situations (example: two different Postgres major versions), using only a single connection. This type of benchmark is designed to study execution plans using `EXPLAIN (ANALYZE, BUFFERS)`. 
 
-This type of benchmark is not applicable for research of Postgres components behavior (e.g., buffer pool and contention in it, WAL writer or checkpointer activities).
+This type of benchmark does not apply to researching of Postgres components behavior (e.g., buffer pool and contention in it, WAL writer or checkpointer activities).
 
 ## When to use
-This approach can be used in varous cases. Some examples:
+This approach can be used in various cases. Some examples:
 - Major Postgres upgrades: compare behavior of planner and executor on old and new version
 - Migration to partitioned schema: compare execution plans and the key characteristics (first of all, `BUFFERS`) before and after partitioning
-- Reconfiguration of Postgres server: e.g., change of planner settings such as `random_page_cost`, or `work_mem`
+- Reconfiguration of Postgres server: e.g., change of the planner settings such as `random_page_cost`, or `work_mem`
 
 ## Shared environments and hardware differences
-Single-session benchmark can be executed on machines that:
+A single-session benchmark can be executed on machines that:
 - have weaker resources compared to production (e.g. less RAM or weaker disks)
 - different filesystem (e.g. ZFS with CoW support vs. ext4)
 - shared environments with significant risks of saturation
 
+The key idea here is that the difference in physical resources is generally doesn't matter (though, you can take it into account – then you can pay more attention to the timing metrics). The research targets the "upper" components of architecture – the planner, the executor. The physical characteristics affect timing metrics, but they don't affect the plan structure and volumes of data / IO-related metrics (rows, BUFFERS) in general. Thus:
+- if we know there is a big difference in underlying physical resources (e.g. beefy RDS instance vs. smaller DBLab machine), we keep it in mind and pay less attention to timing metrics and focus on plan structure comparison and BUFFERS (!)
+- at the same time, if we know there is now difference in physical resources, it makes all sense to look at the timing metrics too; though, it is still recommended to use the plan structures and BUFFERS metrics as primary comparison tools
+
 ## Principles
-However, to allow such differences, we must follow these rules:
-1. **Planner settings** (`select * from pg_settings where category ~ '^Query Tuning'`) and `work_mem` should the the same as on production machines – this is needed to make Postgres planner behave the same as on production.
-    - For example, we can use a very low `shared_buffers` and the very small RAM, but we do need to set `effective_cache_size` as on production – interesting that:
-        - Postgres planner doesn't know anything about actual resources the machine has – CPU, RAM, disk
-        - it doesn't take into account `shared_buffers`, this parameter doesn't affect planner's decisions
-        - but we do need to use the proper `effective_cache_size`, and we can use any value, even much bigger than actual RAM – this value is going to define the planner's behavior (as well as, for example, `work_mem`)
-
-Note: SQL [query](https://postgres.ai/docs/how-to-guides/administration/postgresql-configuration#postgresql-configuration-in-clones) to get all non-default values of the parameters affecting the planner's behavior.
+However, to account for such differences, we must follow these rules:
+1. **Planner settings** (`select * from pg_settings where category ~ '^Query Tuning'`) and `work_mem` should be the same as on production machines – this is needed to make Postgres planner behave the same as on production.
+    - For example, we can use a very low `shared_buffers` and the very small RAM, but we do need to set `effective_cache_size` as on production – interestingly:
+        - Postgres planner is unaware of the machine’s actual resources – CPU, RAM, disk
+        - it doesn't take into account `shared_buffers`, this parameter does not affect the planner's decisions
+        - but we do need to use the proper `effective_cache_size`, and it can be set to any value, even larger than actual RAM, as it defines the planner's behavior
+        - `work_mem`, not being among "Query Tuning" GUC parameters, also affects the planner behavior, so it needs to be adjusted as well
+Note: Use SQL [query](https://postgres.ai/docs/how-to-guides/administration/postgresql-configuration#postgresql-configuration-in-clones) to get all non-default values of the parameters affecting the planner's behavior.
 2. **A clone of production database** has to be used.
-    - Physical level (a clone of PGDATA) is more preferrable than logical one (dump/restore) since it not only gives the same row numbers for tables, but also physical distribution, bloat, `pg_class.relpages`, the changes to have identical plans as on production are very high.
-3. **BUFFERS first, timing second**: understanding that we have hardware differences, different (compared to production) state of caches (Postgres buffer pool and page cache), different filesystem (e.g. ZFS in case of DBLab), and risks of saturated resources since we workin a shared environment and other people or CI/CD pipelines might use a lot of resources, we need to pay less attention to timing metrics, which are very volatile, and prefer using IO and data volume metrics – buffer numbers, rows – as well as plan structure and costs.
+    - Physical level (a clone of PGDATA) is more preferable than logical one (dump/restore) since it not only gives the same row numbers for tables, but also physical distribution, bloat, and `pg_class.relpages`, increasing the likelihood of identical plans as in production.
+3. **BUFFERS first, timing second**: understanding that we have hardware differences, different (compared to production) state of caches (Postgres buffer pool and page cache), different filesystem (e.g. ZFS in case of DBLab), and the risk of resource saturation in a shared environment where other users or CI/CD pipelines may be consuming significant resources, we need to pay less attention to timing metrics, which are very volatile, and prefer using IO and data volume metrics – buffer numbers, rows – as well as plan structure and costs.
 
 ## "pgss_pgsa" – single-session benchmarks for Postgres major upgrades
 For Postgres major upgrades, we can follow this methodology to perform query plan regression analysis:
 1. **pgss samples**: get query groups (normalized queries) from `pg_stat_statements` from production
     - Top-N by `total_exec_time` (most important in terms of resource utilization)
     - Top-N by `mean_exec_time` (most important in terms of UX)
-    - Top-N by `calls` (degradation of high-frequent queries can be very noticeable)
+    - Top-N by `calls` (degradation of high-frequency queries can be very noticeable)
     - optionally, more – e.g., `total_plan_time`
     - optionally, not only from primary but also from standbys
 1. **pgsa samples**:  sample `pg_stat_activity` during substantial period of time to collect samples of queries
-    - `track_activity_query_size` should be increased (default: 1024, which is usually not enough; unfortunately to change it, a restart is required)
-1. Join samples:
-    - move samples to some machine with Postgres and load them for analysis
-    - for newer Postgres versions (14+), use `query_id` (identifier of this backend's most recent query. [Details](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)) to JOIN
-    - for older, use [libpg_query](https://github.com/pganalyze/libpg_query) to calculate fingerprint for both pgss and pgsa samples, and then JOIN // in newer PG14+, we can use `queryid`, if [`compute_query_id`](https://postgresqlco.nf/doc/en/param/compute_query_id/) is configured so `pg_stat_activity` also has it
+    - `track_activity_query_size` should be increased (default: `1024`, which is usually not enough; unfortunately to change it, a restart is required)
+1. Join the samples:
+    - move the samples to a machine with Postgres and load them for analysis
+    - for newer Postgres versions (14+), use `query_id` (identifier of the backend’s most recent query; [details](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)) to `JOIN`
+    - for older versions (PG13 and older), use [libpg_query](https://github.com/pganalyze/libpg_query) to calculate fingerprint for both pgss and pgsa samples, and then JOIN // in newer PG14+, we can use `queryid`, if [`compute_query_id`](https://postgresqlco.nf/doc/en/param/compute_query_id/) is configured so `pg_stat_activity` also has it
     - optionally, leave only N samples for each pgss group
-1. Using DBLab, get two clones – older Postgres version and newer one (e.g. pg_14 and pg_16)
-    - clones must be created based on the same snapshot, so the data in clones is identical
+1. Using DBLab, get two clones – one with an older Postgres version and one with a newer version (e.g., pg_14 and pg_16)
+    - clones must be created based on the same snapshot to ensure identical data
 1. Collect query execution plans using `EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT JSON)`
-    - 2-3 executions are recommended to collect plans with warmed-up caches
-1. Perform regression analysis of plans with focus on:
+    - 2-3 executions are recommended to gather plans with warmed-up caches
+1. Perform regression analysis of plans with a focus on:
     - plan structure
     - data volumes (`BUFFERS`)
     - costs
     - and, finally, timing – as the last metric
 
 ### Collect pgss and pgsa samples
-On production primary (and optionally, on standbys), collect pgss and pgsa samples:
+On production primary (and, optionally, on standbys), collect pgss and pgsa samples:
 
 ```shell
 export PGSS_LIMIT=20
@@ -387,7 +391,7 @@ Import plans for analysis
 
 Note: Replace `version_major` value with versions for old and new Postgres
 
-Dump result it as CSV:
+Dump the results it as a CSV:
 ```shell
 export PLANS_SQL="
   with plans_pg_old as (
@@ -457,7 +461,7 @@ export PLANS_SQL="
 psql -Xc "copy ($PLANS_SQL) to stdout with csv header delimiter ','" > /tmp/plans_pg_old_vs_pg_new.csv
 ```
 
-Upload to a spreadsheet, use conditional formatting to highliad improved and degraded queries.
+Upload to a spreadsheet, use conditional formatting to highlight improved and degraded queries.
 
 TODO:
 - analyze diffs
-- 
GitLab


From fc62c5d96f3186219a8743ba16fa45f21877c8ce Mon Sep 17 00:00:00 2001
From: Nikolay Samokhvalov <nik@postgres.ai>
Date: Wed, 19 Mar 2025 18:52:22 -0700
Subject: [PATCH 4/5] Update file
 0095_single_session_query_regression_performance_testing.md

---
 0095_single_session_query_regression_performance_testing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/0095_single_session_query_regression_performance_testing.md b/0095_single_session_query_regression_performance_testing.md
index 778af16..f9410a5 100644
--- a/0095_single_session_query_regression_performance_testing.md
+++ b/0095_single_session_query_regression_performance_testing.md
@@ -28,7 +28,7 @@ However, to account for such differences, we must follow these rules:
         - it doesn't take into account `shared_buffers`, this parameter does not affect the planner's decisions
         - but we do need to use the proper `effective_cache_size`, and it can be set to any value, even larger than actual RAM, as it defines the planner's behavior
         - `work_mem`, not being among "Query Tuning" GUC parameters, also affects the planner behavior, so it needs to be adjusted as well
-Note: Use SQL [query](https://postgres.ai/docs/how-to-guides/administration/postgresql-configuration#postgresql-configuration-in-clones) to get all non-default values of the parameters affecting the planner's behavior.
+    Note: Use SQL [query](https://postgres.ai/docs/how-to-guides/administration/postgresql-configuration#postgresql-configuration-in-clones) to get all non-default values of the parameters affecting the planner's behavior.
 2. **A clone of production database** has to be used.
     - Physical level (a clone of PGDATA) is more preferable than logical one (dump/restore) since it not only gives the same row numbers for tables, but also physical distribution, bloat, and `pg_class.relpages`, increasing the likelihood of identical plans as in production.
 3. **BUFFERS first, timing second**: understanding that we have hardware differences, different (compared to production) state of caches (Postgres buffer pool and page cache), different filesystem (e.g. ZFS in case of DBLab), and the risk of resource saturation in a shared environment where other users or CI/CD pipelines may be consuming significant resources, we need to pay less attention to timing metrics, which are very volatile, and prefer using IO and data volume metrics – buffer numbers, rows – as well as plan structure and costs.
-- 
GitLab


From 9316952a99a5af2a4e2f31c6d046724fc22d226d Mon Sep 17 00:00:00 2001
From: Nikolay Samokhvalov <nik@postgres.ai>
Date: Wed, 19 Mar 2025 18:53:32 -0700
Subject: [PATCH 5/5] Update file
 0095_single_session_query_regression_performance_testing.md

---
 0095_single_session_query_regression_performance_testing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/0095_single_session_query_regression_performance_testing.md b/0095_single_session_query_regression_performance_testing.md
index f9410a5..a6b42d2 100644
--- a/0095_single_session_query_regression_performance_testing.md
+++ b/0095_single_session_query_regression_performance_testing.md
@@ -18,7 +18,7 @@ A single-session benchmark can be executed on machines that:
 
 The key idea here is that the difference in physical resources is generally doesn't matter (though, you can take it into account – then you can pay more attention to the timing metrics). The research targets the "upper" components of architecture – the planner, the executor. The physical characteristics affect timing metrics, but they don't affect the plan structure and volumes of data / IO-related metrics (rows, BUFFERS) in general. Thus:
 - if we know there is a big difference in underlying physical resources (e.g. beefy RDS instance vs. smaller DBLab machine), we keep it in mind and pay less attention to timing metrics and focus on plan structure comparison and BUFFERS (!)
-- at the same time, if we know there is now difference in physical resources, it makes all sense to look at the timing metrics too; though, it is still recommended to use the plan structures and BUFFERS metrics as primary comparison tools
+- at the same time, if we know there is a small difference in physical resources, it makes all sense to look at the timing metrics too; though, it is still recommended to use the plan structures and BUFFERS metrics as primary comparison tools
 
 ## Principles
 However, to account for such differences, we must follow these rules:
-- 
GitLab