Map basebackup tablespaces using a tablespace_map file

Windows can't reliably restore symbolic links from a tar format, so
instead during backup start we create a tablespace_map file, which is
used by the restoring postgres to create the correct links in pg_tblspc.
The backup protocol also now has an option to request this file to be
included in the backup stream, and this is used by pg_basebackup when
operating in tar mode.

This is done on all platforms, not just Windows.

This means that pg_basebackup will not not work in tar mode against 9.4
and older servers, as this protocol option isn't implemented there.

Amit Kapila, reviewed by Dilip Kumar, with a little editing from me.

Author: adunstan

doc/src/sgml/backup.sgml

@@ -836,8 +836,11 @@ SELECT pg_start_backup('label');
      <function>pg_start_backup</> creates a <firstterm>backup label</> file,
      called <filename>backup_label</>, in the cluster directory with
      information about your backup, including the start time and label
-     string.  The file is critical to the integrity of the backup, should
-     you need to restore from it.
+     string.  The function also creates a <firstterm>tablespace map</> file,
+     called <filename>tablespace_map</>, in the cluster directory with
+     information about tablespace symbolic links in <filename>pg_tblspc/</>
+     if one or more such link is present.  Both files are critical to the
+     integrity of the backup, should you need to restore from it.
     </para>
 
     <para>
@@ -965,17 +968,20 @@ SELECT pg_stop_backup();
 
    <para>
     It's also worth noting that the <function>pg_start_backup</> function
-    makes a file named <filename>backup_label</> in the database cluster
-    directory, which is removed by <function>pg_stop_backup</>.
-    This file will of course be archived as a part of your backup dump file.
-    The backup label file includes the label string you gave to
-    <function>pg_start_backup</>, as well as the time at which
-    <function>pg_start_backup</> was run, and the name of the starting WAL
-    file.  In case of confusion it is therefore possible to look inside a
-    backup dump file and determine exactly which backup session the dump file
-    came from.  However, this file is not merely for your information; its
-    presence and contents are critical to the proper operation of the system's
-    recovery process.
+    makes files named <filename>backup_label</> and
+    <filename>tablesapce_map</> in the database cluster directory,
+    which are removed by <function>pg_stop_backup</>.  These files will of
+    course be archived as a part of your backup dump file.  The backup label
+    file includes the label string you gave to <function>pg_start_backup</>,
+    as well as the time at which <function>pg_start_backup</> was run, and
+    the name of the starting WAL file.  In case of confusion it is therefore
+    possible to look inside a backup dump file and determine exactly which
+    backup session the dump file came from.  The tablespace map file includes
+    the symbolic link names as they exist in the directory
+    <filename>pg_tblspc/</> and the full path of each symbolic link.
+    These files are not merely for your information; their presence and
+    contents are critical to the proper operation of the system's recovery
+    process.
    </para>
 
    <para>

doc/src/sgml/func.sgml

@@ -16591,11 +16591,12 @@ SELECT set_config('log_statement_stats', 'off', false);
     <function>pg_start_backup</> accepts an
     arbitrary user-defined label for the backup.  (Typically this would be
     the name under which the backup dump file will be stored.)  The function
-    writes a backup label file (<filename>backup_label</>) into the
-    database cluster's data directory, performs a checkpoint,
-    and then returns the backup's starting transaction log location as text.
-    The user can ignore this result value, but it is
-    provided in case it is useful.
+    writes a backup label file (<filename>backup_label</>) and, if there
+    are any links in the <filename>pg_tblspc/</> directory, a tablespace map
+    file (<filename>tablespace_map</>) into the database cluster's data
+    directory, performs a checkpoint, and then returns the backup's starting
+    transaction log location as text.  The user can ignore this result value,
+    but it is provided in case it is useful.
 <programlisting>
 postgres=# select pg_start_backup('label_goes_here');
  pg_start_backup
@@ -16610,7 +16611,8 @@ postgres=# select pg_start_backup('label_goes_here');
    </para>
 
    <para>
-    <function>pg_stop_backup</> removes the label file created by
+    <function>pg_stop_backup</> removes the label file and, if it exists,
+    the <filename>tablespace_map</> file created by
     <function>pg_start_backup</>, and creates a backup history file in
     the transaction log archive area.  The history file includes the label given to
     <function>pg_start_backup</>, the starting and ending transaction log locations for

doc/src/sgml/protocol.sgml

@@ -1882,7 +1882,7 @@ The commands accepted in walsender mode are:
   </varlistentry>
 
   <varlistentry>
-    <term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>] [<literal>WAL</literal>] [<literal>NOWAIT</literal>] [<literal>MAX_RATE</literal> <replaceable>rate</replaceable>]
+    <term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>] [<literal>WAL</literal>] [<literal>NOWAIT</literal>] [<literal>MAX_RATE</literal> <replaceable>rate</replaceable>] [<literal>TABLESPACE_MAP</literal>]
      <indexterm><primary>BASE_BACKUP</primary></indexterm>
     </term>
     <listitem>
@@ -1968,6 +1968,19 @@ The commands accepted in walsender mode are:
          </para>
         </listitem>
        </varlistentry>
+
+       <varlistentry>
+        <term><literal>TABLESPACE_MAP</literal></term>
+        <listitem>
+         <para>
+          Include information about symbolic links present in the directory
+          <filename>pg_tblspc</filename> in a file named
+          <filename>tablespace_map</filename>. The tablespace map file includes
+          each symbolic link name as it exists in the directory
+          <filename>pg_tblspc/</> and the full path of that symbolic link.
+         </para>
+        </listitem>
+       </varlistentry>
       </variablelist>
      </para>
      <para>

doc/src/sgml/ref/pg_basebackup.sgml

@@ -587,11 +587,23 @@ PostgreSQL documentation
    tablespaces.
   </para>
 
+  <para>
+   When tar format mode is used, it is the user's responsibility to unpack each
+   tar file before starting postgres. If there are additional tablespaces, the
+   tar files for them need to be unpacked in the correct locations. In this
+   case the symbolic links for those tablespaces will be created by Postgres
+   according to the contents of the <filename>tablespace_map</> file that is
+   included in the <filename>base.tar</> file.
+  </para>
+
   <para>
    <application>pg_basebackup</application> works with servers of the same
    or an older major version, down to 9.1. However, WAL streaming mode (-X
-   stream) only works with server version 9.3 and later.
+   stream) only works with server version 9.3 and later, and tar format mode
+   (--format=tar) of the current version only works with server version 9.5
+   or later.
   </para>
+
  </refsect1>
 
  <refsect1>