Replace "amgetmulti" AM functions with "amgetbitmap", in which the whole

indexscan always occurs in one call, and the results are returned in a
TIDBitmap instead of a limited-size array of TIDs.  This should improve
speed a little by reducing AM entry/exit overhead, and it is necessary
infrastructure if we are ever to support bitmap indexes.

In an only slightly related change, add support for TIDBitmaps to preserve
(somewhat lossily) the knowledge that particular TIDs reported by an index
need to have their quals rechecked when the heap is visited.  This facility
is not really used yet; we'll need to extend the forced-recheck feature to
plain indexscans before it's useful, and that hasn't been coded yet.
The intent is to use it to clean up 8.3's horrid @@@ kluge for text search
with weighted queries.  There might be other uses in future, but that one
alone is sufficient reason.

Heikki Linnakangas, with some adjustments by me.

Author: tglsfdc

doc/src/sgml/catalogs.sgml

@@ -1,4 +1,4 @@
-<!-- $PostgreSQL: pgsql/doc/src/sgml/catalogs.sgml,v 2.163 2008/03/10 12:55:13 mha Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/catalogs.sgml,v 2.164 2008/04/10 22:25:25 tgl Exp $ -->
 <!--
  Documentation of the system catalogs, directed toward PostgreSQL developers
  -->
@@ -473,10 +473,10 @@
      </row>
 
      <row>
-      <entry><structfield>amgetmulti</structfield></entry>
+      <entry><structfield>amgetbitmap</structfield></entry>
       <entry><type>regproc</type></entry>
       <entry><literal><link linkend="catalog-pg-proc"><structname>pg_proc</structname></link>.oid</literal></entry>
-      <entry><quote>Fetch multiple tuples</quote> function</entry>
+      <entry><quote>Fetch all valid tuples</quote> function</entry>
      </row>
 
      <row>

doc/src/sgml/indexam.sgml

@@ -1,4 +1,4 @@
-<!-- $PostgreSQL: pgsql/doc/src/sgml/indexam.sgml,v 2.23 2007/04/06 22:33:41 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/indexam.sgml,v 2.24 2008/04/10 22:25:25 tgl Exp $ -->
 
 <chapter id="indexam">
  <title>Index Access Method Interface Definition</title>
@@ -320,23 +320,16 @@ amgettuple (IndexScanDesc scan,
 
   <para>
 <programlisting>
-boolean
-amgetmulti (IndexScanDesc scan,
-            ItemPointer tids,
-            int32 max_tids,
-            int32 *returned_tids);
+int64
+amgetbitmap (IndexScanDesc scan,
+             TIDBitmap *tbm);
 </programlisting>
-   Fetch multiple tuples in the given scan.  Returns TRUE if the scan should
-   continue, FALSE if no matching tuples remain.  <literal>tids</> points to
-   a caller-supplied array of <literal>max_tids</>
-   <structname>ItemPointerData</> records, which the call fills with TIDs of
-   matching tuples.  <literal>*returned_tids</> is set to the number of TIDs
-   actually returned.  This can be less than <literal>max_tids</>, or even
-   zero, even when the return value is TRUE.  (This provision allows the
-   access method to choose the most efficient stopping points in its scan,
-   for example index page boundaries.)  <function>amgetmulti</> and
+   Fetch all tuples in the given scan and add them to the caller-supplied
+   TIDBitmap (that is, OR the set of tuple IDs into whatever set is already
+   in the bitmap).  The number of tuples fetched is returned. 
+   <function>amgetbitmap</> and
    <function>amgettuple</> cannot be used in the same index scan; there
-   are other restrictions too when using <function>amgetmulti</>, as explained
+   are other restrictions too when using <function>amgetbitmap</>, as explained
    in <xref linkend="index-scanning">.
   </para>
 
@@ -491,20 +484,17 @@ amrestrpos (IndexScanDesc scan);
 
   <para>
    Instead of using <function>amgettuple</>, an index scan can be done with 
-   <function>amgetmulti</> to fetch multiple tuples per call.  This can be
+   <function>amgetbitmap</> to fetch all tuples in one call.  This can be
    noticeably more efficient than <function>amgettuple</> because it allows
    avoiding lock/unlock cycles within the access method.  In principle
-   <function>amgetmulti</> should have the same effects as repeated
+   <function>amgetbitmap</> should have the same effects as repeated
    <function>amgettuple</> calls, but we impose several restrictions to
-   simplify matters.  In the first place, <function>amgetmulti</> does not
-   take a <literal>direction</> argument, and therefore it does not support
-   backwards scan nor intrascan reversal of direction.  The access method
-   need not support marking or restoring scan positions during an
-   <function>amgetmulti</> scan, either.  (These restrictions cost little
-   since it would be difficult to use these features in an
-   <function>amgetmulti</> scan anyway: adjusting the caller's buffered
-   list of TIDs would be complex.)  Finally, <function>amgetmulti</> does
-   not guarantee any locking of the returned tuples, with implications
+   simplify matters.  First of all, <function>amgetbitmap</> returns all 
+   tuples at once and marking or restoring scan positions isn't 
+   supported. Secondly, the tuples are returned in a bitmap which doesn't
+   have any specific ordering, which is why <function>amgetbitmap</> doesn't
+   take a <literal>direction</> argument.  Finally, <function>amgetbitmap</>
+   does not guarantee any locking of the returned tuples, with implications
    spelled out in <xref linkend="index-locking">.
   </para>
 
@@ -605,9 +595,8 @@ amrestrpos (IndexScanDesc scan);
   </para>
 
   <para>
-   In an <function>amgetmulti</> index scan, the access method need not
-   guarantee to keep an index pin on any of the returned tuples.  (It would be
-   impractical to pin more than the last one anyway.)  Therefore
+   In an <function>amgetbitmap</> index scan, the access method does not
+   keep an index pin on any of the returned tuples.  Therefore
    it is only safe to use such scans with MVCC-compliant snapshots.
   </para>
 

src/backend/access/gin/ginget.c

@@ -8,13 +8,15 @@
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * IDENTIFICATION
- *			$PostgreSQL: pgsql/src/backend/access/gin/ginget.c,v 1.10 2008/01/01 19:45:46 momjian Exp $
+ *			$PostgreSQL: pgsql/src/backend/access/gin/ginget.c,v 1.11 2008/04/10 22:25:25 tgl Exp $
  *-------------------------------------------------------------------------
  */
 
 #include "postgres.h"
+
 #include "access/gin.h"
 #include "catalog/index.h"
+#include "miscadmin.h"
 #include "utils/memutils.h"
 
 static bool
@@ -476,34 +478,37 @@ scanGetItem(IndexScanDesc scan, ItemPointerData *item)
 #define GinIsVoidRes(s)		( ((GinScanOpaque) scan->opaque)->isVoidRes == true )
 
 Datum
-gingetmulti(PG_FUNCTION_ARGS)
+gingetbitmap(PG_FUNCTION_ARGS)
 {
 	IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
-	ItemPointer tids = (ItemPointer) PG_GETARG_POINTER(1);
-	int32		max_tids = PG_GETARG_INT32(2);
-	int32	   *returned_tids = (int32 *) PG_GETARG_POINTER(3);
+	TIDBitmap *tbm = (TIDBitmap *) PG_GETARG_POINTER(1);
+	int64		ntids;
 
 	if (GinIsNewKey(scan))
 		newScanKey(scan);
 
-	*returned_tids = 0;
-
 	if (GinIsVoidRes(scan))
-		PG_RETURN_BOOL(false);
+		PG_RETURN_INT64(0);
 
 	startScan(scan);
 
-	do
+	ntids = 0;
+	for (;;)
 	{
-		if (scanGetItem(scan, tids + *returned_tids))
-			(*returned_tids)++;
-		else
+		ItemPointerData iptr;
+
+		CHECK_FOR_INTERRUPTS();
+
+		if (!scanGetItem(scan, &iptr))
 			break;
-	} while (*returned_tids < max_tids);
+
+		tbm_add_tuples(tbm, &iptr, 1, false);
+		ntids++;
+	}
 
 	stopScan(scan);
 
-	PG_RETURN_BOOL(*returned_tids == max_tids);
+	PG_RETURN_INT64(ntids);
 }
 
 Datum

src/backend/access/gist/gistget.c

@@ -8,21 +8,24 @@
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/access/gist/gistget.c,v 1.69 2008/01/01 19:45:46 momjian Exp $
+ *	  $PostgreSQL: pgsql/src/backend/access/gist/gistget.c,v 1.70 2008/04/10 22:25:25 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
 #include "postgres.h"
 
 #include "access/gist_private.h"
 #include "executor/execdebug.h"
+#include "miscadmin.h"
 #include "pgstat.h"
 #include "utils/memutils.h"
 
 
 static OffsetNumber gistfindnext(IndexScanDesc scan, OffsetNumber n,
 			 ScanDirection dir);
-static int	gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids, int maxtids, bool ignore_killed_tuples);
+static int64 gistnext(IndexScanDesc scan, ScanDirection dir,
+					  ItemPointer tid, TIDBitmap *tbm,
+					  bool ignore_killed_tuples);
 static bool gistindex_keytest(IndexTuple tuple, IndexScanDesc scan,
 				  OffsetNumber offset);
 
@@ -114,32 +117,37 @@ gistgettuple(PG_FUNCTION_ARGS)
 	 * tuples, continue looping until we find a non-killed tuple that matches
 	 * the search key.
 	 */
-	res = (gistnext(scan, dir, &tid, 1, scan->ignore_killed_tuples)) ? true : false;
+	res = (gistnext(scan, dir, &tid, NULL, scan->ignore_killed_tuples) > 0) ? true : false;
 
 	PG_RETURN_BOOL(res);
 }
 
 Datum
-gistgetmulti(PG_FUNCTION_ARGS)
+gistgetbitmap(PG_FUNCTION_ARGS)
 {
 	IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
-	ItemPointer tids = (ItemPointer) PG_GETARG_POINTER(1);
-	int32		max_tids = PG_GETARG_INT32(2);
-	int32	   *returned_tids = (int32 *) PG_GETARG_POINTER(3);
+	TIDBitmap *tbm = (TIDBitmap *) PG_GETARG_POINTER(1);
+	int64	   ntids;
 
-	*returned_tids = gistnext(scan, ForwardScanDirection, tids, max_tids, false);
+	ntids = gistnext(scan, ForwardScanDirection, NULL, tbm, false);
 
-	PG_RETURN_BOOL(*returned_tids == max_tids);
+	PG_RETURN_INT64(ntids);
 }
 
 /*
- * Fetch a tuples that matchs the search key; this can be invoked
- * either to fetch the first such tuple or subsequent matching
- * tuples. Returns true iff a matching tuple was found.
+ * Fetch tuple(s) that match the search key; this can be invoked
+ * either to fetch the first such tuple or subsequent matching tuples.
+ *
+ * This function is used by both gistgettuple and gistgetbitmap. When
+ * invoked from gistgettuple, tbm is null and the next matching tuple
+ * is returned in *tid. When invoked from getbitmap, tid is null and
+ * all matching tuples are added to tbm. In both cases, the function
+ * result is the number of returned tuples.
  */
-static int
-gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
-		 int maxtids, bool ignore_killed_tuples)
+static int64
+gistnext(IndexScanDesc scan, ScanDirection dir,
+		 ItemPointer tid, TIDBitmap *tbm,
+		 bool ignore_killed_tuples)
 {
 	Page		p;
 	OffsetNumber n;
@@ -148,7 +156,7 @@ gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
 	IndexTuple	it;
 	GISTPageOpaque opaque;
 	bool		resetoffset = false;
-	int			ntids = 0;
+	int64		ntids = 0;
 
 	so = (GISTScanOpaque) scan->opaque;
 
@@ -174,6 +182,8 @@ gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
 
 	for (;;)
 	{
+		CHECK_FOR_INTERRUPTS();
+
 		/* First of all, we need lock buffer */
 		Assert(so->curbuf != InvalidBuffer);
 		LockBuffer(so->curbuf, GIST_SHARE);
@@ -285,20 +295,21 @@ gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
 				 * return success. Note that we keep "curbuf" pinned so that
 				 * we can efficiently resume the index scan later.
 				 */
-
 				ItemPointerSet(&(so->curpos),
 							   BufferGetBlockNumber(so->curbuf), n);
 
 				if (!(ignore_killed_tuples && ItemIdIsDead(PageGetItemId(p, n))))
 				{
 					it = (IndexTuple) PageGetItem(p, PageGetItemId(p, n));
-					tids[ntids] = scan->xs_ctup.t_self = it->t_tid;
 					ntids++;
-
-					if (ntids == maxtids)
+					if (tbm != NULL)
+						tbm_add_tuples(tbm, &it->t_tid, 1, false);
+					else 
 					{
+						*tid = scan->xs_ctup.t_self = it->t_tid;
+
 						LockBuffer(so->curbuf, GIST_UNLOCK);
-						return ntids;
+						return ntids; /* always 1 */
 					}
 				}
 			}
@@ -308,7 +319,6 @@ gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
 				 * We've found an entry in an internal node whose key is
 				 * consistent with the search key, so push it to stack
 				 */
-
 				stk = (GISTSearchStack *) palloc(sizeof(GISTSearchStack));
 
 				it = (IndexTuple) PageGetItem(p, PageGetItemId(p, n));
@@ -318,7 +328,6 @@ gistnext(IndexScanDesc scan, ScanDirection dir, ItemPointer tids,
 
 				stk->next = so->stack->next;
 				so->stack->next = stk;
-
 			}
 
 			if (ScanDirectionIsBackward(dir))