Allow opclasses to provide tri-valued GIN consistent functions.

With the GIN "fast scan" feature, GIN can skip items without fetching all
the keys for them, if it can prove that they don't match regardless of
those keys. So far, it has done the proving by calling the boolean
consistent function with all combinations of TRUE/FALSE for the unfetched
keys, but since that's O(n^2), it becomes unfeasible with more than a few
keys. We can avoid calling consistent with all the combinations, if we can
tell the operator class implementation directly which keys are unknown.

This commit includes a triConsistent function for the built-in array and
tsvector opclasses.

Alexander Korotkov, with some changes by me.

Author: hlinnaka

doc/src/sgml/gin.sgml

@@ -74,15 +74,15 @@
 
  <para>
    All it takes to get a <acronym>GIN</acronym> access method working is to
-   implement four (or five) user-defined methods, which define the behavior of
+   implement a few user-defined methods, which define the behavior of
    keys in the tree and the relationships between keys, indexed items,
    and indexable queries. In short, <acronym>GIN</acronym> combines
    extensibility with generality, code reuse, and a clean interface.
  </para>
 
  <para>
-   The four methods that an operator class for
-   <acronym>GIN</acronym> must provide are:
+   There are three methods that an operator class for
+   <acronym>GIN</acronym> must provide:
 
  <variablelist>
     <varlistentry>
@@ -190,7 +190,18 @@
 
      </listitem>
     </varlistentry>
+  </variablelist>
+
+  An operator class must also provide a function to check if an indexed item
+  matches the query. It comes in two flavors, a boolean <function>consistent</>
+  function, and a ternary <function>triConsistent</> function.
+  <function>triConsistent</> covers the functionality of both, so providing
+  triConsistent alone is sufficient. However, if the boolean variant is
+  significantly cheaper to calculate, it can be advantegous to provide both.
+  If only the boolean variant is provided, some optimizations that depend on
+  refuting index items before fetching all the keys are disabled. 
 
+  <variablelist>
     <varlistentry>
      <term><function>bool consistent(bool check[], StrategyNumber n, Datum query,
         int32 nkeys, Pointer extra_data[], bool *recheck,
@@ -241,10 +252,38 @@
       </para>
      </listitem>
     </varlistentry>
+
+    <varlistentry>
+     <term><function>GinLogicValue triConsistent(GinLogicValue check[], StrategyNumber n, Datum query,
+        int32 nkeys, Pointer extra_data[],
+        Datum queryKeys[], bool nullFlags[])</></term>
+     <listitem>
+      <para>
+       <function>triConsistent</> is similar to <function>consistent</>,
+       but instead of a boolean <literal>check[]</>, there are three possible
+       values for each key: <literal>GIN_TRUE</>, <literal>GIN_FALSE</> and
+       <literal>GIN_MAYBE</>. <literal>GIN_FALSE</> and <literal>GIN_TRUE</>
+       have the same meaning as regular boolean values.
+       <literal>GIN_MAYBE</> means that the presence of that key is not known.
+       When <literal>GIN_MAYBE</> values are present, the function should only
+       return GIN_TRUE if the item matches whether or not the index item
+       contains the corresponding query keys. Likewise, the function must
+       return GIN_FALSE only if the item does not match, whether or not it 
+       contains the GIN_MAYBE keys. If the result depends on the GIN_MAYBE
+       entries, ie. the match cannot be confirmed or refuted based on the
+       known query keys, the function must return GIN_MAYBE.
+      </para>
+      <para>
+       When there are no GIN_MAYBE values in the <literal>check</> vector,
+       <literal>GIN_MAYBE</> return value is equivalent of setting
+       <literal>recheck</> flag in the boolean <function>consistent</> function.
+      </para>
+     </listitem>
+    </varlistentry>
   </variablelist>
 
-  Optionally, an operator class for
-  <acronym>GIN</acronym> can supply a fifth method:
+  Optionally, an operator class for <acronym>GIN</acronym> can supply the
+  following method:
 
   <variablelist>
     <varlistentry>
@@ -282,8 +321,9 @@
   above vary depending on the operator class.  The item values passed to
   <function>extractValue</> are always of the operator class's input type, and
   all key values must be of the class's <literal>STORAGE</> type.  The type of
-  the <literal>query</> argument passed to <function>extractQuery</> and
-  <function>consistent</> is whatever is specified as the right-hand input
+  the <literal>query</> argument passed to <function>extractQuery</>,
+  <function>consistent</> and <function>triConsistent</> is whatever is
+  specified as the right-hand input
   type of the class member operator identified by the strategy number.
   This need not be the same as the item type, so long as key values of the
   correct type can be extracted from it.

doc/src/sgml/xindex.sgml

@@ -567,7 +567,10 @@
       </row>
       <row>
        <entry><function>consistent</></entry>
-       <entry>determine whether value matches query condition</entry>
+       <entry>
+        determine whether value matches query condition (boolean variant)
+        (optional if support function 6 is present)
+       </entry>
        <entry>4</entry>
       </row>
       <row>
@@ -580,6 +583,14 @@
        </entry>
        <entry>5</entry>
       </row>
+      <row>
+       <entry><function>triConsistent</></entry>
+       <entry>
+        determine whether value matches query condition (ternary variant)
+        (optional if support function 4 is present)
+       </entry>
+       <entry>6</entry>
+      </row>
      </tbody>
     </tgroup>
    </table>

src/backend/access/gin/ginarrayproc.c

@@ -218,3 +218,87 @@ ginarrayconsistent(PG_FUNCTION_ARGS)
 
 	PG_RETURN_BOOL(res);
 }
+
+/*
+ * triconsistent support function
+ */
+Datum
+ginarraytriconsistent(PG_FUNCTION_ARGS)
+{
+	GinLogicValue *check = (GinLogicValue *) PG_GETARG_POINTER(0);
+	StrategyNumber strategy = PG_GETARG_UINT16(1);
+
+	/* ArrayType  *query = PG_GETARG_ARRAYTYPE_P(2); */
+	int32		nkeys = PG_GETARG_INT32(3);
+
+	/* Pointer	   *extra_data = (Pointer *) PG_GETARG_POINTER(4); */
+	/* Datum	   *queryKeys = (Datum *) PG_GETARG_POINTER(5); */
+	bool	   *nullFlags = (bool *) PG_GETARG_POINTER(6);
+	GinLogicValue	res;
+	int32		i;
+
+	switch (strategy)
+	{
+		case GinOverlapStrategy:
+			/* must have a match for at least one non-null element */
+			res = GIN_FALSE;
+			for (i = 0; i < nkeys; i++)
+			{
+				if (!nullFlags[i])
+				{
+					if (check[i] == GIN_TRUE)
+					{
+						res = GIN_TRUE;
+						break;
+					}
+					else if (check[i] == GIN_MAYBE && res == GIN_FALSE)
+					{
+						res = GIN_MAYBE;
+					}
+				}
+			}
+			break;
+		case GinContainsStrategy:
+			/* must have all elements in check[] true, and no nulls */
+			res = GIN_TRUE;
+			for (i = 0; i < nkeys; i++)
+			{
+				if (check[i] == GIN_FALSE || nullFlags[i])
+				{
+					res = GIN_FALSE;
+					break;
+				}
+				if (check[i] == GIN_MAYBE)
+				{
+					res = GIN_MAYBE;
+				}
+			}
+			break;
+		case GinContainedStrategy:
+			/* can't do anything else useful here */
+			res = GIN_MAYBE;
+			break;
+		case GinEqualStrategy:
+			/*
+			 * Must have all elements in check[] true; no discrimination
+			 * against nulls here.	This is because array_contain_compare and
+			 * array_eq handle nulls differently ...
+			 */
+			res = GIN_MAYBE;
+			for (i = 0; i < nkeys; i++)
+			{
+				if (check[i] == GIN_FALSE)
+				{
+					res = GIN_FALSE;
+					break;
+				}
+			}
+			break;
+		default:
+			elog(ERROR, "ginarrayconsistent: unknown strategy number: %d",
+				 strategy);
+			res = false;
+	}
+
+	PG_RETURN_GIN_LOGIC_VALUE(res);
+}

src/backend/access/gin/ginlogic.c

@@ -61,7 +61,7 @@ trueTriConsistentFn(GinScanKey key)
  * A helper function for calling a regular, binary logic, consistent function.
  */
 static bool
-normalBoolConsistentFn(GinScanKey key)
+directBoolConsistentFn(GinScanKey key)
 {
 	/*
 	 * Initialize recheckCurItem in case the consistentFn doesn't know it
@@ -81,6 +81,53 @@ normalBoolConsistentFn(GinScanKey key)
 									 PointerGetDatum(key->queryCategories)));
 }
 
+/*
+ * A helper function for calling a native ternary logic consistent function.
+ */
+static GinLogicValue
+directTriConsistentFn(GinScanKey key)
+{
+	return DatumGetGinLogicValue(FunctionCall7Coll(key->triConsistentFmgrInfo,
+										  key->collation,
+										  PointerGetDatum(key->entryRes),
+										  UInt16GetDatum(key->strategy),
+										  key->query,
+										  UInt32GetDatum(key->nuserentries),
+										  PointerGetDatum(key->extra_data),
+										  PointerGetDatum(key->queryValues),
+									 PointerGetDatum(key->queryCategories)));
+}
+
+/*
+ * This function implements a binary logic consistency check, using a ternary
+ * logic consistent function provided by the opclass. GIN_MAYBE return value
+ * is interpreted as true with recheck flag.
+ */
+static bool
+shimBoolConsistentFn(GinScanKey key)
+{
+	GinLogicValue result;
+	result = DatumGetGinLogicValue(FunctionCall7Coll(key->triConsistentFmgrInfo,
+										  key->collation,
+										  PointerGetDatum(key->entryRes),
+										  UInt16GetDatum(key->strategy),
+										  key->query,
+										  UInt32GetDatum(key->nuserentries),
+										  PointerGetDatum(key->extra_data),
+										  PointerGetDatum(key->queryValues),
+									 PointerGetDatum(key->queryCategories)));
+	if (result == GIN_MAYBE)
+	{
+		key->recheckCurItem = true;
+		return true;
+	}
+	else
+	{
+		key->recheckCurItem = false;
+		return result;
+	}
+}
+
 /*
  * This function implements a tri-state consistency check, using a boolean
  * consistent function provided by the opclass.
@@ -124,12 +171,12 @@ shimTriConsistentFn(GinScanKey key)
 	 * function as is.
 	 */
 	if (nmaybe == 0)
-		return normalBoolConsistentFn(key);
+		return directBoolConsistentFn(key);
 
 	/* First call consistent function with all the maybe-inputs set FALSE */
 	for (i = 0; i < nmaybe; i++)
 		key->entryRes[maybeEntries[i]] = GIN_FALSE;
-	curResult = normalBoolConsistentFn(key);
+	curResult = directBoolConsistentFn(key);
 
 	for (;;)
 	{
@@ -147,7 +194,7 @@ shimTriConsistentFn(GinScanKey key)
 		if (i == nmaybe)
 			break;
 
-		boolResult = normalBoolConsistentFn(key);
+		boolResult = directBoolConsistentFn(key);
 		recheck |= key->recheckCurItem;
 
 		if (curResult != boolResult)
@@ -175,8 +222,17 @@ ginInitConsistentFunction(GinState *ginstate, GinScanKey key)
 	else
 	{
 		key->consistentFmgrInfo = &ginstate->consistentFn[key->attnum - 1];
+		key->triConsistentFmgrInfo = &ginstate->triConsistentFn[key->attnum - 1];
 		key->collation = ginstate->supportCollation[key->attnum - 1];
-		key->boolConsistentFn = normalBoolConsistentFn;
-		key->triConsistentFn = shimTriConsistentFn;
+
+		if (OidIsValid(ginstate->consistentFn[key->attnum - 1].fn_oid))
+			key->boolConsistentFn = directBoolConsistentFn;
+		else
+			key->boolConsistentFn = shimBoolConsistentFn;
+
+		if (OidIsValid(ginstate->triConsistentFn[key->attnum - 1].fn_oid))
+			key->triConsistentFn =  directTriConsistentFn;
+		else
+			key->triConsistentFn =  shimTriConsistentFn;
 	}
 }

src/backend/access/gin/ginutil.c

@@ -67,9 +67,31 @@ initGinState(GinState *state, Relation index)
 		fmgr_info_copy(&(state->extractQueryFn[i]),
 					   index_getprocinfo(index, i + 1, GIN_EXTRACTQUERY_PROC),
 					   CurrentMemoryContext);
-		fmgr_info_copy(&(state->consistentFn[i]),
-					   index_getprocinfo(index, i + 1, GIN_CONSISTENT_PROC),
-					   CurrentMemoryContext);
+		/*
+		 * Check opclass capability to do tri-state or binary logic consistent
+		 * check.
+		 */
+		if (index_getprocid(index, i + 1, GIN_TRICONSISTENT_PROC) != InvalidOid)
+		{
+			fmgr_info_copy(&(state->triConsistentFn[i]),
+			   index_getprocinfo(index, i + 1, GIN_TRICONSISTENT_PROC),
+						   CurrentMemoryContext);
+		}
+
+		if (index_getprocid(index, i + 1, GIN_CONSISTENT_PROC) != InvalidOid)
+		{
+			fmgr_info_copy(&(state->consistentFn[i]),
+						   index_getprocinfo(index, i + 1, GIN_CONSISTENT_PROC),
+						   CurrentMemoryContext);
+		}
+
+		if (state->consistentFn[i].fn_oid == InvalidOid &&
+			state->triConsistentFn[i].fn_oid == InvalidOid)
+		{
+			elog(ERROR, "missing GIN support function (%d or %d) for attribute %d of index \"%s\"",
+				 GIN_CONSISTENT_PROC, GIN_TRICONSISTENT_PROC,
+				 i + 1, RelationGetRelationName(index));
+		}
 
 		/*
 		 * Check opclass capability to do partial match.