F.30. pg_plan_advice — help the planner get the right plan #

F.30.1. Getting Started
F.30.2. How It Works
F.30.3. Advice Targets
F.30.4. Advice Tags
F.30.5. Advice Feedback
F.30.6. Advice Collectors
F.30.7. Functions
F.30.8. Configuration Parameters
F.30.9. Author

The pg_plan_advice allows key planner decisions to be described, reproduced, and altered using a special-purpose "plan advice" mini-language. It is intended to allow stabilization of plan choices that the user believes to be good, as well as experimentation with plans that the planner believes to be non-optimal.

Note that, since the planner often makes good decisions, overriding its judgement can easily backfire. For example, if the distribution of the underlying data changes, the planner normally has the option to adjust the plan in an attempt to preserve good performance. If the plan advice prevents this, a very poor plan may be chosen. It is important to use plan advice only when risks of constraining the planner's choices are outweighed by the benefits.

F.30.1. Getting Started #

In order to use this module, the pg_plan_advice module must be loaded. You can do this on a system-wide basis by adding pg_plan_advice to shared_preload_libraries and restarting the server, or by adding it to session_preload_libraries and starting a new session, or by loading it into an individual session using the LOAD command. If you wish to use the collector interface, you must also the pg_plan_advice extension in the database where you wish to use the collector. Use the command CREATE EXTENSION pg_plan_advice to do this. If you do not wish to use the collector interface, this step is not required.

Once the pg_plan_advice module is loaded, EXPLAIN will support a PLAN_ADVICE option. You can use this option to see a plan advice string for the chosen plan. For example:

EXPLAIN (COSTS OFF, PLAN_ADVICE)
        SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id;
             QUERY PLAN
------------------------------------
 Hash Join
   Hash Cond: (f.dim_id = d.id)
   ->  Seq Scan on join_fact f
   ->  Hash
         ->  Seq Scan on join_dim d
 Generated Plan Advice:
   JOIN_ORDER(f d)
   HASH_JOIN(d)
   SEQ_SCAN(f d)
   NO_GATHER(f d)

In this example, the user has not specified any advice; instead, the planner has been permitted to make whatever decisions it thinks best, and those decisions are memorialized in the form of an advice string. JOIN_ORDER(f d) means that f should be the driving table, and the first table to which it should be joined is d. HASH_JOIN(d) means that d should appear on the inner side of a hash join. SEQ_SCAN(f d) means that both f and d should be accessed via a sequential scan. NO_GATHER(f d) means that neither f nor d should appear beneath a Gather or Gather Merge node. For more details on the plan advice mini-language, see the information on advice targets and advice tags, below.

If you want to see the advice strings for a large number of queries, or an entire workload, running EXPLAIN (PLAN_ADVICE) for each one may not be convenient. In such situations, it can be more convenient to use an advice collector.

Once you have an advice string for a query, you can use it to control how that query is planned. You can do this by setting pg_plan_advice.advice to the advice string you've chosen. This can be an advice string that was generated by the system, or one you've written yourself. One good way of creating your own advice string is to take the string generated by the system and pick out just those elements that you wish to enforce. In the example above, pg_plan_advice emits advice for the join order, the join method, the scan method, and the use of parallelism, but you might only want to control the join order:

SET pg_plan_advice.advice = 'JOIN_ORDER(f d)';
EXPLAIN (COSTS OFF)
        SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id;
             QUERY PLAN
------------------------------------
 Hash Join
   Hash Cond: (f.dim_id = d.id)
   ->  Seq Scan on join_fact f
   ->  Hash
         ->  Seq Scan on join_dim d
 Supplied Plan Advice:
   JOIN_ORDER(f d) /* matched */

Since the PLAN_ADVICE option to EXPLAIN was not specified, no advice string is generated for the plan. However, the supplied plan advice is still shown so that anyone looking at the EXPLAIN output knows that the chosen plan was influenced by plan advice. If information about supplied plan advice is not desired, it can be suppressed by configuring pg_plan_advice.always_explain_supplied_advice = false. For each piece of supplied advice, the output shows advice feedback indicating whether or not the advice was successfully applied to the query. In this case, the feedback says /* matched */, which means that f and d were found in the query and that the resulting query plan conforms to the specified advice.

F.30.2. How It Works #

Plan advice is written imperatively; that is, it specifies what should be done. However, at an implementation level, pg_plan_advice works by telling the core planner what should not be done. In other words, it operates by constraining the planner's choices, not by replacing it. Therefore, no matter what advice you provide, you will only ever get a plan that the core planner would have considered for the query in question. If you attempt to force what you believe to be the correct plan by supplying an advice string, and the planner still fails to produce the desired plan, this means that either there is a bug in your advice string, or the plan in question was not considered viable by the core planner. This commonly happens for one of two reasons. First, it might be the planner believes that the plan you're trying to force would be semantically incorrect -- that is, it would produce the wrong results -- and for that reason it wasn't considered. Second, it might be that the planner rejected the plan you were hoping to generate on some grounds other than cost. For example, given a very simple query such as SELECT * FROM some_table, the query planner will decide that the use of an index is worthless here before it performs any costing calculations. You cannot force it to use an index for this query even if you set enable_seqscan = false, and you can't force it to use an index using plan advice, either.

Specifying plan advice should never cause planner failure. However, if you specify plan advice that asks for something impossible, you may get a plan where some plan nodes are flagged as Disabled: true in the EXPLAIN output. In some cases, such plans will be basically the same plan you would have gotten with no supplied advice at all, but in other cases, they may be much worse. For example:

SET pg_plan_advice.advice = 'JOIN_ORDER(x f d)';
EXPLAIN (COSTS OFF)
        SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id;
                     QUERY PLAN
----------------------------------------------------
 Nested Loop
   Disabled: true
   ->  Seq Scan on join_fact f
   ->  Index Scan using join_dim_pkey on join_dim d
         Index Cond: (id = f.dim_id)
 Supplied Plan Advice:
   JOIN_ORDER(x f d) /* partially matched */

Because neither f nor d is the first table in the JOIN_ORDER() specification, the planner disables all direct joins between the two of them, thinking that the join to x should happen first. Since planning isn't allowed to fail, a disabled plan between the two tables is eventually selected anyway, but here it's a Nested Loop rather than the Hash Join that was chosen in the above example where no advice was specified. There are several different ways that this kind of thing can happen; when it does, the resulting plan is generally worse than if no advice had been specified at all. Therefore, it is a good idea to validate that the advice you specify applies to the query to which it is applied and that the results are as expected.

F.30.3. Advice Targets #

An advice target uniquely identifies a particular instance of a particular table involved in a particular query. In simple cases, such as the examples shown above, the advice target is simply the relation alias. However, a more complex syntax is required when subqueries are used, when tables are partitioned, or when the same relation alias is mentioned more than once in the same subquery (e.g. (foo JOIN bar ON foo.a = bar.a) x JOIN foo ON x.b = foo.b. Any combination of these three things can occur simultaneously: a relation could be mentioned more than once, be partitioned, and be used inside of a subquery.

Because of this, the general syntax for a relation identifier is:

alias_name#occurrence_number/partition_schema.partition_name@plan_name

All components except for the alias_name are optional and included only when required. When a component is omitted, the associated punctuation must also be omitted. For the first occurrence of a table within a given subquery, generated advice will omit the occurrence number, but it is legal to write #1, if desired. The partition schema and partition name are included only for children of partitioned tables. In generated advice, we always incude both, but it is legal to omit the schema. The plan name is omitted for the toplevel plan, and must be included for any subplan.

It is not always easy to determine the correct advice target by examining the query; for instance, if the planner pulls up a subquery into the parent query level, everything inside of it becomes part of the parent query level, and uses the parent query's subplan name (or no subplan name, if pulled up to the toplevel). Furthermore, the correct subquery name is sometimes not obvious. For example, when two queries are joined using an operation such as UNION or INTERSECT, no name for the subqueries is present in the SQL syntax; instead, a system-generated name is assigned to each branch. The easiest way to discover the proper advice targets is to use EXPLAIN (PLAN_ADVICE) and examine the generated advice.

F.30.4. Advice Tags #

An advice tag specifies a particular behavior that should be enforced for some portion of the query, such as a particular join order or join method. All advice tags take advice targets as arguments, and many allow lists of advice targets, which in some cases can be nested multiple levels deep. Several different classes of advice targets exist, each controlling a different aspect of query planning.

F.30.4.1. Scan Method Advice #

SEQ_SCAN(target [ ... ])
TID_SCAN(target [ ... ])
INDEX_SCAN(target index_name [ ... ])
INDEX_ONLY_SCAN(target index_name [ ... ])
FOREIGN_SCAN((target [ ... ]) [ ... ])
BITMAP_HEAP_SCAN(target bitmap_target [ ... ])

SEQ_SCAN specifies that each target table should be scanned using a Seq Scan. TID_SCAN specifies that each target table should be scanned using a TID Scan or TID Range Scan.

INDEX_SCAN specifies that each target table should be scanned using an Index Scan on the given index name. INDEX_ONLY_SCAN is similar, but specifies the use of an Index Only Scan. In either case, the index name can be, but does not have to be, schema-qualified.

FOREIGN_SCAN specifies that a foreign join between a several foreign tables should be pushed down to a remote server so that it can be implenented as a single Foreign Scan. Specifying FOREIGN_SCAN for a single foreign table is neither necessary nor permissible: a Foreign Scan will need to be used regardless. If you want to prevent a join from being pushed down, consider using the JOIN_ORDER tag for that purpose.

BITMAP_HEAP_SCAN specifies that each target table should be scanned using a Bitmap Heap Scan. The bitmap_target is currently syntax-checked and then ignored, which is busted. XXX: FIXME somehow.

The planner supports many types of scans other than those listed here; however, in most of those cases, there is no meaningful decision to be made, and hence no need for advice. For example, the output of a set-returning function that appears in the FROM clause can only ever be scanned using a Function Scan, so there is no opportunity for advice to change anything.

F.30.4.2. Join Order Advice #

JOIN_ORDER(join_order_item [ ... ])

where join_order_item is:

advice_target |
( join_order_item [ ... ] ) |
{ join_order_item [ ... ] }

When JOIN_ORDER is used without any sublists, it specifies an outer-deep join with the first advice target as the driving table, joined to each subsequent advice target in turn in the order specified. For instance, JOIN_ORDER(a b c) means that a should be the driving table, and that it should be joined first to b and then to c. If there are more tables in the query than a, b, and c, the rest can be joined afterwards in any manner.

If a JOIN_ORDER list contains a parenthesized sublist, it specifies a non-outer-deep join. The tables in the sublist must first be joined to each other much as if the sublist were a toplevel JOIN_ORDER list, and the resulting join product must then appear on the inner side of a join at the appropriate point in the join order. For example, JOIN_ORDER(a (b c) d) requires a plan of this form:

Join
  ->  Join
        -> Scan on a
        -> Join
             -> Scan on b
             -> Scan on c
  ->  Scan on d

If a JOIN_ORDER list contains a sublist surronded by curly braces, this also specifies a non-outer-deep join. However, the join order within the sublist is not constrained. For example, specifiying JOIN_ORDER(a {b c} d would allow the scans of b and c to be swapped in the previous example, which is not allowed when parenthese are used.

Parenthesized sublists can be arbitrarily nested, but sublists surrounded by curly braces cannot themselves contain sublists.

Multiple instances of JOIN_ORDER() can sometimes be needed in order to fully constraint the join order. This occurs when there are multiple join problems that are optimized separately by the planner. This can happen due to the presence of subqueries, or because there is a partitionwise join. In the latter case, each branch of the partitionwise join can have its own join order, independent of every other branch.

F.30.4.3. Join Method Advice #

join_method_name(join_method_item [ ... ])

where join_method_name is:

{ MERGE_JOIN_MATERIALIZE | MERGE_JOIN_PLAIN | NESTED_LOOP_MATERIALIZE | NESTED_LOOP_PLAIN | HASH_JOIN }

and join_method_item is:

advice_target |
( advice_target [ ... ] ) }

Join method advice specifies the table, or set of tables, that should appear on the inner side of a join using the named join method. For example, HASH_JOIN(a b) means that each of a and b should appear on the inner side of a hash join; a confirming plan must contain at least two hash joins, one of which has a and nothing else on the inner side, and the other of which has b and nothing else on the inner side. On the other hand, HASH_JOIN((a b)) means that the join product of a and b should appear together on the inner side of a a single hash join.

Note that join method advice implies a negative join order constraint. Since the named table or tables must be on the inner side of a join using the specified method, none of them can be the driving table for the entire join problem. Moreover, no table inside the set should be joined to any table outside the set until all tables within the set have been joined to each other. For example, if the advice specifies HASH_JOIN((a b)) and the system begins by joining either of those tables to some third table c, the resulting plan could never be compliant with the request to put exactly those two tables on the inner side of a hash join. When using both join order advice and join method advice for the same query, it is a good idea to make sure that they do not mandate incompatible join orders.

F.30.4.4. Partitionwise Advice #

PARTITIONWISE(partitionwise_item [ ... ])

where partitionwise_item is:

advice_target |
( advice_target [ ... ] ) }

When applied to a single target, PARTITIONWISE specifies that the specified table should not be part of any partitionwise join. When applied to a list of targets, PARTITIONWISE specifies that exactly that set of tables should be joined in partitionwise fashion. Note that, regardless of what advice is specified, no partitionwise joins will be possible if enable_partitionwise_join = off.

F.30.4.5. Semijoin Uniqueness Advice #

SEMIJOIN_UNIQUE(sj_unique_item [ ... ])
SEMIJOIN_NON_UNIQUE(sj_unique_item [ ... ])

where sj_unique_item is:

advice_target |
( advice_target [ ... ] ) }

The planner sometimes has a choice between implementing a semijoin directly and implememnting a semijoin by making the nullable side unique and then performing an inner join. SEMIJOIN_UNIQUE specifies the latter strategy, while SEMIJOIN_NON_UNIQUE specifies the former strategy. In either case, the argument is the single table or list of tables that appear beneath the nullable side of the join.

F.30.4.6. Parallel Query Advice #

GATHER(gather_item [ ... ])
GATHER_MERGE(gather_item [ ... ])
NO_GATHER(advice_target [ ... ])

where gather_item is:

advice_target |
( advice_target [ ... ] ) }

GATHER or GATHER_MERGE specifies that Gather or Gather Merge, respectively, should be placed on top of the single table specified as a target, or on top of the join between the list of tables specified as a target. This means that GATHER(a b c) is a request for three different Gather nodes, while GATHER((a b c)) is a request for a single Gather node on top of a 3-way join.

NO_GATHER specifies that none of the tables given as arguments should appear beneath a Gather or Gather Merge node.

F.30.5. Advice Feedback #

EXPLAIN provides feedback on whether supplied advice was successfully applied to the query in the form of a comment on each piece of supplied advice. For example:

SET pg_plan_advice.advice = 'hash_join(f g) join_order(f g) index_scan(f no_such_index)';
SET
rhaas=# EXPLAIN (COSTS OFF)                                                     SELECT * FROM jo_fact f
        LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id
        LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id
        WHERE val1 = 1 AND val2 = 1;
                            QUERY PLAN
-------------------------------------------------------------------
 Hash Join
   Hash Cond: ((d1.id = f.dim1_id) AND (d2.id = f.dim2_id))
   ->  Nested Loop
         ->  Seq Scan on jo_dim2 d2
               Filter: (val2 = 1)
         ->  Materialize
               ->  Seq Scan on jo_dim1 d1
                     Filter: (val1 = 1)
   ->  Hash
         ->  Seq Scan on jo_fact f
 Supplied Plan Advice:
   INDEX_SCAN(f no_such_index) /* matched, inapplicable, failed */
   HASH_JOIN(f) /* matched */
   HASH_JOIN(g) /* not matched */
   JOIN_ORDER(f g) /* partially matched */

For this query, f is a valid advice target, but g is not. Therefore, the request to place f on the inner side of a hash join is listed as matched, but the request to place g on the inner side of a hash join is listed as not matched. The JOIN_ORDER advice tag involves one valid target and one invald target, and so is listed as partially matched. Note that HASH_JOIN(f g) is actually a request for two logically separate behaviors, whereas JOIN_ORDER(f g) is a single request. When providing advice feedback, EXPLAIN shows each logical request separtely, together with all the feedback applicable to that request type.

Advice feedback can include any of the folllowing:

  • matched means that all of the specified advice targets were observed during query planning.

  • partially matched means that some but not all of the specified advice targetes were observed during query planning.

  • not matched means that none of the specified advice targets were observed during query planning. This may be happen if the advice simply doesn't match the query, or it may occur if the relevant portion of the query was not planned, perhaps because it was gated by a condition that was simplified to constant false.

  • inapplicable means that the advice tag could not be applied to the advice targets for some reason. For example, this will happen if the use of a nonexistent index is requested, or if an attempt is made to control semijoin uniquness for a non-semijoin.

  • conflicting means that two or more pieces of advice request incompatible behaviors. For example, if you advise a sequential scan and an index scan for the same table, both requests will be flagged as conflicting. This also commonly happens if join method advice or semijoin uniqueness advice implies a join order incompatible with the one explicitly specified; see Section F.30.4.3.

  • failed means that query plan does not comply with the advice. This only occurs for entries that are also shown as matched. It frequently occurs for entries that are also marked as conflicting or inapplicable. However, it can also occur when the advice is valid insofar as pg_plan_advice is able to determine, but the planner is not able to construct a legal plan that can comply with the advice. It is important to note that the sanity checks performed by pg_plan_advice are fairly superficial and focused mostly on looking for logical inconsistencies in the advice string; only the planner knows what will actually work.

All advice should be marked as exactly one of matched, partially matched, or not matched.

F.30.6. Advice Collectors #

pg_plan_advice can be configured to automatically generate advice every time a query is planned and store the query and the generated advice string either in local or shared memory.

To enable a collector, you must first set a collection limit. When the number of queries for which advice has been stored exceeds the collection limit, the oldest queries and the corresponding advice will be discarded. Then, you must adjust a separate setting to actually enable advice collection. For the local collector, set the collection limit by configuring pg_plan_advice.local_collection_limit to a value greater than zero, and then enable advice collection by setting pg_plan_advice.local_collector = true. For the shared collector, the procedure is the same, except that the names of the settings are pg_plan_advice.shared_collection_limit and pg_plan_advice.shared_collector. Note that the local collector stores query texts and advice strings in backend-local memory, and the shared collector does the same in dynamic shared memory, so configuring large limits may result in considerable memory consumption.

Once the collector is enabled, you can run any queries for which you wish to see the generated plan advice. Then, you can examine what has been collected using whichever of SELECT * FROM pg_get_collected_local_advice() or SELECT * FROM pg_get_collected_shared_advice() corresponds to the collector you enabled. To discard the collected advice and release memory, you can call pg_clear_collected_local_advice() or pg_clear_collected_shared_advice().

In addition to the query texts an advice strings, the advice collectors will also store the OID of the role that caused the query to be planned, the OID of the database in which the query was planned, the query ID, and the time at which the collection occurred. This module does not automatically enable query ID computation; therefore, if you want the query ID value to be populated in collected advice, be sure to configure enable_query_id = on. Otherwise, the query ID may always show as 0.

F.30.7. Functions #

Note that these functions will only be available if the pg_plan_advice extension has been installed in the current database, which is not mandatory, since much of the functionality of this module can be used without installing the extension.

pg_clear_collected_local_advice() returns void

Removes all collected query texts and advice strings from backend-local memory.

pg_get_collected_local_advice() returns setof (id bigint, userid oid, dbid oid, queryid bigint, collection_time timestamptz, query text, advice text)

Returns all query texts and advice strings stored in the local advice collector.

pg_clear_collected_shared_advice() returns void

Removes all collected query texts and advice strings from shared memory.

pg_get_collected_shared_advice() returns setof (id bigint, userid oid, dbid oid, queryid bigint, collection_time timestamptz, query text, advice text)

Returns all query texts and advice strings stored in the shared advice collector.

F.30.8. Configuration Parameters #

pg_plan_advice.advice (string)

pg_plan_advice.advice is an advice string to be used during query planning.

pg_plan_advice.always_explain_supplied_advice (boolean)

pg_plan_advice.always_explain_supplied_advice causes EXPLAIN to always show any supplied advice and the associated advice feedback. The default value is true. If set to false, this information will be displayed only when EXPLAIN (PLAN_ADVICE) is used.

pg_plan_advice.always_store_advice_details (boolean)

pg_plan_advice.always_store_advice_details allows EXPLAIN to show details related to plan advice even when prepared queries are used. The default value is false. When planning a prepared query, we do not know whether EXPLAIN will be used, so by default, to reduce overhead, we do not generate plan advice, and we do not generate feedback on supplied advice. This means that if EXPLAIN EXECUTE is later used on the prepared query, it will not be able to show this information. Changing this setting to true avoids this problem, but adds additional overhead. It is probably a good idea to enable this option only in sessions where it is needed, rather than on a system-wide basis.

pg_plan_advice.local_collector (boolean)

pg_plan_advice.local_collector enables the local advice collector. The default value is false.

pg_plan_advice.local_collection_limit (integer)

pg_plan_advice.local_collection_limit sets the maximum number of query texts and advice strings retained by the local advice collector. The default value is 0.

pg_plan_advice.shared_collector (boolean)

pg_plan_advice.shared_collector enables the shared advice collector. The default value is false. Only superusers and users with the appropriate SET privilege can change this setting.

pg_plan_advice.shared_collection_limit (integer)

pg_plan_advice.shared_collection_limit sets the maximum number of query texts and advice strings retained by the shared advice collector. The default value is 0. Only superusers and users with the appropriate SET privilege can change this setting.

pg_plan_advice.trace_mask (boolean)

When pg_plan_advice.trace_mask is true, pg_plan_advice will print messages during query planning each time that pg_plan_advice alters the mask of allowable query plan types in response to supplied plan advice. The default values is false. The messages printed by this setting are not excepted to be useful except for purposes of debugging this module.

F.30.9. Author #

Robert Haas