Updated documentation.

Author: Antonin Houska

README.md

@@ -1,11 +1,37 @@
 # pg_rewrite
 
-`pg_rewrite` aims to be a set of tools to perform maintenance tasks which
-require a table to be rewritten (i.e. the table data to be copied to a new
-storage) and which are expected to limit the access to the table as little as
-possible.
+`pg_rewrite` is a tool to rewrite table (i.e. to copy its data to a new
+file). It allows both read and write access to the table during the rewriting.
 
-# INSTALL
+Following are the most common reasons to rewrite a table:
+
+1.  Change data type of column(s)
+
+    Typically this is needed if the existing data type is running out of
+    values. For example, you may need to change `interger` type to
+    `bigint`. `ALTER TABLE` command can do that too, but it allows neither
+    write nor read access to the table during the rewriting.
+
+2.  Partition the table
+
+    If you realize that your table is getting much bigger than expected and
+    that partitioning would make your life easier, the next question may be
+    how to copy the existing data to the new, partitioned table without
+    stopping all the applications that run DML commands on the table. (When
+    you decide to use partitioning, the amount of data to copy might already
+    be significant, so the copying might need a while.)
+
+3.  Change order of columns
+
+    If you conclude that a different order of columns would save significant
+    disk space (due to reduced paddding), the problem boils down to copying
+    data to a new table like in 2). Again, you may need `pg_rewrite` to make
+    the change smooth.
+
+Note that the following use cases can be combined in a single rewrited.
+
+
+# INSTALLATION
 
 Install PostgreSQL before proceeding. Make sure to have `pg_config` binary,
 these are typically included in `-dev` and `-devel` packages. PostgreSQL server
@@ -14,6 +40,7 @@ version 13 or later is required.
 ```bash
 git clone https://github.com/cybertec-postgresql/pg_rewrite.git
 cd pg_rewrite
+git checkout <the latest stable version>
 make
 make install
 ```
@@ -34,28 +61,28 @@ CREATE EXTENSION pg_rewrite;
 
 # USAGE
 
-Assuming you have a table defined like this:
+Assume you have a table defined like this
 
 ```
 CREATE TABLE measurement (
-    id              serial,
+    id              int,
     city_id         int not null,
     logdate         date not null,
     peaktemp        int,
-    unitsales       int,
     PRIMARY KEY(id, logdate)
 );
 ```
 
-You need to create a partitioned table having the same columns and data types:
+and you need to replace it with a partitioned table. At the same time, you
+want to change the data type of the `id` column to `bigint`.
+
 
 ```
 CREATE TABLE measurement_aux (
-    id              serial,
+    id              bigint,
     city_id         int not null,
     logdate         date not null,
     peaktemp        int,
-    unitsales       int,
     PRIMARY KEY(id, logdate)
 ) PARTITION BY RANGE (logdate);
 ```
@@ -73,37 +100,66 @@ CREATE TABLE measurement_y2006m03 PARTITION OF measurement_aux
 -- ...
 ```
 
-*It's essential that both the source (`measurement`) and destination
+*It's essential that both the source (`measurement`) and target
 (`measurement_aux`) table have an identity index. The easiest way to ensure
 this is to create `PRIMARY KEY` or `UNIQUE` constraint. Also note that the key
-(i.e. column list) of the identity index of the source and destination table
-must be identical. The identity is needed to process data changes that
-applications make while data is being copied from the source to the
-destination table.*
-
-Also, unless you've set `rewrite.check_constraints` to `false`, make sure that
-the destination table has all the constraints that the source table has.
+(i.e. column list) of the identity index of the source and target table must
+be identical. The identity is needed to process data changes that applications
+make while data is being copied from the source to the target table.*
 
-Then, in order to copy the data into the destination table, run the
-`partition_table()` function and pass it both the source and destination table,
-as well as a new table name for the source table. For example:
+Then, in order to copy the data into the target table, run the
+`rewrite_table()` function and pass it both the source and target table, as
+well as a new table name for the source table. For example:
 
 ```
-SELECT partition_table('measurement', 'measurement_aux', 'measurement_old');
+SELECT rewrite_table('measurement', 'measurement_aux', 'measurement_old');
 ```
 
-The call will copy data from `measurement` to `measurement_aux`, then it will
-lock `measurement` exclusively and rename (1) `measurement` to
-`measurement_old`, (2) `measurement_aux` to `measurement`. Thus `measurement`
-ends up to be the partitioned table, while `measurement_old` is the original,
-non-partitioned table.
+The call will first copy all rows from `measurement` to `measurement_aux`. The
+it will apply to `measurement_aux` all the data changes (INSERT, UPDATE,
+DELETE) that took place in `measurement` during the copying. Next, it will
+lock `measurement` so that neither read nor write access is possible. Finally
+it will rename `measurement` to `measurement_old` and `measurement_aux` to
+`measurement`. Thus `measurement` ends up to be the partitioned table, while
+`measurement_old` is the original, non-partitioned table.
+
+If a column of the target table has a different data type from the
+corresponding column of the source table, an implicit or assignment cast must
+exist between the two types.
+
+# Constraints
+
+The target table should obviously have the same constraints as the source
+table. It's recommended to handle constraints creation this way:
+
+1.  Add PRIMARY KEY, UNIQUE and EXCLUDE constraints of the source table to the
+    target table before you call `rewrite_table()`. These are enforced during
+    the rewriting, so any violation would make `rewrite_table()` fail
+    (ROLLBACK). (The constraints must have been enforced in the source table,
+    but it does not hurt to check them in the target table, especially if the
+    column data type is being changed.)
+
+2.  Add NOT NULL constraints. `rewrite_table()` by-passes validation of these,
+    but all the rows it inserts into the target table must have been validated
+    in the source table. Even if the column data tape is different in the
+    target table, the data type conversion should not turn non-NULL value to
+    NULL or vice versa.
+
+3.  CHECK and FOREIGN KEY are created automatically by `rewrite_table()` when
+    all the data changes have been applied to the target table. However, these
+    constraints are created as NOT VALID, so you need to use the `ALTER TABLE
+    ... VALIDATE CONSTRAINT ...` command to validate them.
+
+    Of course, the function could create the constraints immediately as valid,
+    but that would imply blocking access to the table for significant time.
+
 
 # Progress monitoring
 
-If `partition_table()` takes long time to finish, you might be interested in
-the progress. The `pg_rewrite_progress` view shows all the pending calls of
-the function in the current database. The `src_table`, `dst_table` and
-`src_table_new` columns contain the arguments of the `partition_table()`
+If `rewrite_table()` takes long time to finish, you might be interested in the
+progress. The `pg_rewrite_progress` view shows all the pending calls of the
+function in the current database. The `src_table`, `dst_table` and
+`src_table_new` columns contain the arguments of the `rewrite_table()`
 function. `ins_initial` is the number of tuples inserted into the new table
 storage during the "initial load stage", i.e. the number of tuples present in
 the table before the processing started. On the other hand, `ins`, `upd` and
@@ -113,44 +169,29 @@ incorporated into the partitioned table, otherwise they'd get lost.)
 
 # Limitations
 
-Please consider the following before you try to use the function:
-
-* Foreign table partitions are not supported.
-
-* It's not expected that the table that you try to partition is referenced by
-  any foreign key. The problem is that the destination table is initially
-  empty, so you won't be able to create the foreign keys that reference it.
+If the target table is partitioned, it's not allowed to have foreign
+tables as partitions.
 
 # Configuration
 
 Following is the description of the configuration variables that affect
 behavior of the functions of this extension.
 
-* `rewrite.check_constraints`
-
-Before copying of the data starts, it's checked whether the destination table
-has the same constraints as the source table, and throws an ERROR if a
-difference is found. The point is that due to (accidentally) missing
-constraint on the destination table, data that violate constraints on the
-source table would be allowed to appear in the destination table as soon as
-the processing is finished. Even an extra constraint on the destination table
-is a problem because the extension only assumes that all the data it copies do
-satisfy constraints on the source table, however it does not validate them
-against the additional constraints on the destination table.
-
-By setting `rewrite.check_constraints` to `false`, the user can turn off the
-constraint checks. Please be very cautions before you do so.
-
-The default value is `true`.
-
 * `rewrite.max_xlock_time`
 
 Although the table being processed is available for both read and write
 operations by other transactions most of the time, an exclusive lock is needed
-to finalize the processing (i.e. to process the remaining concurrent changes
-and to rename the tables). If the extension function seems to block access to
-tables too much, consider setting `rewrite.max_xlock_time` GUC parameter. For
-example:
+to finalize the processing (i.e. to do the table renaming), which blocks both
+read and write access. This should take very short time that users should
+harly notice.
+
+However, if a significant amount of changes took place in the source table
+while the extension was waiting for the (exclusive) lock, the outage might
+take proportionally longer time. The point is that those changes need to be
+propagated to the target table before the exclusive lock can be released.
+
+If the extension function seems to block access to tables too much, consider
+setting `rewrite.max_xlock_time` GUC parameter. For example:
 
 ```
 SET rewrite.max_xlock_time TO 100;
@@ -169,21 +210,14 @@ it needs.
 
 # Concurrency
 
-1. The extension does not prevent other transactions from altering table at
-   certain stages of the processing. If a `disruptive command` (i.e. `ALTER
-   TABLE`) manages to commit before the processing could finish, the table
-   processing aborts and all changes done are rolled back.
+1. While the rewrite_table() function is executing, `ALTER TABLE` command on
+   the same table should be blocked until the rewriting is done. However, in
+   some cases the `ALTER TABLE` command and the rewrite_table() function might
+   end up in a deadlock. Therefore it's recommended not to run ALTER TABLE on
+   a table which is being rewritten.
 
-2. The functions of this extension allow for MVCC-unsafe behavior described in
+2. The `rewrite_table()` function allows for MVCC-unsafe behavior described in
    the first paragraph of [mvcc-caveats][1].
 
-# Locking
-
-Since the table renaming requires an exclusive lock, applications won't be able
-to access the table that you try to process for very short time. However, if a
-significant amount of changes took place in the source table while the
-extension was waiting for the lock, the outage will take proportionally longer
-time. The point is that those changes need to be propagated to the destination
-table before the exclusive lock can be released.
 
-[1]: https://www.postgresql.org/docs/13/static/mvcc-caveats.html
+[1]: https://www.postgresql.org/docs/current/mvcc-caveats.html

pg_rewrite.control

@@ -1,5 +1,5 @@
 # pg_rewrite extension
-comment = 'Tools for maintenance that requires table rewriting.'
+comment = 'Tool for maintenance that requires table rewriting.'
 default_version = '1.3'
 module_pathname = '$libdir/pg_rewrite'
 relocatable = true