Merge pull request #253407 from mulander/schema-based-sharding

Schema based sharding

Author: v-dirichards

articles/cosmos-db/postgresql/TOC.yml

@@ -62,6 +62,9 @@
         - name: Design a real-time dashboard
           href: tutorial-design-database-realtime.md
           displayName: tutorial, real-time
+        - name: Design for microservices
+          href: tutorial-design-database-microservices.md
+          displayName: tutorial, microservices
     - name: Administer
       items:
         - name: Set up private access
@@ -71,6 +74,8 @@
   items:
     - name: Clusters
       href: concepts-cluster.md
+    - name: Sharding models
+      href: concepts-sharding-models.md
     - name: Distributed data
       items:
         - name: Nodes and tables

articles/cosmos-db/postgresql/concepts-colocation.md

@@ -7,7 +7,7 @@ ms.service: cosmos-db
 ms.subservice: postgresql
 ms.custom: ignite-2022
 ms.topic: conceptual
-ms.date: 05/06/2019
+ms.date: 10/01/2023
 ---
 
 # Table colocation in Azure Cosmos DB for PostgreSQL
@@ -18,13 +18,13 @@ Colocation means storing related information together on the same nodes. Queries
 
 ## Data colocation for hash-distributed tables
 
-In Azure Cosmos DB for PostgreSQL, a row is stored in a shard if the hash of the value in the distribution column falls within the shard's hash range. Shards with the same hash range are always placed on the same node. Rows with equal distribution column values are always on the same node across tables.
+In Azure Cosmos DB for PostgreSQL, a row is stored in a shard if the hash of the value in the distribution column falls within the shard's hash range. Shards with the same hash range are always placed on the same node. Rows with equal distribution column values are always on the same node across tables. The concept of hash-distributed tables is also known as [row-based sharding](concepts-sharding-models.md#row-based-sharding). In [schema-based sharding](concepts-sharding-models.md#schema-based-sharding), tables within a distributed schema are always colocated.
 
 :::image type="content" source="media/concepts-colocation/colocation-shards.png" alt-text="Diagram shows shards with the same hash range placed on the same node for events shards and page shards." border="false":::
 
 ## A practical example of colocation
 
-Consider the following tables that might be part of a multi-tenant web
+Consider the following tables that might be part of a multitenant web
 analytics SaaS:
 
 ```sql
@@ -153,4 +153,4 @@ In some cases, queries and table schemas must be changed to include the tenant I
 
 ## Next steps
 
-- See how tenant data is colocated in the [multi-tenant tutorial](tutorial-design-database-multi-tenant.md).
+- See how tenant data is colocated in the [multitenant tutorial](tutorial-design-database-multi-tenant.md).

articles/cosmos-db/postgresql/concepts-distributed-data.md

@@ -6,7 +6,7 @@ author: jonels-msft
 ms.service: cosmos-db
 ms.subservice: postgresql
 ms.topic: conceptual
-ms.date: 10/26/2022
+ms.date: 09/29/2023
 ---
 
 # Nodes and tables in Azure Cosmos DB for PostgreSQL
@@ -25,23 +25,22 @@ allows the database to scale by adding more nodes to the cluster.
 
 Every cluster has a coordinator node and multiple workers. Applications
 send their queries to the coordinator node, which relays it to the relevant
-workers and accumulates their results. Applications are not able to connect
-directly to workers.
+workers and accumulates their results.
 
-Azure Cosmos DB for PostgreSQL allows the database administrator to *distribute* tables,
-storing different rows on different worker nodes. Distributed tables are the
-key to Azure Cosmos DB for PostgreSQL performance. Failing to distribute tables leaves them entirely
-on the coordinator node and cannot take advantage of cross-machine parallelism.
+Azure Cosmos DB for PostgreSQL allows the database administrator to *distribute* tables and/or schemas,
+storing different rows on different worker nodes. Distributed tables and/or schemas are the
+key to Azure Cosmos DB for PostgreSQL performance. Failing to distribute tables and/or schemas leaves them entirely
+on the coordinator node and can't take advantage of cross-machine parallelism.
 
 For each query on distributed tables, the coordinator either routes it to a
 single worker node, or parallelizes it across several depending on whether the
-required data lives on a single node or multiple. The coordinator decides what
+required data lives on a single node or multiple. With [schema-based sharding](concepts-sharding-models.md#schema-based-sharding), the coordinator routes the queries directly to the node that hosts the schema. In both schema-based sharding and [row-based sharding](concepts-sharding-models.md#row-based-sharding), the coordinator decides what
 to do by consulting metadata tables. These tables track the DNS names and
 health of worker nodes, and the distribution of data across nodes.
 
 ## Table types
 
-There are three types of tables in a cluster, each
+There are five types of tables in a cluster, each
 stored differently on nodes and used for different purposes.
 
 ### Type 1: Distributed tables
@@ -77,7 +76,15 @@ values like order statuses or product categories.
 
 When you use Azure Cosmos DB for PostgreSQL, the coordinator node you connect to is a regular PostgreSQL database. You can create ordinary tables on the coordinator and choose not to shard them.
 
-A good candidate for local tables would be small administrative tables that don't participate in join queries. An example is a users table for application sign-in and authentication.
+A good candidate for local tables would be small administrative tables that don't participate in join queries. An example is a `users` table for application sign-in and authentication.
+
+### Type 4: Local managed tables
+
+Azure Cosmos DB for PostgreSQL might automatically add local tables to metadata if a foreign key reference exists between a local table and a reference table. Additionally locally managed tables can be manually created by executing [create_reference_table](reference-functions.md#citus_add_local_table_to_metadata) citus_add_local_table_to_metadata function on regular local tables. Tables present in metadata are considered managed tables and can be queried from any node, Citus knows to route to the coordinator to obtain data from the local managed table. Such tables are displayed as local in [citus_tables](reference-metadata.md#distributed-tables-view) view.
+
+### Type 5: Schema tables
+
+With [schema-based sharding](concepts-sharding-models.md#schema-based-sharding) introduced in Citus 12.0, distributed schemas are automatically associated with individual colocation groups. Tables created in those schemas are automatically converted to colocated distributed tables without a shard key. Such tables are considered schema tables and are displayed as schema in [citus_tables](reference-metadata.md#distributed-tables-view) view.
 
 ## Shards
 

articles/cosmos-db/postgresql/concepts-nodes.md

@@ -0,0 +1,68 @@
+---
+title: Sharding models - Azure Cosmos DB for PostgreSQL
+description: What is sharding, and what sharding models are available in Azure Cosmos DB for PostgreSQL
+ms.author: adamwolk
+author: mulander
+ms.service: cosmos-db
+ms.subservice: postgresql
+ms.topic: conceptual
+ms.date: 09/08/2023
+---
+
+# Sharding models
+
+[!INCLUDE [PostgreSQL](../includes/appliesto-postgresql.md)]
+
+Sharding is a technique used in database systems and distributed computing to horizontally partition data across multiple servers or nodes. It involves breaking up a large database or dataset into smaller, more manageable parts called Shards. A shard contains a subset of the data, and together shards form the complete dataset.
+
+Azure Cosmos DB for PostgreSQL offers two types of data sharding, namely row-based and schema-based. Each option comes with its own [Sharding tradeoffs](#sharding-tradeoffs), allowing you to choose the approach that best aligns with your application's requirements.
+
+## Row-based sharding
+
+The traditional way in which Azure Cosmos DB for PostgreSQL shards tables is the single database, shared schema model also known as row-based sharding, tenants coexist as rows within the same table. The tenant is determined by defining a [distribution column](./concepts-nodes.md#distribution-column), which allows splitting up a table horizontally.
+
+Row-based is the most hardware efficient way of sharding. Tenants are densely packed and distributed among the nodes in the cluster. This approach however requires making sure that all tables in the schema have the distribution column and that all queries in the application filter by it. Row-based sharding shines in IoT workloads and for achieving the best margin out of hardware use.
+
+Benefits:
+
+* Best performance
+* Best tenant density per node
+
+Drawbacks:
+
+* Requires schema modifications
+* Requires application query modifications
+* All tenants must share the same schema
+
+## Schema-based sharding
+
+Available with Citus 12.0 in Azure Cosmos DB for PostgreSQL, schema-based sharding is the shared database, separate schema model, the schema becomes the logical shard within the database. Multitenant apps can use a schema per tenant to easily shard along the tenant dimension. Query changes aren't required and the application only needs a small modification to set the proper search_path when switching tenants. Schema-based sharding is an ideal solution for microservices, and for ISVs deploying applications that can't undergo the changes required to onboard row-based sharding.
+
+Benefits:
+
+* Tenants can have heterogeneous schemas
+* No schema modifications required
+* No application query modifications required
+* Schema-based sharding SQL compatibility is better compared to row-based sharding
+
+Drawbacks:
+
+* Fewer tenants per node compared to row-based sharding
+
+## Sharding tradeoffs
+
+<br />
+
+|| Schema-based sharding | Row-based sharding|
+|---|---|---|
+|Multi-tenancy model|Separate schema per tenant|Shared tables with tenant ID columns|
+|Citus version|12.0+|All versions|
+|Extra steps compared to vanilla PostgreSQL|None, only a config change|Use create_distributed_table on each table to distribute & colocate tables by tenant ID|
+|Number of tenants|1-10k|1-1 M+|
+|Data modeling requirement|No foreign keys across distributed schemas|Need to include a tenant ID column (a distribution column, also known as a sharding key) in each table, and in primary keys, foreign keys|
+|SQL requirement for single node queries|Use a single distributed schema per query|Joins and WHERE clauses should include tenant_id column|
+|Parallel cross-tenant queries|No|Yes|
+|Custom table definitions per tenant|Yes|No|
+|Access control|Schema permissions|Schema permissions|
+|Data sharing across tenants|Yes, using reference tables (in a separate schema)|Yes, using reference tables|
+|Tenant to shard isolation|Every tenant has its own shard group by definition|Can give specific tenant IDs their own shard group via isolate_tenant_to_new_shard|

articles/cosmos-db/postgresql/concepts-sharding-models.md

@@ -6,7 +6,7 @@ author: jonels-msft
 ms.service: cosmos-db
 ms.subservice: postgresql
 ms.topic: conceptual
-ms.date: 05/16/2023
+ms.date: 10/01/2023
 ---
 
 # Cluster upgrades in Azure Cosmos DB for PostgreSQL
@@ -16,7 +16,7 @@ ms.date: 05/16/2023
 The Azure Cosmos DB for PostgreSQL managed service can handle upgrades of both the
 PostgreSQL server, and the Citus extension. All clusters are created with [the latest Citus version](./reference-extensions.md#citus-extension) available for the major PostgreSQL version you select during cluster provisioning. When you select a PostgreSQL version such as PostgreSQL 15 for in-place cluster upgrade, the latest Citus version supported for selected PostgreSQL version is going to be installed. 
 
-If you need to upgrade the Citus version only, you can do so by using an in-place upgrade. For instance, you may want to upgrade Citus 11.0 to Citus 11.3 on your PostgreSQL 14 cluster without upgrading Postgres version. 
+If you need to upgrade the Citus version only, you can do so by using an in-place upgrade. For instance, you might want to upgrade Citus 11.0 to Citus 11.3 on your PostgreSQL 14 cluster without upgrading Postgres version. 
 
 ## Upgrade precautions
 
@@ -30,10 +30,14 @@ Also, upgrading a major version of Citus can introduce changes in behavior.
 It's best to familiarize yourself with new product features and changes to
 avoid surprises.
 
+Noteworthy Citus 12 changes:
+* The default rebalance strategy changed from `by_shard_count` to `by_disk_size`.
+* Support for PostgreSQL 13 has been dropped as of this version.
+
 Noteworthy Citus 11 changes:
 
-* Table shards may disappear in your SQL client. Their visibility
-  is now controlled by
+* Table shards might disappear in your SQL client. You can control their visibility
+  using
   [citus.show_shards_for_app_name_prefixes](reference-parameters.md#citusshow_shards_for_app_name_prefixes-text).
 * There are several [deprecated
   features](https://www.citusdata.com/updates/v11-0/#deprecated-features).

articles/cosmos-db/postgresql/concepts-upgrade.md

@@ -38,7 +38,7 @@ queries.
 > [!NOTE]
 > To take advantage of newly added nodes you must [rebalance distributed table
 > shards](howto-scale-rebalance.md), which means moving some
-> [shards](concepts-distributed-data.md#shards) from existing nodes
+> [shards](concepts-nodes.md#shards) from existing nodes
 > to the new ones. Rebalancing can work in the background, and requires no
 > downtime.
 

articles/cosmos-db/postgresql/howto-scale-grow.md

@@ -15,7 +15,7 @@ ms.date: 01/30/2023
 [!INCLUDE [PostgreSQL](../includes/appliesto-postgresql.md)]
 
 To take advantage of newly added nodes, rebalance distributed table
-[shards](concepts-distributed-data.md#shards). Rebalancing moves shards from existing nodes to the new ones. Azure Cosmos DB for PostgreSQL offers
+[shards](concepts-nodes.md#shards). Rebalancing moves shards from existing nodes to the new ones. Azure Cosmos DB for PostgreSQL offers
 zero-downtime rebalancing, meaning queries continue without interruption during
 shard rebalancing.