Merge pull request #22 from cybertec-postgresql/multisite-v4.1.2

Merge in v4.1.2

Author: ants

.github/workflows/mapping.py

@@ -1 +1 @@
-versions = {'etcd': '9.6', 'etcd3': '16', 'consul': '17', 'exhibitor': '12', 'raft': '14', 'kubernetes': '15'}
+versions = {'etcd': '9.6', 'etcd3': '17', 'consul': '18', 'exhibitor': '12', 'raft': '14', 'kubernetes': '16'}

.github/workflows/tests.yaml

@@ -20,7 +20,7 @@ jobs:
         os: [ubuntu, windows, macos]
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
     - name: Set up Python 3.7
       uses: actions/setup-python@v5
@@ -91,18 +91,25 @@ jobs:
       run: python .github/workflows/run_tests.py
       if: matrix.os != 'macos'
 
+    - name: Set up Python 3.14
+      uses: actions/setup-python@v5
+      with:
+        python-version: 3.14
+      if: matrix.os != 'macos'
+    - name: Install dependencies
+      run: python .github/workflows/install_deps.py
+      if: matrix.os != 'macos'
+    - name: Run tests and flake8
+      run: python .github/workflows/run_tests.py
+      if: matrix.os != 'macos'
+
     - name: Combine coverage
       run: python .github/workflows/run_tests.py combine
 
-    - name: Install coveralls
-      run: python -m pip install coveralls
-
-    - name: Upload Coverage
-      env:
-        COVERALLS_FLAG_NAME: unit-${{ matrix.os }}
-        COVERALLS_PARALLEL: 'true'
-        GITHUB_TOKEN: ${{ secrets.github_token }}
-      run: python -m coveralls --service=github
+    - name: Upload coverage reports to Codecov
+      uses: codecov/codecov-action@v5
+      with:
+        flags: unit-${{ matrix.os }}
 
   behave:
     runs-on: ${{ fromJson('{"ubuntu":"ubuntu-22.04","windows":"windows-latest","macos":"macos-14"}')[matrix.os] }}
@@ -114,7 +121,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu]
-        python-version: [3.7, 3.13]
+        python-version: [3.7, 3.14]
         dcs: [etcd, etcd3, consul, exhibitor, kubernetes, raft]
         include:
         - os: macos
@@ -128,12 +135,14 @@ jobs:
           dcs: etcd3
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
     - uses: nolar/setup-k3d-k3s@v1
+      with:
+        github-token: ${{ secrets.GITHUB_TOKEN }}
       if: matrix.dcs == 'kubernetes'
     - name: Add postgresql and citus apt repo
       run: |
@@ -164,17 +173,6 @@ jobs:
       run: bash <(curl -Ls https://coverage.codacy.com/get.sh) report -r cobertura.xml -l Python --partial
       if: ${{ env.SECRETS_AVAILABLE == 'true' }}
 
-  coveralls-finish:
-    name: Finalize coveralls.io
-    needs: unit
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/setup-python@v5
-    - run: python -m pip install coveralls
-    - run: python -m coveralls --service=github --finish
-      env:
-        GITHUB_TOKEN: ${{ secrets.github_token }}
-
   codacy-final:
     name: Finalize Codacy
     needs: behave
@@ -186,25 +184,25 @@ jobs:
   pyright:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
-    - name: Set up Python 3.13
+    - name: Set up Python 3.14
       uses: actions/setup-python@v5
       with:
-        python-version: 3.13
+        python-version: 3.14
 
     - name: Install dependencies
       run: python -m pip install -r requirements.txt psycopg2-binary psycopg
 
     - uses: jakebailey/pyright-action@v2
       with:
-        version: 1.1.405
+        version: 1.1.408
 
   ydiff:
     name: Test compatibility with the latest version of ydiff
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
     - name: Set up Python 3.13
       uses: actions/setup-python@v5
@@ -220,7 +218,7 @@ jobs:
   docs:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
     - name: Set up Python 3.11
       uses: actions/setup-python@v5
@@ -244,7 +242,7 @@ jobs:
   isort:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v5
 
     - name: Set up Python 3.12
       uses: actions/setup-python@v5

README.rst

@@ -1,10 +1,55 @@
 |Tests Status| |Coverage Status|
 
+.. image:: docs/_static/patroni-logo.svg
+   :height: 128px
+   :width: 128px
+
 Patroni: A Template for PostgreSQL HA with ZooKeeper, etcd or Consul
 --------------------------------------------------------------------
 
 You can find a version of this documentation that is searchable and also easier to navigate at `patroni.readthedocs.io <https://patroni.readthedocs.io>`__.
 
+**Important!**
+  Running Patroni on **memory-restricted systems with Python 3.11+**
+
+----
+
+If you run Patroni on a system with strict memory limits, for example with ``vm.overcommit_memory=2`` (recommended for PostgreSQL), and use Python 3.11 or newer, you may observe unexpected behavior:
+
+- Patroni appears healthy
+- PostgreSQL continues to run
+- Patroni **REST API becomes unresponsive**
+- The operating system reports that Patroni is listening on the REST API port
+- Patroni logs look normal; however, following messages may appear once: ``Exception ignored in thread started by: <object repr() failed>``, ``MemoryError``
+- Kernel logs may contain messages such as   ``not enough memory for the allocation``
+
+This behavior is caused by a `bug in Python 3.11+ <https://github.com/python/cpython/issues/140746>`__.
+Under strict memory conditions, starting a new thread may hang indefinitely when there is not enough free memory.
+
+Recommended solution
+--------------------
+
+Recent Patroni releases (4.1.1+, 4.0.8+) reduce the impact of this issue by starting all required threads early during startup, before the system is under memory pressure.
+
+Additional recommendations (Linux, glibc)
+-----------------------------------------
+
+When running with ``vm.overcommit_memory=2`` (recommended for PostgreSQL), we also recommend starting Patroni with the following environment variables configured:
+
+- ``MALLOC_ARENA_MAX=1`` - reduces the amount of virtual memory allocated by glibc for multi-threaded
+  applications
+- ``PG_MALLOC_ARENA_MAX=`` - resets the value of ``MALLOC_ARENA_MAX`` for PostgreSQL processes started by Patroni.
+
+In addition, you may tune the following Patroni configuration parameters:
+
+- ``thread_stack_size`` - stack size used for threads started by Patroni. Lowering this value reduces memory usage of the Patroni process. The default value set by Patroni is ``512kB``. Increase ``thread_stack_size`` if Patroni experience stack-related crashes; otherwise the default value is sufficient.
+- ``thread_pool_size`` - size of the thread pool used by Patroni for asynchronous tasks and REST API communication with other members during leader race or failsafe checks. The default value is ``5``, which is sufficient for three-node clusters.
+- ``restapi.thread_pool_size`` - size of the thread pool used to process REST API requests. The default value is ``5``, allowing up to five parallel REST API requests. Note that requests involving SQL queries are effectively serialized because a single database connection is used, so increasing this value typically provides no benefit.
+
+----
+
+PostgreSQL High Availability and Patroni
+----------------------------------------
 
 There are many ways to run high availability with PostgreSQL; for a list, see the `PostgreSQL Documentation <https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling>`__.
 
@@ -173,5 +218,5 @@ When connecting from an application, always use a non-superuser. Patroni require
 
 .. |Tests Status| image:: https://github.com/patroni/patroni/actions/workflows/tests.yaml/badge.svg
    :target: https://github.com/patroni/patroni/actions/workflows/tests.yaml?query=branch%3Amaster
-.. |Coverage Status| image:: https://coveralls.io/repos/patroni/patroni/badge.svg?branch=master
-   :target: https://coveralls.io/github/patroni/patroni?branch=master
+.. |Coverage Status| image:: https://codecov.io/gh/patroni/patroni/graph/badge.svg?token=qWNJyFTeul 
+   :target: https://codecov.io/gh/patroni/patroni

docs/ENVIRONMENT.rst

@@ -8,9 +8,12 @@ It is possible to override some of the configuration parameters defined in the P
 Global/Universal
 ----------------
 -  **PATRONI\_CONFIGURATION**: it is possible to set the entire configuration for the Patroni via ``PATRONI_CONFIGURATION`` environment variable. In this case any other environment variables will not be considered!
+-  **PATRONI\_THREAD\_POOL\_SIZE**: size of thread pool used by Patroni to execute asynchronous tasks and communicate via REST API with other members during leader race or failsafe checks. Minimal value is ``5``, default value is ``5``.
+-  **PATRONI\_THREAD\_STACK\_SIZE**: specifies the stack size to be used for threads started by Patroni. Value must be aligned by ``64kB``. Minimal value is ``64kB``,  default value (set by Patroni) is ``512kB``.
 -  **PATRONI\_NAME**: name of the node where the current instance of Patroni is running. Must be unique for the cluster.
 -  **PATRONI\_NAMESPACE**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **PATRONI\_SCOPE**: cluster name
+-  **PG\_MALLOC\_ARENA\_MAX**: custom value for ``MALLOC_ARENA_MAX`` environment variable for  ``postmaster`` process. If not set, ``postmaster`` will inherit ``MALLOC_ARENA_MAX`` value.
 
 Log
 ---
@@ -193,6 +196,7 @@ PostgreSQL
 
 REST API
 --------
+-  **PATRONI\_RESTAPI\_THREAD\_POOL\_SIZE**: size of thread pool used by Patroni to process REST API requests. Minimal value is ``5``, default value is ``5``.
 -  **PATRONI\_RESTAPI\_CONNECT\_ADDRESS**: IP address and port to access the REST API.
 -  **PATRONI\_RESTAPI\_LISTEN**: IP address and port that Patroni will listen to, to provide health-check information for HAProxy.
 -  **PATRONI\_RESTAPI\_USERNAME**: Basic-auth username to protect unsafe REST API endpoints.

docs/_static/patroni-logo.png

@@ -38,7 +38,7 @@ You can find below an overview of steps for converting an existing Postgres clus
 
    #. Create a YAML configuration file for Patroni. You can use :ref:`Patroni configuration generation and validation tooling <validate_generate_config>` for that.
 
-      * **Note (specific for the primary node):** If you have replication slots being used for replication between cluster members, then it is recommended that you enable ``use_slots`` and configure the existing replication slots as permanent via the ``slots`` configuration item. Be aware that Patroni automatically creates replication slots for replication between members, and drops replication slots that it does not recognize, when ``use_slots`` is enabled. The idea of using permanent slots here is to allow your existing slots to persist while the migration to Patroni is in progress. See :ref:`YAML Configuration Settings <yaml_configuration>` for details.
+      * **Note (specific for the primary node):** If you have replication slots being used for replication between cluster members, then it is recommended that you enable ``use_slots`` and configure the existing replication slots as permanent via the ``slots`` configuration item. Be aware that Patroni automatically creates replication slots for replication between members, and drops replication slots that it does not recognize, when ``use_slots`` is enabled. The idea of using permanent slots here is to allow your existing slots to persist while the migration to Patroni is in progress. See :ref:`Dynamic Configuration Settings <dynamic_configuration>` for details.
 
    #. Start Patroni using the ``patroni`` systemd service unit. It automatically detects that Postgres is already running and starts monitoring the instance.
 
@@ -47,7 +47,7 @@ You can find below an overview of steps for converting an existing Postgres clus
    #. Immediate restart of the standby nodes.
    #. Scheduled restart of the primary node within a maintenance window.
 
-#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through :ref:`patronictl edit-config cluster-name member-name <patronictl_edit_config_parameters>` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
+#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through :ref:`patronictl edit-config cluster-name <patronictl_edit_config_parameters>` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
 
    .. code-block:: sql
 

docs/_static/patroni-logo.svg

@@ -157,6 +157,16 @@ How can I change my environment configuration?
 
     Take care to not cause a failover in the cluster! You might be interested in checking :ref:`patronictl_pause`.
 
+How can I reduce repetitive heartbeat log lines during normal operation?
+    If your logs are too noisy because of repeated lines like ``Lock owner: ...`` and ``no action. I am ...``,
+    configure ``log.deduplicate_heartbeat_logs: true``.
+
+    You can set it either in the Patroni YAML file (:ref:`log_settings`) or with
+    ``PATRONI_LOG_DEDUPLICATE_HEARTBEAT_LOGS=true``.
+
+    Keep in mind this reduces log volume by suppressing repeated heartbeat messages, but you also lose per-loop
+    heartbeat visibility that can help during failover diagnostics.
+
 What occurs if I change a Postgres GUC that requires a reload?
     When you change the dynamic or the local configuration as explained in the previous questions, Patroni will take care of reloading the Postgres configuration for you.
 

docs/existing_data.rst

@@ -4,7 +4,7 @@
 HA multi datacenter
 ===================
 
-The high availability of a PostgreSQL cluster deployed in multiple data centers is based on replication, which can be synchronous or asynchronous (`replication_modes <replication_modes.rst>`_).
+The high availability of a PostgreSQL cluster deployed in multiple data centers is based on replication, which can be synchronous or asynchronous (see :ref:`replication modes <replication_modes>`).
 
 In both cases, it is important to be clear about the following concepts: