Add Docker Compose profiles for dev and prod server setups (#198)

Since we are moving towards always running against Postgres, just
using `lnt runserver` without any Postgres server will eventually
stop being supported. To avoid creating a barrier for running local
instances, I want to make it easier to run a "real" server locally
backed by Postgres.

To do that, the easiest way is to simplify the Docker Compose setup
so that a full service can be brought up for local development in a
single command. This is achieved using profiles:

- Default (no profile): starts db + webserver, built from source, bound to
  localhost:8000. Suitable for local development:

    docker compose -f docker/compose.yaml --env-file docker/dev.env up

- prod profile: adds Nginx reverse proxy on port 80. The production
  deployment uses docker/compose.prod.yaml as an override to pull the
  pre-built LNT image from ghcr.io and bind volumes to the host
  filesystem, like it used to.

Author: ldionne

deployment/compose.env.tpl

@@ -2,4 +2,4 @@ LNT_DB_PASSWORD=${__db_password__}
 LNT_AUTH_TOKEN=${__auth_token__}
 LNT_IMAGE=${__lnt_image__}
 LNT_NGINX_CONFIG=${__lnt_nginx_config__}
-LNT_EXTERNAL_PORT=${__lnt_external_port__}
+LNT_NGINX_EXTERNAL_PORT=${__lnt_nginx_external_port__}

deployment/compose.prod.yaml

@@ -0,0 +1,27 @@
+#
+# This file contains overrides for production deployments of LNT:
+# - Bind volumes inside the Docker container to the EC2 host filesystem
+#   (e.g. /var/lib/lnt -> /var/lib/lnt) for ease of access.
+# - Pull a pre-built image of the LNT webserver instead of building from source.
+#   This allows controling the version of LNT used from Terraform.
+#
+
+services:
+  webserver:
+    build: !reset
+    image: ghcr.io/llvm/llvm-lnt:${LNT_IMAGE:-latest}
+
+volumes:
+  instance:
+    driver: local
+    driver_opts:
+      o: bind
+      type: none
+      device: /var/lib/lnt
+
+  database:
+    driver: local
+    driver_opts:
+      o: bind
+      type: none
+      device: /var/lib/postgresql

deployment/ec2-volume-mapping.yaml

@@ -81,9 +81,9 @@ data "cloudinit_config" "startup_scripts" {
           content     = file("${path.module}/../docker/compose.yaml")
         },
         {
-          path        = "/etc/lnt/ec2-volume-mapping.yaml"
+          path        = "/etc/lnt/compose.prod.yaml"
           permissions = "0400" # read-only for owner
-          content     = file("${path.module}/ec2-volume-mapping.yaml")
+          content     = file("${path.module}/compose.prod.yaml")
         },
         {
           path        = "/etc/lnt/compose.env"
@@ -93,7 +93,7 @@ data "cloudinit_config" "startup_scripts" {
             __auth_token__        = local.lnt_auth_token,
             __lnt_image__         = local.lnt_image,
             __lnt_nginx_config__  = "/etc/lnt/nginx.conf",
-            __lnt_external_port__ = local.lnt_external_port,
+            __lnt_nginx_external_port__ = local.lnt_external_port,
           })
         },
         {

deployment/main.tf

@@ -32,6 +32,7 @@ fi
 
 echo "Starting LNT service with Docker compose"
 docker compose --file /etc/lnt/compose.yaml                 \
-               --file /etc/lnt/ec2-volume-mapping.yaml      \
+               --file /etc/lnt/compose.prod.yaml            \
+               --profile nginx                              \
                --env-file /etc/lnt/compose.env              \
                up

deployment/on-ec2-boot.sh

@@ -1,6 +1,12 @@
-# This file composes a full service running LNT. A LNT service is comprised
-# of a container running a Postgres database, a container running a Nginx
-# reverse proxy and a container running a LNT webserver using gunicorn.
+# This file composes a full service running LNT. A LNT service is comprised of:
+# - A container running a Postgres database
+# - A LNT webserver using gunicorn connected to that database
+# - Optionally, a Nginx reverse proxy in front of the webserver
+#
+# The webserver is built from source and binds to localhost:8000, which is useful
+# for local development.
+#
+# When desired, the 'nginx' profile can be used, which adds the Nginx reverse proxy.
 #
 # In order to compose the full service, some secrets are required. They are taken
 # as environment variables, which means they can be stored in a file and included
@@ -13,33 +19,26 @@
 #     The authentication token used to require authentication to
 #     perform destructive actions.
 #
-# Additionally, the following aspects of the container can be customized:
-#
-#   LNT_IMAGE
-#     The version of the llvm-lnt webserver image used for the service.
-#     Defaults to 'latest'.
+# Additionally, the following aspects can be customized:
 #
 #   LNT_NGINX_CONFIG
-#     The path to the configuration file to use for Nginx. By default, the nginx
-#     configuration in this source tree is used, however this must be specified
-#     whenever running from a host where the source tree is not available, or
-#     when a fundamentally different configuration is desired.
+#     The path to the configuration file to use for Nginx. By default, './nginx.conf'
+#     is used, however this must be specified whenever running from a host where the
+#     source tree is not available, or when a fundamentally different configuration
+#     is desired.
 #
-#   LNT_EXTERNAL_PORT
-#     The externally-visible port that will be exposed by the service to interact
-#     with the webserver. This defaults to '8000', which is consistent with what
-#     LNT uses by default for local servers and development.
+#   LNT_NGINX_EXTERNAL_PORT
+#     The externally-visible port that will be exposed by the Nginx reverse proxy.
+#     This defaults to '80', which is the convention for HTTP servers.
 
 name: llvm-lnt
 
 services:
   webserver:
     container_name: webserver
-    image: ghcr.io/llvm/llvm-lnt:${LNT_IMAGE:-latest}
-    # Useful for local debugging
-    # build:
-    #   context: ../
-    #   dockerfile: docker/lnt.dockerfile
+    build:
+      context: ../
+      dockerfile: docker/lnt.dockerfile
     environment:
       - DB_USER=lntuser
       - DB_HOST=dbserver
@@ -56,9 +55,7 @@ services:
         condition: on-failure
     volumes:
       - instance:/var/lib/lnt
-    # Only expose this port to the internal Docker network, but not to
-    # the outside world.
-    expose:
+    ports:
       - "8000"
     networks:
       - lnt_network
@@ -78,17 +75,16 @@ services:
       - lnt_network
 
   nginx:
+    profiles: [nginx]
     container_name: nginx_reverse_proxy
     image: docker.io/nginx:latest
     restart: unless-stopped
     volumes:
       - ${LNT_NGINX_CONFIG:-./nginx.conf}:/etc/nginx/conf.d/default.conf:ro
     depends_on:
       - webserver
-    # The Nginx server listens to port 80 inside the container, but exposes
-    # the specified port to the outside world.
     ports:
-      - "${LNT_EXTERNAL_PORT:-8000}:80"
+      - "${LNT_NGINX_EXTERNAL_PORT:-80}:80"
     networks:
       - lnt_network
     logging:

docker/compose.yaml

@@ -0,0 +1,2 @@
+LNT_DB_PASSWORD=dev-password
+LNT_AUTH_TOKEN=dev-token

docker/dev.env

@@ -1,4 +1,4 @@
-# This Dockerfile defines an image that contains a production LNT server.
+# This Dockerfile defines an image that contains a minimal LNT server.
 # It requires additional information passed as environment variables:
 #
 #   DB_USER

docker/lnt.dockerfile

@@ -3,65 +3,51 @@
 Running a LNT Server
 ====================
 
-Running a LNT server locally is easy and can be sufficient for basic tasks. To do
-so:
+We provide a Docker Compose service that brings up a LNT web server attached
+to a Postgres database. To start the server, run::
 
-#. Install ``lnt`` as explained in the :ref:`installation section <installation>`.
-
-#. Create a LNT installation::
-
-    lnt create path/to/installation
-
-   This will create the LNT configuration file and the default database at the
-   specified path.
-
-#. You can then run the server on that installation::
+   docker compose -f docker/compose.yaml --env-file docker/dev.env up
 
-    lnt runserver path/to/installation
+Once the server is running, you are ready to submit data to it. See the section
+on :ref:`importing data <importing_data>` for details.
 
-   Note that running the server in this way is not recommended for production, since
-   this server is single-threaded and uses a SQLite database.
+The ``dev.env`` file provides default secrets suitable for local use. For your
+own deployment, create an env file with your own values for ``LNT_DB_PASSWORD``
+and ``LNT_AUTH_TOKEN``. Refer to the Docker Compose file for all available
+environment variables.
 
-#. You are now ready to submit data to the server. See the section on :ref:`importing data <importing_data>`
-   for details.
+Adding an Nginx Reverse Proxy
+-----------------------------
 
-#. While the above is enough for most use cases, you can also customize your installation.
-   To do so, edit the generated ``lnt.cfg``, for example to:
+To add an Nginx reverse proxy in front of the webserver (as done for the
+lnt.llvm.org deployment), use the ``nginx`` profile::
 
-   a. Update the databases list.
-   b. Update the public URL the server is visible at.
-   c. Update the ``nt_emailer`` configuration.
+   docker compose -f docker/compose.yaml --profile nginx --env-file <env-file> up
 
+This starts the Nginx service on port 80 (configurable via
+``LNT_NGINX_EXTERNAL_PORT``) in addition to the database and webserver.
 
-Server Architecture
--------------------
+Running without Docker
+----------------------
 
-The LNT web app is currently implemented as a Flask WSGI web app, with Jinja2
-for the templating engine. The hope is to eventually move to a more AJAXy web
-interface. The database layer uses SQLAlchemy for its ORM, and is typically
-backed by SQLite or Postgres.
+Running a ``lnt`` server outside Docker is possible too:
 
-Running a Production Server on Docker
--------------------------------------
+#. Install ``lnt`` as explained in the :ref:`installation section <installation>`.
 
-We provide a Docker Compose service that can be used to easily bring up a fully working
-production server within minutes. The service can be built and run with::
+#. Create a LNT installation::
 
-   docker compose --file docker/compose.yaml --env-file <env-file> up
+      lnt create path/to/installation
 
-``<env-file>`` should be the path to a file containing environment variables
-required by the containers. Please refer to the Docker Compose file for details.
-This service runs a Nginx server that acts as a reverse proxy for the LNT web
-server, which is itself attached to a Postgres database. For production use, we
-recommend using this service and tweaking the desired aspects in your custom setup
-(for example redirecting ports or changing volume binds).
+#. You can then run the server on that installation::
 
-Rebuilding the LNT Webserver Docker Image
------------------------------------------
+      lnt runserver path/to/installation
 
-By default, the Docker compose setup will use a published version of the LNT
-Docker image from ghcr.io. To use a locally-built Docker image instead, use::
+Note that running the server in this way is not recommended for production, since
+it runs a single-threaded Flask server instead of running behind ``gunicorn``.
 
-   docker build --file docker/lnt.dockerfile .
+Server Architecture
+-------------------
 
-and then replace the Docker image in the Docker compose file.
+The LNT web app is implemented as a Flask WSGI web app, with Jinja2 for the
+templating engine. The database layer uses SQLAlchemy for its ORM, backed by
+Postgres.