From f65c1594c3b92718ee51d92d2f6a3447f3799d9a Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 18:12:17 -0600 Subject: [PATCH 01/12] docs(connections): match the connections page to the shipped server Four fixes, all verified against the published @malloy-publisher/server@0.0.228. Environment variable substitution: the page said Publisher does not support it and told readers to put real credential values in the config file. Both halves are wrong. config.ts substitutes any ${VAR} in a string value, including package locations, so credentials can stay in the environment. Documented the actual matching rule (uppercase, braced only) and the failure mode when a variable is missing: the config silently fails to load and the server comes up serving nothing while /api/v0/status still reports "serving". Reserved connection name: the attached-database examples declared an environment-level connection named "duckdb", which is reserved for the per-package sandbox. Booting that config logs "Error initializing environment ...; skipping environment" and drops the entire environment, so the server reports "serving" with zero environments. Renamed to shared_duckdb, updated the Malloy source line that referenced it, and added a note so the name does not get changed back. The parquet example's duckdb.table(...) is the package sandbox and is left alone. Presto: not a supported connection type. The set is postgres, bigquery, snowflake, trino, databricks, mysql, duckdb, motherduck, ducklake. Dropped the sentence and retitled the section to Trino. Config key: the examples used the deprecated top-level "projects" key. It still works via back-compat, but it warns on every boot and this page is where readers copy their config from, so the examples now use "environments". --- .../publishing/connections.malloynb | 70 ++++++++++++------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index 46041033..5e7b22a9 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -9,13 +9,24 @@ Configure database connections for Malloy Publisher deployments. For an overview Publisher uses `publisher.config.json` for database connections. This file lives in your server root directory. -**Important:** Publisher does not support environment variable substitution. Put actual credential values directly in the config file. +**Keeping credentials out of the file:** any `${VAR}` inside a string value is replaced with that environment variable's value when the config loads, so passwords and keys can stay in the environment: + +```json +"postgresConnection": { + "host": "${PG_HOST}", + "password": "${PG_PASSWORD}" +} +``` + +Substitution applies to every string in the config, including package `location` values. Only uppercase, braced names are matched: `$PG_PASSWORD` (no braces) and `${lowercase}` are left as literal text. + +Set every variable your config references before starting. If one is missing, the whole config fails to load and Publisher comes up serving nothing, which is easy to miss because `/api/v0/status` still reports `"operationalState": "serving"`. Check `/api/v0/environments` to confirm your environments actually loaded. ### Basic Structure ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -53,7 +64,7 @@ For packages with embedded parquet/CSV files, **no connection configuration is n ```json { - "projects": [ + "environments": [ { "name": "default", "packages": [ @@ -88,16 +99,18 @@ my-analytics/ DuckDB can federate queries to external databases (BigQuery, Snowflake, PostgreSQL) using attached databases. This lets you query cloud data warehouses through DuckDB. +**Note:** the connection name `duckdb` is reserved for the per-package sandbox described above, so an environment-level DuckDB connection needs a different name or the server refuses to start. These examples use `shared_duckdb`; reference that same name in your Malloy code. + **Attach BigQuery:** ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ { - "name": "duckdb", + "name": "shared_duckdb", "type": "duckdb", "duckdbConnection": { "attachedDatabases": [ @@ -121,19 +134,19 @@ DuckDB can federate queries to external databases (BigQuery, Snowflake, PostgreS In your Malloy model: ```malloy -source: events is duckdb.table('my_bq.my_dataset.events') +source: events is shared_duckdb.table('my_bq.my_dataset.events') ``` **Attach Snowflake:** ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ { - "name": "duckdb", + "name": "shared_duckdb", "type": "duckdb", "duckdbConnection": { "attachedDatabases": [ @@ -162,12 +175,12 @@ source: events is duckdb.table('my_bq.my_dataset.events') ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ { - "name": "duckdb", + "name": "shared_duckdb", "type": "duckdb", "duckdbConnection": { "attachedDatabases": [ @@ -219,7 +232,7 @@ source: events is duckdb.table('my_bq.my_dataset.events') ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -244,11 +257,11 @@ source: events is duckdb.table('my_bq.my_dataset.events') **With service account (recommended for production):** -Embed the service account JSON directly in the config: +The service account JSON goes in `serviceAccountKeyJson` as a string. To keep the key out of the file, put it in an environment variable and reference it as `${BIGQUERY_SA_JSON}`: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -258,7 +271,7 @@ Embed the service account JSON directly in the config: "bigqueryConnection": { "defaultProjectId": "my-gcp-project", "location": "US", - "serviceAccountKeyJson": "{\n \"type\": \"service_account\",\n \"project_id\": \"my-gcp-project\",\n ...rest of service account JSON...\n}" + "serviceAccountKeyJson": "${BIGQUERY_SA_JSON}" } } ], @@ -268,11 +281,20 @@ Embed the service account JSON directly in the config: } ``` +Then start Publisher with the key in the environment: + +```bash +export BIGQUERY_SA_JSON="$(cat service-account.json)" +npx @malloy-publisher/server --server_root . +``` + +The whole key can also be pasted inline as a JSON string (`"{\n \"type\": \"service_account\", ...}"`) if you would rather not use an environment variable. + **With gcloud auth (development only):** ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -299,7 +321,7 @@ Embed the service account JSON directly in the config: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -326,7 +348,7 @@ Embed the service account JSON directly in the config: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -355,7 +377,7 @@ Embed the service account JSON directly in the config: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -383,7 +405,7 @@ Embed the service account JSON directly in the config: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -407,11 +429,11 @@ Embed the service account JSON directly in the config: --- -## Trino and Presto +## Trino ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -430,17 +452,15 @@ Embed the service account JSON directly in the config: } ``` -For Presto, use `"type": "presto"` with the same configuration options. - --- ## Multi-Environment Configuration -You can configure multiple projects for different environments: +You can configure multiple environments, for example one per deployment stage: ```json { - "projects": [ + "environments": [ { "name": "staging", "connections": [ From 4694a6f45d62baea6820766a50a3099a258f1e9a Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 18:16:17 -0600 Subject: [PATCH 02/12] docs(rest-api): fix the query examples, which returned 404 Both examples on this page posted to a `queryResults/{model}` path. That path does not exist and never has, so both 404ed regardless of the environment or package you substituted. The endpoint is: POST /api/v0/environments/{env}/packages/{pkg}/models/{path}/query The JavaScript example also sent the query as a `?query=` string on a GET. The route is POST only and reads the query from a JSON body, so that request 404ed too. The documented response was wrong in a way that broke the sample code: `result` is a JSON string, not an array, so the old `return data.result` handed callers a string where they expected rows. The example now parses it, and the page says so explicitly. Added a note on `compactJson`, which is what produces the plain array of row objects the page was already showing. The examples now use the samples Publisher serves out of the box, so a reader can run them as-is against `npx @malloy-publisher/server`. Both were verified by extracting them from this file and executing them against a real server: the curl returns the documented response byte for byte, and the JavaScript returns an actual array. --- .../user_guides/publishing/rest_api.malloynb | 41 +++++++++++++------ 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/src/documentation/user_guides/publishing/rest_api.malloynb b/src/documentation/user_guides/publishing/rest_api.malloynb index 894160aa..48642c73 100644 --- a/src/documentation/user_guides/publishing/rest_api.malloynb +++ b/src/documentation/user_guides/publishing/rest_api.malloynb @@ -27,7 +27,7 @@ The REST API lets you: | Capability | Description | |------------|-------------| -| **Browse models** | List projects, packages, and models available on the server | +| **Browse models** | List environments, packages, and models available on the server | | **Get model schema** | Retrieve sources, measures, dimensions, and views defined in a model | | **Execute queries** | Run Malloy queries and get JSON results | | **Explore databases** | List schemas, tables, and columns from connected databases | @@ -37,38 +37,53 @@ The REST API lets you: ## Quick Example -Run a query against a model: +Run a query against a model. The environment and package below are the samples Publisher serves out of the box, so this works as-is against a server started with `npx @malloy-publisher/server`: ```bash -curl -X POST "http://localhost:4000/api/v0/projects/my-project/packages/ecommerce/queryResults/orders.malloy" \ +curl -X POST "http://localhost:4000/api/v0/environments/examples/packages/storefront/models/storefront.malloy/query" \ -H "Content-Type: application/json" \ - -d '{"query": "run: orders -> { aggregate: order_count, total_revenue }"}' + -d '{"query": "run: order_items -> { aggregate: order_count is count(), total_revenue is sum(sale_price) }", "compactJson": true}' ``` Response: ```json { - "result": [ - { "order_count": 12345, "total_revenue": 1234567.89 } - ] + "result": "[{\"order_count\":25356,\"total_revenue\":2098177.9700000403}]", + "resource": "/api/v0/environments/examples/packages/storefront/models/storefront.malloy/query" } ``` +Note that `result` is a **JSON string**, not an object, so parse it before use: +`JSON.parse(body.result)`. + +`compactJson: true` gives you the plain array of row objects shown above. Leave it out (the default) +and `result` contains the full Malloy result instead: the same rows plus the schema and per-cell type +metadata that the renderer needs to draw charts and tables. + --- ## Common Patterns ### Fetch and Display Data +The query goes in the request body, not the query string. The endpoint accepts POST only. + ```javascript async function fetchMetrics() { const response = await fetch( - 'http://localhost:4000/api/v0/projects/prod/packages/analytics/queryResults/orders.malloy?query=' + - encodeURIComponent('run: orders -> { aggregate: total_revenue }') + 'http://localhost:4000/api/v0/environments/examples/packages/storefront/models/storefront.malloy/query', + { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + query: 'run: order_items -> { aggregate: total_revenue is sum(sale_price) }', + compactJson: true, + }), + } ); - const data = await response.json(); - return data.result; + const body = await response.json(); + return JSON.parse(body.result); // -> [{ total_revenue: 2098177.97 }] } ``` @@ -77,8 +92,8 @@ async function fetchMetrics() { Instead of writing queries inline, reference views defined in your model: ```javascript -// Model has: view: monthly_summary is { ... } -const query = 'run: orders -> monthly_summary'; +// storefront.malloy defines: view: business_overview is { ... } +const query = 'run: order_items -> business_overview'; ``` This keeps business logic in the model, not scattered across API calls. From eb9c8c7a80297d8b5242c7900a4ceaeb08289564 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 18:24:54 -0600 Subject: [PATCH 03/12] docs(publishing): fix the quick start, which served nothing The quick start said local files need no config file. They do. Following the page exactly (create a package, run npx --server_root .) produced a server with zero environments: /api/v0/environments returned [] and the package was never discovered, because Publisher only serves packages that publisher.config.json lists. There is no directory scan. The page's own diagram already showed a publisher.config.json that step 1 never told you to create. Added the missing step, and renumbered the two that follow. Other fixes on the page: - The health check and the Compose healthcheck both used a bare /status. That is not a route. The web app's catch-all answers it with 200 and an HTML page, so "curl -f http://localhost:4000/status" succeeds even when the server has loaded nothing at all, which makes the container healthcheck incapable of failing. Both now use /api/v0/status. - "List Projects" curled /api/v1/projects. There is no v1 API, so that also fell through to the web app and returned HTML rather than JSON. - Dropped the PG_CONNECT_TIMEOUT_SECONDS row. The variable was removed from Publisher along with the code that read it; the README dropped the same row already. - The Docker examples mounted packages over /publisher/packages, which is the server's own code inside the image. - Repointed a dead README#configuration anchor at docs/configuration.md. - "Restart to pick up model edits" does not work: Publisher serves a copy, so a plain restart still serves the old model. Documented --init and --watch-env, and noted that --init deletes publisher_data without prompting. - Added gs:// and s3:// to the package locations, and updated the deprecated "projects" config key and prose. Verified against @malloy-publisher/server@0.0.228 by extracting the config blocks from this file and running the page's own commands: the quick start now yields one environment and one package, where it previously yielded none. --- .../publishing/publishing.malloynb | 99 ++++++++++++++----- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index 7f6b442c..2291f0c4 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -35,9 +35,28 @@ Create `publisher.json`: } ``` -### 2. Start Publisher +### 2. List It in a Config File -Navigate to the **parent directory** (the folder containing `my-analytics/`), then run: +Publisher only serves packages that `publisher.config.json` lists, so create one in the **parent +directory** (the folder containing `my-analytics/`): + +```json +{ + "frozenConfig": false, + "environments": [ + { + "name": "default", + "packages": [ + { "name": "my-analytics", "location": "./my-analytics" } + ] + } + ] +} +``` + +### 3. Start Publisher + +From that same parent directory, run: ```bash npx @malloy-publisher/server --server_root . @@ -46,7 +65,7 @@ npx @malloy-publisher/server --server_root . The `--server_root` should point to the directory that **contains** your package folder(s), not the package itself. ``` -projects/ ← Run npx from HERE (--server_root .) +my-workspace/ ← Run npx from HERE (--server_root .) ├── publisher.config.json ← Server configuration └── my-analytics/ ← This is your package ├── publisher.json @@ -65,19 +84,25 @@ Then open `http://localhost:4000` to explore the sample models. For alternative deployment methods (Docker, build from source), see [Deployment & Configuration](#deployment-configuration) below. -### 3. Open Browser +### 4. Open Browser -Go to `http://localhost:4000` to browse and query your models. +Go to `http://localhost:4000` to browse and query your models. If the package list is empty, the +config did not load: check the startup log rather than the status endpoint, which reports `serving` +either way. -That's it for local files. If your models use DuckDB with `.parquet`, `.csv`, or `.json` files, you're done—no config file needed. +That's it for local files. If your models use DuckDB with `.parquet`, `.csv`, or `.json` files, no +connection configuration is needed on top of the config above, because every package gets its own +DuckDB sandbox automatically. -**Note:** If you edit a `.malloy` file while Publisher is running, restart it to pick up changes. +**Note:** Publisher copies your package when it loads it, so editing a `.malloy` file does not change +what a running server serves. Restart with `--init` to pick the edit up, or add `--watch-env default` +when you start, which mounts the package in place and recompiles it as you edit. -**Want to connect to a database?** Create `publisher.config.json` in the same directory: +**Want to connect to a database?** Add a connection to the same `publisher.config.json`: ```json { - "projects": [ + "environments": [ { "name": "default", "connections": [ @@ -110,15 +135,21 @@ See [Publisher Connections](connections.malloynb) for BigQuery, Snowflake, and o ## Package Locations -Publisher can load packages from various sources: +A package's `location` can be any of these: **Local Filesystem** -- Relative paths: `./package`, `../package`, `~/package` +- Relative paths: `./package`, `../package`, `~/package` (relative paths resolve against `--server_root`) - Absolute paths: `/absolute/path/to/package` **GitHub** - `https://github.com/owner/repo/tree/branch/package-path` +**Google Cloud Storage** +- `gs://bucket/path/to/package` + +**Amazon S3** +- `s3://bucket/path/to/package` + --- ## Deployment & Configuration @@ -144,12 +175,16 @@ bun run start Mount your config and packages into the container: ```bash -docker run -p 4000:4000 \ - -v ./publisher.config.json:/publisher/publisher.config.json \ - -v ./packages:/publisher/packages \ +docker run -p 4000:4000 -p 4040:4040 \ + -v ./publisher.config.json:/publisher/publisher.config.json:ro \ + -v ./packages:/publisher/my-packages \ ms2data/malloy-publisher ``` +Mount your packages anywhere except `/publisher/packages`: that path holds the server's own code +inside the image, and mounting over it breaks the container. Point each package's `location` at +wherever you mounted them (`./my-packages/...` above). + ### Docker Compose ```yaml @@ -162,19 +197,23 @@ services: - "4000:4000" - "4040:4040" # MCP endpoint for AI agents volumes: - - ./publisher.config.json:/publisher/publisher.config.json - - ./packages:/publisher/packages + - ./publisher.config.json:/publisher/publisher.config.json:ro + - ./packages:/publisher/my-packages restart: unless-stopped healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:4000/status"] + test: ["CMD", "curl", "-f", "http://localhost:4000/api/v0/status"] interval: 30s timeout: 10s retries: 3 ``` +The healthcheck must point at `/api/v0/status`. A bare `/status` is not a route: the web app's +catch-all answers it with `200` and an HTML page, so `curl -f` succeeds no matter what state the +server is in, and the healthcheck can never fail. + ### Environment Variables -Publisher reads runtime settings from environment variables. The most common ones are below; see the [Publisher README](https://github.com/malloydata/publisher#configuration) for the full list and matching CLI flags. +Publisher reads runtime settings from environment variables. The most common ones are below; see the [configuration reference](https://github.com/malloydata/publisher/blob/main/docs/configuration.md) for the full list and matching CLI flags. | Env var | Default | Meaning | |---|---|---| @@ -183,7 +222,6 @@ Publisher reads runtime settings from environment variables. The most common one | `SERVER_ROOT` | `.` (cwd) | Directory containing `publisher.config.json`. | | `LOG_LEVEL` | `debug` | One of `error`, `warn`, `info`, `verbose`, `debug`, `silly`. | | `GOOGLE_APPLICATION_CREDENTIALS` | _unset_ | Fallback path to a GCP service-account JSON for BigQuery connections that don't include inline auth. Ignored when the connection config provides its own credentials. | -| `PG_CONNECT_TIMEOUT_SECONDS` | `5` | Connection timeout (seconds) for Postgres-backed DuckLake manifest catalogs (`materializationStorage`). Bad credentials or an unreachable host return HTTP 422 in ~5s rather than hanging the publisher. No effect on user-facing Postgres connections or non-PG catalogs (SQLite, MySQL). | --- @@ -192,15 +230,21 @@ Publisher reads runtime settings from environment variables. The most common one ### Health Check ```bash -curl http://localhost:4000/status +curl http://localhost:4000/api/v0/status ``` -### List Projects +### List Environments ```bash -curl http://localhost:4000/api/v1/projects +curl http://localhost:4000/api/v0/environments ``` +A `200` from either endpoint means the server is up, but it does not mean your packages loaded: a +config that fails to load leaves Publisher reporting `"operationalState": "serving"` with an empty +environment list. If a package is missing, count what `/api/v0/environments` returns before assuming +the server is fine, and check the startup log for `Failed to load package` or +`Error initializing environment`. + See the [REST API](rest_api.malloynb) documentation for all available endpoints. ### Test in Browser @@ -216,13 +260,13 @@ See the [REST API](rest_api.malloynb) documentation for all available endpoints. ## State Persistence -Publisher persists configuration changes in a local DuckDB database (`publisher.db`). This means changes made via the REST API—adding projects, packages, or connections—survive server restarts. +Publisher persists configuration changes in a local DuckDB database (`publisher.db`). This means changes made via the REST API—adding environments, packages, or connections—survive server restarts. ### How It Works 1. **First start**: Publisher reads `publisher.config.json` and syncs it to `publisher.db` 2. **Subsequent starts**: Publisher loads from the database, ignoring config file changes -3. **API changes**: Adding/removing projects, packages, or connections updates the database +3. **API changes**: Adding/removing environments, packages, or connections updates the database ### Reinitializing from Config @@ -238,6 +282,11 @@ Use `--init` when: - You want to reset to the original configuration - You're troubleshooting configuration issues +`--init` deletes `publisher_data/` and rebuilds it from the config file, without prompting. Anything +that only exists in the database or in Publisher's copy of a package is discarded, including +environments, packages, and connections added through the API or the UI. Keep your source packages +outside `publisher_data/` (as the layout above does) and `--init` is safe to run. + **Note:** On first run, Publisher automatically creates the database and syncs from `publisher.config.json`—no `--init` needed. ### Mutable vs Frozen Configuration @@ -247,7 +296,7 @@ By default, Publisher allows configuration changes via the API. To lock the conf ```json { "frozenConfig": true, - "projects": [...] + "environments": [...] } ``` From c46a05bb927dbee54ca2e531f958530ed4ab9817 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 18:30:50 -0600 Subject: [PATCH 04/12] docs(sdk): fix the embedding example, which crashed at render The EmbeddedQueryResult example passed optionalPackageName and optionalProjectName. Those props were removed from the SDK in September 2025 and replaced by a single resourceUri, so the example addressed nothing and threw "Invalid URL" when the component parsed it at render. Verified against the published SDK 0.0.228: the old payload throws, the new one round-trips. They are easy to miss in a grep because both names survive as local variables inside the component. The page also said the two names are only needed when the server holds multiple projects or packages. A resource URI is always required. Other fixes: - ServerProvider takes baseURL, not server. There is no server prop, so both examples were silently ignored and the provider fell back to its default. The CORS walkthrough carried the same wrong name, which is the line most local setups copy. - The default is not http://localhost:4000. The SDK derives /api/v0 on whatever origin serves the page, which is why the example app proxies through Vite. - --server_root ./malloy-samples pointed at a directory the publisher repo does not contain; packages/server holds the config that serves the example packages the app expects. - Dropped the .env line from the project layout. VITE_PUBLISHER_API appears only in .env.example and no source file reads it; the URL comes from the Vite proxy or from baseURL. - The renderer does not pick charts on its own. Results are tables unless a render tag says otherwise. - Corrected the example app's view names to the ones it actually ships. The new example was extracted from this file and typechecked against the published SDK, then exercised at runtime. --- .../publishing/publisher_sdk.malloynb | 44 +++++++++++-------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/src/documentation/user_guides/publishing/publisher_sdk.malloynb b/src/documentation/user_guides/publishing/publisher_sdk.malloynb index 9c77b9ca..db54d71c 100644 --- a/src/documentation/user_guides/publishing/publisher_sdk.malloynb +++ b/src/documentation/user_guides/publishing/publisher_sdk.malloynb @@ -14,8 +14,8 @@ The fastest way to understand the SDK is to run the example data app: git clone https://github.com/malloydata/publisher.git cd publisher -# Start the Publisher server -npx @malloy-publisher/server --server_root ./malloy-samples +# Start the Publisher server, serving the bundled example packages +npx @malloy-publisher/server --server_root packages/server # In a new terminal, run the example app cd examples/data-app @@ -27,10 +27,10 @@ npm run dev Open [http://localhost:5173](http://localhost:5173) to see a working dashboard with embedded Malloy visualizations. The example app demonstrates: -- **Malloy Samples Dashboard** – Pre-configured charts from the names dataset +- **Storefront dashboard** – A fixed grid of SDK tiles over the storefront model - **Single Embed** – Embedding a single query result -- **Dynamic Dashboard** – Adding charts at runtime -- **Interactive Dashboard** – Using raw data with custom Recharts visualizations +- **Dynamic Dashboard** – Adding and arranging widgets at runtime +- **Interactive** – Using raw data with custom Recharts visualizations --- @@ -55,34 +55,40 @@ function App() { } ``` -By default, connects to `http://localhost:4000`. For production: +By default the SDK talks to `/api/v0` on the same origin the page is served from. That is what you want when Publisher serves your app itself, and it is why the example app proxies `/api/v0` through Vite to port 4000. To point at a different server, pass `baseURL`: ```tsx - + ``` ### 2. EmbeddedQueryResult -Renders a Malloy query result. The component picks the visualization automatically—tables for flat data, bar charts for single-dimension aggregates, or whatever [renderer tag](../../language/tags.malloynb) you specify in your model (e.g., `# line_chart`). +Renders a Malloy query result. Results render as a table unless the query or the model carries a [renderer tag](../../language/tags.malloynb) (e.g. `# bar_chart`, `# line_chart`), which is what selects a visualization. ```tsx -import { EmbeddedQueryResult } from "@malloy-publisher/sdk"; +import { + EmbeddedQueryResult, + createEmbeddedQueryResult, + encodeResourceUri, +} from "@malloy-publisher/sdk"; function MyChart() { - const embeddedQuery = JSON.stringify({ - modelPath: "orders.malloy", - query: "run: orders -> { group_by: region; aggregate: total_revenue }", - optionalPackageName: "ecommerce", - optionalProjectName: "production" + const embeddedQuery = createEmbeddedQueryResult({ + query: "run: order_items -> { group_by: brand; aggregate: total_sales }", + resourceUri: encodeResourceUri({ + environmentName: "examples", + packageName: "storefront", + modelPath: "storefront.malloy", + }), }); return ; } ``` -The `optionalPackageName` and `optionalProjectName` parameters are required when your Publisher serves multiple projects or packages—they tell the SDK which model to use. +`resourceUri` is how you address a model: it names the environment, the package, and the model path. It is always required, not only when the server holds more than one package, and `createEmbeddedQueryResult` throws without it. Instead of writing the query inline, you can reference a view defined in the model by passing `queryName` together with `sourceName`. --- @@ -113,11 +119,13 @@ my-data-app/ │ │ └── widgets.json # Saved embed configurations │ └── components/ │ └── CustomChart.tsx -├── .env # VITE_PUBLISHER_API=http://localhost:4000 ├── package.json -└── vite.config.ts +└── vite.config.ts # Proxies /api/v0 to the Publisher server ``` +There is no environment variable that points the SDK at a Publisher server. Either proxy `/api/v0` in +`vite.config.ts` (below) or pass `baseURL` to `ServerProvider`. + --- ## CORS Configuration @@ -142,7 +150,7 @@ export default defineConfig({ Then use relative paths in ServerProvider: ```tsx - + ``` --- From 59188bcce95d8572ff10307822372aa401bf4886 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 18:40:03 -0600 Subject: [PATCH 05/12] docs(explorer): fix the model URL and the hover action labels The URL format left out the file extension. The web app routes :environmentName/:packageName/* and then branches on the extension, so /examples/storefront/storefront renders "Unrecognized file type: storefront" while /examples/storefront/storefront.malloy opens the Explorer. Both were loaded in a browser to confirm. Also named the first segment for what it is, an environment. Three of the documented hover actions do not exist in the shipped Explorer. Checked against the malloy-explorer build the SDK depends on: "Add as Sort", "Add to Query" and "Add as Nested Query" have no hits in the bundle, while "Add as order by", "Add view" and "Add as new nested query" do. A reader scanning the hover menu for "Sort" will not find it, because Malloy's concept is order by. Confirmed live as well: hovering a field shows "Add as group by", and the order by action reports "Order by is only available for fields in the output", which is a restriction the page did not mention. Lowercased the remaining labels to match what the UI actually renders. --- .../user_guides/publishing/explorer.malloynb | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/documentation/user_guides/publishing/explorer.malloynb b/src/documentation/user_guides/publishing/explorer.malloynb index c3bb9cd0..743eb4c9 100644 --- a/src/documentation/user_guides/publishing/explorer.malloynb +++ b/src/documentation/user_guides/publishing/explorer.malloynb @@ -19,7 +19,7 @@ Deploy Publisher ([see Publishing](publishing.malloynb)) and share the URL with Publisher model list -The URL format is: `http://localhost:4000/project/package/model` +The URL format is: `http://localhost:4000///`, where the model path includes the file extension, for example `http://localhost:4000/examples/storefront/storefront.malloy`. Anyone with access to Publisher can explore your semantic models—no Malloy knowledge required. @@ -61,9 +61,9 @@ The panel is organized into three sections: Attributes you can group by, filter on, or sort with. Dimensions are grouped by their source. For example, in a query centered around `order_items`, you might also see dimensions from joined models like `users` or `products`. Hovering over a dimension reveals actions: -- **Add as Group By** – Segment results by this dimension -- **Add as Filter** – Apply a filter based on the field -- **Add as Sort** – Sort results by this value +- **Add as group by** – Segment results by this dimension +- **Add as filter** – Apply a filter based on the field +- **Add as order by** – Sort results by this value (only available for fields already in the output) Dimension Hover Actions @@ -72,9 +72,9 @@ Hovering over a dimension reveals actions: Predefined metrics you can aggregate, filter on, or sort with. These include calculations such as totals, averages, counts, and ratios. Hovering over a measure provides actions: -- **Add as Aggregate** – Include the metric in results -- **Add as Filter** – Use the measure to restrict results -- **Add as Sort** – Sort results based on the metric value +- **Add as aggregate** – Include the metric in results +- **Add as filter** – Use the measure to restrict results +- **Add as order by** – Sort results based on the metric value (only available for fields already in the output) Measure Hover Actions @@ -83,8 +83,8 @@ Hovering over a measure provides actions: Saved queries defined in the underlying Malloy model. Views often represent curated KPIs, commonly-used explorations, or analytical building blocks. Hovering over a view provides actions: -- **Add to Query** – Load the view's query -- **Add as Nested Query** – Add the view as a nested subquery +- **Add view** – Load the view's query +- **Add as new nested query** – Add the view as a nested subquery View Hover Actions @@ -156,7 +156,7 @@ Malloy's nesting feature enables rich, multidimensional analysis—and Explorer To add a nested query: - Select **Nest Query** from the Query Panel **More Actions** menu -- Or hover over a view in the Source Panel and select **Add as Nested Query** +- Or hover over a view in the Source Panel and select **Add as new nested query** Nest Action From 7814a555a71553f64e19fe0a306834e9bd7893bc Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 20:52:49 -0600 Subject: [PATCH 06/12] docs(publishing): fix four defects found reviewing this branch Four corrections, three of them to text this branch itself added. The SDK embedding example did not compile. It grouped by `brand`, which belongs to the joined `products` source and is not reachable unqualified, so the query returned 400 "'brand' is not defined". The model's own top_brands view uses `products.brand`. The snippet typechecked against the SDK, which is why this survived: the types were right and the query inside them was not. Every Malloy query on these pages is now executed against the shipped storefront package. The reserved-connection-name note said the server "refuses to start" when an environment-level connection is named `duckdb`. It does not. It starts, logs the error, skips the whole environment, and keeps reporting "serving" with nothing loaded, which is the same silent failure these pages warn about two sections earlier. Corrected to the real behaviour. Removed `~/package` from the package locations. Publisher never expands a tilde (no homedir handling exists), so a `~/...` location makes the server stat a literal `~` directory and the package fails to load. The claim was pre-existing, but this branch added "resolved against --server_root" next to it, which makes it self-refuting. Dropped `cp .env.example .env` from the example-app steps, which contradicted the new line stating no environment variable points the SDK at a server, and made the JavaScript comment show the exact value the endpoint returns rather than a rounded one, since the page prints the raw float a few lines earlier on purpose. --- src/documentation/user_guides/publishing/connections.malloynb | 2 +- .../user_guides/publishing/publisher_sdk.malloynb | 3 +-- src/documentation/user_guides/publishing/publishing.malloynb | 2 +- src/documentation/user_guides/publishing/rest_api.malloynb | 2 +- 4 files changed, 4 insertions(+), 5 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index 5e7b22a9..5506d623 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -99,7 +99,7 @@ my-analytics/ DuckDB can federate queries to external databases (BigQuery, Snowflake, PostgreSQL) using attached databases. This lets you query cloud data warehouses through DuckDB. -**Note:** the connection name `duckdb` is reserved for the per-package sandbox described above, so an environment-level DuckDB connection needs a different name or the server refuses to start. These examples use `shared_duckdb`; reference that same name in your Malloy code. +**Note:** the connection name `duckdb` is reserved for the per-package sandbox described above, so an environment-level DuckDB connection needs a different name. Using `duckdb` here does not stop the server: it logs the error, skips that whole environment, and carries on reporting `"operationalState": "serving"` with nothing loaded. These examples use `shared_duckdb`; reference that same name in your Malloy code. **Attach BigQuery:** diff --git a/src/documentation/user_guides/publishing/publisher_sdk.malloynb b/src/documentation/user_guides/publishing/publisher_sdk.malloynb index db54d71c..8b658cf1 100644 --- a/src/documentation/user_guides/publishing/publisher_sdk.malloynb +++ b/src/documentation/user_guides/publishing/publisher_sdk.malloynb @@ -19,7 +19,6 @@ npx @malloy-publisher/server --server_root packages/server # In a new terminal, run the example app cd examples/data-app -cp .env.example .env npm install npm run dev ``` @@ -76,7 +75,7 @@ import { function MyChart() { const embeddedQuery = createEmbeddedQueryResult({ - query: "run: order_items -> { group_by: brand; aggregate: total_sales }", + query: "run: order_items -> { group_by: products.brand; aggregate: total_sales }", resourceUri: encodeResourceUri({ environmentName: "examples", packageName: "storefront", diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index 2291f0c4..27fc2406 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -138,7 +138,7 @@ See [Publisher Connections](connections.malloynb) for BigQuery, Snowflake, and o A package's `location` can be any of these: **Local Filesystem** -- Relative paths: `./package`, `../package`, `~/package` (relative paths resolve against `--server_root`) +- Relative paths: `./package`, `../package` (resolved against `--server_root`) - Absolute paths: `/absolute/path/to/package` **GitHub** diff --git a/src/documentation/user_guides/publishing/rest_api.malloynb b/src/documentation/user_guides/publishing/rest_api.malloynb index 48642c73..3fe380ca 100644 --- a/src/documentation/user_guides/publishing/rest_api.malloynb +++ b/src/documentation/user_guides/publishing/rest_api.malloynb @@ -83,7 +83,7 @@ async function fetchMetrics() { } ); const body = await response.json(); - return JSON.parse(body.result); // -> [{ total_revenue: 2098177.97 }] + return JSON.parse(body.result); // -> [{ total_revenue: 2098177.9700000403 }] } ``` From 19405100f9cd049480d8d156d9b697061cba971f Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 20:55:45 -0600 Subject: [PATCH 07/12] docs(publishing): correct the credential guidance and three overstated claims The environment-variable section told readers that ${VAR} keeps credentials out of the config file, and left it there. That is true but it is not the whole story, and the missing half is the security-relevant one: Publisher's connection endpoints return the resolved configuration, password included, to anyone who can reach them, and Publisher ships with no authentication and binds all interfaces. Verified against 0.0.228 by putting a sentinel password in an environment variable and reading it back in plaintext from an unauthenticated GET /api/v0/environments/default/connections. Substituting a variable protects the file and your git history, not the server, and the page now says so. Also noted that a variable which is set but empty is not an error: only a completely unset one fails the load, so a blank ${PG_PASSWORD} becomes a blank password rather than a failure. Three claims this branch added were overstated: - The Compose healthcheck note implied /api/v0/status reports whether the server is serving anything. It returns 200 as soon as the process is up, including with zero environments loaded, so it checks liveness only. - The troubleshooting step named two log strings, neither of which is emitted when the config itself fails to parse. That case logs "Error reading publisher.config.json", which is exactly the case a missing variable produces. - The REST quick example works against the bundled samples only for a zero-argument start, because any --server_root or --config disables the bundled default config. The precondition is now stated. Finally, the Vite proxy example forwarded /api while the surrounding prose and the real example app both use /api/v0. --- .../user_guides/publishing/connections.malloynb | 6 ++++-- .../user_guides/publishing/publisher_sdk.malloynb | 2 +- .../user_guides/publishing/publishing.malloynb | 10 ++++++---- .../user_guides/publishing/rest_api.malloynb | 2 +- 4 files changed, 12 insertions(+), 8 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index 5506d623..5f2a0a03 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -9,7 +9,7 @@ Configure database connections for Malloy Publisher deployments. For an overview Publisher uses `publisher.config.json` for database connections. This file lives in your server root directory. -**Keeping credentials out of the file:** any `${VAR}` inside a string value is replaced with that environment variable's value when the config loads, so passwords and keys can stay in the environment: +**Keeping credentials out of the file:** any `${VAR}` inside a string value is replaced with that environment variable's value when the config loads, so passwords and keys can stay out of the file and out of version control: ```json "postgresConnection": { @@ -20,7 +20,9 @@ Publisher uses `publisher.config.json` for database connections. This file lives Substitution applies to every string in the config, including package `location` values. Only uppercase, braced names are matched: `$PG_PASSWORD` (no braces) and `${lowercase}` are left as literal text. -Set every variable your config references before starting. If one is missing, the whole config fails to load and Publisher comes up serving nothing, which is easy to miss because `/api/v0/status` still reports `"operationalState": "serving"`. Check `/api/v0/environments` to confirm your environments actually loaded. +This keeps credentials out of the file, not out of the server. Publisher's connection endpoints return the resolved configuration, password included, to anyone who can reach them, and Publisher ships with no authentication and binds all interfaces by default. A Publisher that holds database credentials belongs on a trusted network or behind your own auth. + +Set every variable your config references before starting. If one is unset, the whole config fails to load and Publisher comes up serving nothing, which is easy to miss because `/api/v0/status` still reports `"operationalState": "serving"`. Check `/api/v0/environments` to confirm your environments actually loaded. A variable that is set but empty is not an error: it substitutes an empty string, so a blank `${PG_PASSWORD}` becomes a blank password rather than a failure. ### Basic Structure diff --git a/src/documentation/user_guides/publishing/publisher_sdk.malloynb b/src/documentation/user_guides/publishing/publisher_sdk.malloynb index 8b658cf1..73353c71 100644 --- a/src/documentation/user_guides/publishing/publisher_sdk.malloynb +++ b/src/documentation/user_guides/publishing/publisher_sdk.malloynb @@ -137,7 +137,7 @@ If your React app runs on a different port/domain than Publisher, configure CORS export default defineConfig({ server: { proxy: { - '/api': { + '/api/v0': { target: 'http://localhost:4000', changeOrigin: true, }, diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index 27fc2406..fc05d8fd 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -208,8 +208,9 @@ services: ``` The healthcheck must point at `/api/v0/status`. A bare `/status` is not a route: the web app's -catch-all answers it with `200` and an HTML page, so `curl -f` succeeds no matter what state the -server is in, and the healthcheck can never fail. +catch-all answers it with `200` and an HTML page, so `curl -f` passes even when the API is not +serving at all. Note that `/api/v0/status` returns `200` as soon as the server is up, including when +no environment loaded, so this checks liveness rather than that your packages are being served. ### Environment Variables @@ -242,8 +243,9 @@ curl http://localhost:4000/api/v0/environments A `200` from either endpoint means the server is up, but it does not mean your packages loaded: a config that fails to load leaves Publisher reporting `"operationalState": "serving"` with an empty environment list. If a package is missing, count what `/api/v0/environments` returns before assuming -the server is fine, and check the startup log for `Failed to load package` or -`Error initializing environment`. +the server is fine, and check the startup log, where the cause is named: `Failed to load package` +for one bad package, `Error initializing environment` when a whole environment is dropped, and +`Error reading publisher.config.json` when the config itself could not be parsed. See the [REST API](rest_api.malloynb) documentation for all available endpoints. diff --git a/src/documentation/user_guides/publishing/rest_api.malloynb b/src/documentation/user_guides/publishing/rest_api.malloynb index 3fe380ca..db10edbc 100644 --- a/src/documentation/user_guides/publishing/rest_api.malloynb +++ b/src/documentation/user_guides/publishing/rest_api.malloynb @@ -37,7 +37,7 @@ The REST API lets you: ## Quick Example -Run a query against a model. The environment and package below are the samples Publisher serves out of the box, so this works as-is against a server started with `npx @malloy-publisher/server`: +Run a query against a model. The environment and package below are the samples Publisher serves out of the box, so this works as-is against a server started with no arguments (`npx @malloy-publisher/server`). If you passed `--server_root` or `--config`, substitute your own environment, package and model names. ```bash curl -X POST "http://localhost:4000/api/v0/environments/examples/packages/storefront/models/storefront.malloy/query" \ From 983b2c55c4d0af2fb4286cfb8400195f817d4879 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Thu, 16 Jul 2026 21:30:49 -0600 Subject: [PATCH 08/12] docs(publishing): fix the example-app steps, watch mode, and the log strings More defects found reviewing this branch, most of them in text the branch itself added. The example-app walkthrough crashed. examples/data-app resolves react and @emotion from the repo-root node_modules (its vite config aliases them there), but the steps never installed at the root. Reproduced on a fresh checkout: bun run dev exits 1 with "Cannot read file" for each aliased package; with a root install present the dev server comes up on 5173 with no alias errors. The example also needs its own install, since it is not part of the workspace, so both are now listed. Use bun at the root: the repo ships only bun.lock and its engines require bun. Watch mode was documented as an alternative to --init when it is not one. Once Publisher has copied a package into publisher_data, adding --watch-env on its own does nothing: the in-place mount is only set up when that copy is absent, so edits are still ignored, and /api/v0/watch-mode/status reports "enabled": true the whole time. Verified by isolating the flag: --watch-env alone leaves the old result, --init --watch-env picks the edit up live. The instruction is now --init --watch-env. The troubleshooting strings were mapped to the wrong causes. A config that fails to parse logs "Failed to parse", never "Error reading publisher.config.json"; that second string is what an unset ${VAR} produces. A reader with a malformed config would have grepped for a string the server never prints. Both are now listed against the failure that emits them. The credential note scoped the exposure to the connection endpoints. It is wider: /api/v0/status and /api/v0/environments return the resolved password too, and /api/v0/status is the endpoint this guide tells you to health-check. Verified by reading a sentinel password back from each. The deployment section, which had no authentication caveat at all, now carries one. Also corrected two Explorer labels the earlier sweep missed: the menu item is "Nest query", and the Add Query Element list is sentence case like the rest. --- .../publishing/connections.malloynb | 2 +- .../user_guides/publishing/explorer.malloynb | 14 ++++++------- .../publishing/publisher_sdk.malloynb | 11 ++++++---- .../publishing/publishing.malloynb | 20 +++++++++++++++---- 4 files changed, 31 insertions(+), 16 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index 5f2a0a03..ee4500f5 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -20,7 +20,7 @@ Publisher uses `publisher.config.json` for database connections. This file lives Substitution applies to every string in the config, including package `location` values. Only uppercase, braced names are matched: `$PG_PASSWORD` (no braces) and `${lowercase}` are left as literal text. -This keeps credentials out of the file, not out of the server. Publisher's connection endpoints return the resolved configuration, password included, to anyone who can reach them, and Publisher ships with no authentication and binds all interfaces by default. A Publisher that holds database credentials belongs on a trusted network or behind your own auth. +This keeps credentials out of the file, not out of the server. Publisher resolves `${VAR}` at load time and then serves the resolved connection configuration, password included, from its API: `/api/v0/environments/{environment}/connections` returns it, and so do `/api/v0/environments` and `/api/v0/status`, the endpoint most people wire into a health check. Publisher ships with no authentication and binds all interfaces by default, so anyone who can reach the port can read your database credentials. A Publisher that holds credentials belongs on a trusted network or behind your own authentication. Set every variable your config references before starting. If one is unset, the whole config fails to load and Publisher comes up serving nothing, which is easy to miss because `/api/v0/status` still reports `"operationalState": "serving"`. Check `/api/v0/environments` to confirm your environments actually loaded. A variable that is set but empty is not an error: it substitutes an empty string, so a blank `${PG_PASSWORD}` becomes a blank password rather than a failure. diff --git a/src/documentation/user_guides/publishing/explorer.malloynb b/src/documentation/user_guides/publishing/explorer.malloynb index 743eb4c9..6f14e989 100644 --- a/src/documentation/user_guides/publishing/explorer.malloynb +++ b/src/documentation/user_guides/publishing/explorer.malloynb @@ -108,13 +108,13 @@ The Query Panel is where queries come together. It provides a structured, visual This menu lets you add fields by operation type: -- Add Group By -- Add Aggregate -- Add Filter -- Add View +- Add group by +- Add aggregate +- Add filter +- Add view - Limit -- Order By -- Add Blank Nested Query +- Order by +- Add blank nested query Add Query Element @@ -155,7 +155,7 @@ Hovering over a section shows additional ways to add elements. Malloy's nesting feature enables rich, multidimensional analysis—and Explorer gives you a no-code way to use it. To add a nested query: -- Select **Nest Query** from the Query Panel **More Actions** menu +- Select **Nest query** from the Query Panel **More Actions** menu - Or hover over a view in the Source Panel and select **Add as new nested query** Nest Action diff --git a/src/documentation/user_guides/publishing/publisher_sdk.malloynb b/src/documentation/user_guides/publishing/publisher_sdk.malloynb index 73353c71..ae8fbf1c 100644 --- a/src/documentation/user_guides/publishing/publisher_sdk.malloynb +++ b/src/documentation/user_guides/publishing/publisher_sdk.malloynb @@ -10,17 +10,20 @@ Build custom data applications with the Publisher SDK. Embed live, governed anal The fastest way to understand the SDK is to run the example data app: ```bash -# Clone the publisher repo +# Clone the publisher repo and install at the root. The example app resolves +# react and @emotion from the repo-root node_modules, so this step is required. git clone https://github.com/malloydata/publisher.git cd publisher +bun install # Start the Publisher server, serving the bundled example packages npx @malloy-publisher/server --server_root packages/server -# In a new terminal, run the example app +# In a new terminal, install and run the example app. It is not part of the +# workspace, so it needs its own install as well as the one above. cd examples/data-app -npm install -npm run dev +bun install +bun run dev ``` Open [http://localhost:5173](http://localhost:5173) to see a working dashboard with embedded Malloy visualizations. diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index fc05d8fd..eee20296 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -95,8 +95,14 @@ connection configuration is needed on top of the config above, because every pac DuckDB sandbox automatically. **Note:** Publisher copies your package when it loads it, so editing a `.malloy` file does not change -what a running server serves. Restart with `--init` to pick the edit up, or add `--watch-env default` -when you start, which mounts the package in place and recompiles it as you edit. +what a running server serves. Restart with `--init` to re-copy from source and pick the edit up. + +While you are iterating on a model, restart with `--init --watch-env default` instead: that mounts +the package in place and recompiles it as you save. `--watch-env` needs `--init` here, because the +in-place mount is only set up when Publisher has not already copied the package into +`publisher_data/`. Adding `--watch-env` on its own to a server root you have already started does +nothing, and does so quietly: `/api/v0/watch-mode/status` still reports `"enabled": true` while your +edits are ignored. **Want to connect to a database?** Add a connection to the same `publisher.config.json`: @@ -185,6 +191,10 @@ Mount your packages anywhere except `/publisher/packages`: that path holds the s inside the image, and mounting over it breaks the container. Point each package's `location` at wherever you mounted them (`./my-packages/...` above). +Publisher has no built-in authentication and binds all interfaces, and its API serves your +connection configuration, database passwords included. Publish it only on a trusted network, or put +your own authentication in front of it. The MCP port (4040) is unauthenticated too. + ### Docker Compose ```yaml @@ -244,8 +254,10 @@ A `200` from either endpoint means the server is up, but it does not mean your p config that fails to load leaves Publisher reporting `"operationalState": "serving"` with an empty environment list. If a package is missing, count what `/api/v0/environments` returns before assuming the server is fine, and check the startup log, where the cause is named: `Failed to load package` -for one bad package, `Error initializing environment` when a whole environment is dropped, and -`Error reading publisher.config.json` when the config itself could not be parsed. +for one bad package, `Error initializing environment` when a whole environment is dropped, +`Failed to parse` when `publisher.config.json` is not valid JSON, and +`Error reading publisher.config.json` when the config parsed but could not be processed, which is +usually a `${VAR}` you have not set. See the [REST API](rest_api.malloynb) documentation for all available endpoints. From 892f719c26b782a36774ab4f3deb7b04e9e94137 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Fri, 17 Jul 2026 06:06:05 -0600 Subject: [PATCH 09/12] docs(publishing): widen the security note beyond credential disclosure The caveat this branch added was scoped to credentials, and the scoping let the most-recommended deployment off the hook. "A Publisher that holds credentials belongs on a trusted network" implicitly exempts the DuckDB path these guides push hardest, the one where no connection is configured at all. That deployment is exposed too. Verified against the published server, unauthenticated, on a config declaring no connections whatsoever: POST /api/v0/environments/default/packages/pkg/connections/duckdb/sqlQuery {"sqlStatement":"SELECT * FROM read_csv_auto('/tmp/secret-probe.csv')"} -> 200 {"rows":[{"col":"SENSITIVE-HOST-FILE-CONTENTS"},...]} The DuckDB sandbox every package gets automatically will read files off the server's filesystem, so the exposure does not depend on having a database configured. The API is also read-write by default, which the caveat did not say. An anonymous POST created a connection (201) and the change persists to publisher.db. frozenConfig: true is what closes that (403), so the Mutable vs Frozen section now names it as a security control for any Publisher reachable beyond localhost, rather than a production nicety. It is not sufficient on its own: with frozenConfig: true the same arbitrary SQL still runs, so the port still has to be kept off untrusted networks. --- .../user_guides/publishing/connections.malloynb | 4 +++- .../user_guides/publishing/publishing.malloynb | 15 +++++++++++---- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index ee4500f5..90f823a4 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -20,7 +20,9 @@ Publisher uses `publisher.config.json` for database connections. This file lives Substitution applies to every string in the config, including package `location` values. Only uppercase, braced names are matched: `$PG_PASSWORD` (no braces) and `${lowercase}` are left as literal text. -This keeps credentials out of the file, not out of the server. Publisher resolves `${VAR}` at load time and then serves the resolved connection configuration, password included, from its API: `/api/v0/environments/{environment}/connections` returns it, and so do `/api/v0/environments` and `/api/v0/status`, the endpoint most people wire into a health check. Publisher ships with no authentication and binds all interfaces by default, so anyone who can reach the port can read your database credentials. A Publisher that holds credentials belongs on a trusted network or behind your own authentication. +This keeps credentials out of the file, not out of the server. Publisher resolves `${VAR}` at load time and then serves the resolved connection configuration, password included, from its API: `/api/v0/environments/{environment}/connections` returns it, and so do `/api/v0/environments` and `/api/v0/status`, the endpoint most people wire into a health check. + +Publisher ships with no authentication and binds all interfaces by default, so anyone who can reach the port can read those credentials, run SQL through any connection you have configured, and (unless you set `frozenConfig: true`) add connections of their own. This is not only a concern for servers that hold credentials: the DuckDB sandbox every package gets automatically will also read files off the server's filesystem, so even a Publisher with no connections configured at all exposes its host. Keep Publisher on a trusted network or behind your own authentication. Set every variable your config references before starting. If one is unset, the whole config fails to load and Publisher comes up serving nothing, which is easy to miss because `/api/v0/status` still reports `"operationalState": "serving"`. Check `/api/v0/environments` to confirm your environments actually loaded. A variable that is set but empty is not an error: it substitutes an empty string, so a blank `${PG_PASSWORD}` becomes a blank password rather than a failure. diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index eee20296..485900b3 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -191,9 +191,12 @@ Mount your packages anywhere except `/publisher/packages`: that path holds the s inside the image, and mounting over it breaks the container. Point each package's `location` at wherever you mounted them (`./my-packages/...` above). -Publisher has no built-in authentication and binds all interfaces, and its API serves your -connection configuration, database passwords included. Publish it only on a trusted network, or put -your own authentication in front of it. The MCP port (4040) is unauthenticated too. +Publisher has no built-in authentication and binds all interfaces. Anyone who can reach the port can +read your connection configuration (database passwords included), run SQL through any connection, and +by default add connections of their own, with those changes persisted to `publisher.db`. The DuckDB +sandbox each package gets automatically will read files off the server's filesystem, so this applies +even with no database configured. Publish Publisher only on a trusted network, or put your own +authentication in front of it. The MCP port (4040) is unauthenticated too. ### Docker Compose @@ -305,7 +308,11 @@ outside `publisher_data/` (as the layout above does) and `--init` is safe to run ### Mutable vs Frozen Configuration -By default, Publisher allows configuration changes via the API. To lock the configuration (e.g., in production): +By default, Publisher allows configuration changes via the API, and that default is unauthenticated: +any caller who can reach the port can add or change a connection, and the change survives restarts. +Set `frozenConfig: true` on any Publisher reachable beyond your own machine. It is the control that +closes config writes (they return `403`), though it does not stop queries, so it is not a substitute +for keeping the port off untrusted networks: ```json { From 6d3a3b4b31d2cfbf2b4d38ce082982b779f0e498 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Fri, 17 Jul 2026 06:32:02 -0600 Subject: [PATCH 10/12] docs(publishing): point the healthcheck at /health instead of /api/v0/status This branch fixed the Compose healthcheck from a bare /status to /api/v0/status, and then documented, on the connections page, that /api/v0/status serves the resolved connection configuration with the password in it. Both statements are true, which made the healthcheck a standing instruction to fetch database credentials every thirty seconds and hand the output to Docker, which keeps it in the container's health log. /health exists in the same build, returns {"status":"UP"} and nothing else, and works with curl -f. Verified with a sentinel password in the config: /api/v0/status returns it, /health, /health/readiness and /health/liveness do not. The Compose healthcheck and the Verify It Works check now use /health, and the note explains why rather than only warning about the bare /status route. The environment listing still shows connection configuration, so the guide keeps recommending /api/v0/environments only as something you run yourself against your own server, which is also what the connections page now describes. --- .../user_guides/publishing/publishing.malloynb | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index 485900b3..ad03e625 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -214,16 +214,18 @@ services: - ./packages:/publisher/my-packages restart: unless-stopped healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:4000/api/v0/status"] + test: ["CMD", "curl", "-f", "http://localhost:4000/health"] interval: 30s timeout: 10s retries: 3 ``` -The healthcheck must point at `/api/v0/status`. A bare `/status` is not a route: the web app's -catch-all answers it with `200` and an HTML page, so `curl -f` passes even when the API is not -serving at all. Note that `/api/v0/status` returns `200` as soon as the server is up, including when -no environment loaded, so this checks liveness rather than that your packages are being served. +Use `/health` for the container healthcheck, not `/status` and not `/api/v0/status`. A bare `/status` +is not a route at all: the web app's catch-all answers it with `200` and an HTML page, so `curl -f` +passes even when the API is not serving. `/api/v0/status` is a real endpoint, but its response +includes your connection configuration, so a healthcheck pointed at it writes database passwords into +the container's health log every interval. `/health` returns just `{"status":"UP"}`. Either way this +checks that the server is up, not that your packages loaded. ### Environment Variables @@ -244,7 +246,7 @@ Publisher reads runtime settings from environment variables. The most common one ### Health Check ```bash -curl http://localhost:4000/api/v0/status +curl http://localhost:4000/health ``` ### List Environments From c22794f9546cb63daf3ff73cd76e3c349bb48988 Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Fri, 17 Jul 2026 06:50:26 -0600 Subject: [PATCH 11/12] docs(publishing): describe /health's response instead of quoting the wrong one The healthcheck note said /health returns just {"status":"UP"}. That is /health/liveness's body. /health returns a liveness and readiness object with components and groups. The point the sentence makes is unaffected, since neither body contains connection configuration, but it quoted a response the server does not produce, so it now describes the response rather than pinning a literal. --- src/documentation/user_guides/publishing/publishing.malloynb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index ad03e625..f8c0c7f0 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -224,8 +224,9 @@ Use `/health` for the container healthcheck, not `/status` and not `/api/v0/stat is not a route at all: the web app's catch-all answers it with `200` and an HTML page, so `curl -f` passes even when the API is not serving. `/api/v0/status` is a real endpoint, but its response includes your connection configuration, so a healthcheck pointed at it writes database passwords into -the container's health log every interval. `/health` returns just `{"status":"UP"}`. Either way this -checks that the server is up, not that your packages loaded. +the container's health log every interval. `/health` reports liveness and readiness and nothing else, +with no configuration in it. Either way this checks that the server is up, not that your packages +loaded. ### Environment Variables From ed0fabe50bf8bca50553436a6c7916cdc502306b Mon Sep 17 00:00:00 2001 From: Monty Lennie Date: Mon, 20 Jul 2026 15:18:50 -0600 Subject: [PATCH 12/12] docs(publishing): correct how a package location resolves This branch had replaced the `~/package` bullet with a claim that relative paths resolve against `--server_root`. Publisher #897 made both halves wrong: a location may be absolute, start with `~/`, or be relative to the directory holding the config it appears in. The connections page said the config lives in your server root, full stop. `--config` can point anywhere, and that is the whole reason a config can sit next to the packages it points at. Signed-off-by: Monty Lennie --- .../user_guides/publishing/connections.malloynb | 2 +- src/documentation/user_guides/publishing/publishing.malloynb | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/documentation/user_guides/publishing/connections.malloynb b/src/documentation/user_guides/publishing/connections.malloynb index 90f823a4..919073bb 100644 --- a/src/documentation/user_guides/publishing/connections.malloynb +++ b/src/documentation/user_guides/publishing/connections.malloynb @@ -7,7 +7,7 @@ Configure database connections for Malloy Publisher deployments. For an overview ## Configuration File -Publisher uses `publisher.config.json` for database connections. This file lives in your server root directory. +Publisher uses `publisher.config.json` for database connections. By default it is read from your server root, and `--config ` points at one anywhere else. Package locations inside it resolve against the directory holding the config, so a config kept next to its packages travels with them. **Keeping credentials out of the file:** any `${VAR}` inside a string value is replaced with that environment variable's value when the config loads, so passwords and keys can stay out of the file and out of version control: diff --git a/src/documentation/user_guides/publishing/publishing.malloynb b/src/documentation/user_guides/publishing/publishing.malloynb index f8c0c7f0..9f9d6772 100644 --- a/src/documentation/user_guides/publishing/publishing.malloynb +++ b/src/documentation/user_guides/publishing/publishing.malloynb @@ -144,9 +144,12 @@ See [Publisher Connections](connections.malloynb) for BigQuery, Snowflake, and o A package's `location` can be any of these: **Local Filesystem** -- Relative paths: `./package`, `../package` (resolved against `--server_root`) +- Relative paths: `./package`, `../package`, resolved against the directory holding the config they appear in +- Home-relative paths: `~/package` - Absolute paths: `/absolute/path/to/package` +Packages do not have to live inside your server root. Keeping a config next to the packages it points at means the two move together. + **GitHub** - `https://github.com/owner/repo/tree/branch/package-path`