Support for external credential stores

[mish-elle]

As enterprise self-hosters, we have a security requirement to manage secrets through a centralized credential store such as AWS Secrets Manager. This provides centralized access control, audit logging, and rotation capabilities.

Currently, all sensitive configuration is passed via environment variables. We'd like to have support added for optionally fetching secrets from an external credential store.

This would apply to secrets such as:

• ENCRYPTION_SECRET
• SUPERTOKENS_REFRESH_TOKEN_KEY
• SUPERTOKENS_ACCESS_TOKEN_KEY
• etc

Precedence order

1. Explicit environment variables (highest priority — allows per-service overrides)
2. Values fetched from credential store
3. Defaults defined in zod schemas (lowest priority)

[n1ru4l]

That makes sense!

In terms of the configuration, the user should explicitly opt-into retrieving the credentials from the store.

1. If nothing is specified, the environment variables are used (if they are missing the service fails to start)
2. If the provider is specified, that provider is used instead of environment variables

What should happen to the service and healthcheck and liveness probe in case the retrieval of the secret(s) fails at

• initial startup (Should it fail to start if it cannot retrieve the credentials initially?)
• refresh/rotation (Do we keep the latest valid value and "notify" that new values cannot be retrieved?)

As for the configuration for loading the credentials from an external store, it should be possible to add other providers later on (e.g. Azure Key Vault). Could you come up with a proposal on how the configuration API for this could look like?

[mish-elle]

@n1ru4l Thanks for the feedback! Here's a proposal we were thinking of

Approach: URI-Based Credential References

Rather than introducing a separate configuration file or a global provider toggle, we propose using
URI-prefixed environment variable values to declare that a secret should be fetched from an
external credential store.

How it works

Any environment variable whose value starts with a recognized credential store URI scheme is
automatically resolved at startup before the application reads its configuration. All other
environment variables are left untouched.

environment:
  ENCRYPTION_SECRET: 'asm://my/secret/path/hive/encryption-secret'
  SUPERTOKENS_REFRESH_TOKEN_KEY: 'asm://my/secret/path/hive/supertokens-refresh-key'
  SUPERTOKENS_ACCESS_TOKEN_KEY: 'asm://my/secret/path/hive/supertokens-access-key'
  # Still supports in place passwords
  CLICKHOUSE_PASSWORD: `changeme`
  # Non-secret env vars are unchanged
  POSTGRES_HOST: 'postgres'
  LOG_LEVEL: 'debug'

At startup, each service resolves credential references before validating configuration.

// environment.ts
import { resolveCredentials } from '@hive/credential-store';

await resolveCredentials(); // resolves all credential URIs in process.env

const configs = {
  base: EnvironmentModel.safeParse(process.env), // sees real values
  // ...
};

The resolveCredentials() call:

1. Scans process.env for values matching <scheme>://<path>[?params]
2. Resolves all secrets in parallel from the external store
3. Replaces process.env values with the resolved plaintext
4. Returns — Zod validation, connection pools, etc. see real values

URI Format

<scheme>://<secret-path>[?key=value&...]

Examples:

URI	Provider	Secret Path
asm://my/secret/path/hive/supertokens-refresh-key	AWS Secrets Manager	my/secret/path/hive/supertokens-refresh-key
azurekv://myvault/supertokens-refresh-key	Azure Key Vault (future)	vault: myvault, secret: supertokens-refresh-key
vault://secret/data/hive/supertokens-refresh-key	HashiCorp Vault (future)	secret/data/hive/supertokens-refresh-key

The scheme prefix determines which provider implementation is used. Adding a new provider means
registering a new scheme and implementing a simple getSecret(path): Promise<string> interface.

Secret organization: one per entry vs. bundled JSON

Both patterns are common in practice, and the URI approach can support either:

One secret per entry (each env var points to its own secret):

ENCRYPTION_SECRET: 'asm://my/secret/path/hive/encryption-secret'
POSTGRES_PASSWORD: 'asm://my/secret/path/hive/postgres-password'
REDIS_PASSWORD: 'asm://my/secret/path/hive/redis-password'

Each secret contains a plain string value. Simple, granular access policies, independent rotation.

Bundled JSON secret (one secret holds multiple key-value pairs, extracted via ?key=):

# All three env vars read from the same secret, extracting different JSON keys
ENCRYPTION_SECRET: 'asm://my/secret/path/hive/all-secrets?key=encryption_secret'
POSTGRES_PASSWORD: 'asm://my/secret/path/hive/all-secrets?key=postgres_password'
REDIS_PASSWORD: 'asm://my/secret/path/hive/all-secrets?key=redis_password'

The secret value in ASM would be a JSON object:

{
  "encryption_secret": "actual-encryption-key",
  "postgres_password": "actual-db-password",
  "redis_password": "actual-redis-password"
}

When ?key= is present, the provider parses the secret as JSON and returns the value for that key.
When absent, the raw secret string is returned as-is.

Alternative: Global provider + empty env var fallback

Configure a global provider, then if an env var is empty, try to fetch it from the store. Secrets
could be organized either as individual entries (using a naming convention for paths) or as a single
JSON secret containing all key-value pairs.

• Ambiguous: can't distinguish "intentionally empty" from "should fetch from store"
• Requires either a rigid naming convention for secret paths, a single bundled JSON secret, or a
  separate allowlist of which vars are secrets
• Implicit behavior is harder to debug

Why we thought URI Approach may be better

• Zero extra files: everything is self-contained in the env var value
• Explicit opt-in: setting ENCRYPTION_SECRET=asm://my/secret/path is an intentional declaration. No
  URI prefix = plain value, no behavior change
• No-op by default: if no env var contains a credential URI, resolveCredentials() scans and returns immediately.
• Extensible: new providers are just new URI schemes (azurekv://, vault://, gcpsm://)
• Self-documenting: looking at the env vars, you can immediately see which values come from a
  credential store and where they're stored

Execution model: application-level resolution

Credential resolution uses an await call in each service's environment/config module. Since
every service already has an environment.ts (or equivalent) that validates process.env with Zod
at module load time, adding await resolveCredentials() before the Zod safeParse() calls is a
minimal change — one import and one line per service.

This keeps credential resolution visible in the application code (not hidden in a preload) while
still running before any configuration validation or connection setup.

Addressing Questions

1. Should the service fail to start if credentials can't be retrieved initially?

Yes, if a user has explicitly declared POSTGRES_PASSWORD=asm://my/secret/path/hive/db-password, they are
stating that this value MUST come from the credential store. If it can't be resolved, starting the
service with a missing password would just lead to a connection failure anyway. Failing fast with a
clear error message is better:

[credential-store] Failed to resolve 1 secret(s):
  POSTGRES_PASSWORD (/my/secret/path/hive/db-password): AccessDeniedException: ...

2. Refresh/rotation — what happens if new values can't be retrieved?

For the initial implementation, credential resolution happens once at startup (before
the application initializes). So, rotation requires a pod restart (rolling restart via deployment update)

Dynamic refresh could be added for application-level secrets (signing
keys, encryption keys) that are accessed per-request rather than baked into connection pools. Connection secrets (database
passwords) are harder to rotate dynamically because they're captured by connection pools at creation
time. I'll have to do some more investigation on this.

3. Can other providers be added later?

Yes, the URI scheme is the extension point. The provider interface is minimal:

interface CredentialStoreProvider {
  getSecret(secretPath: string): Promise<string>
}

Adding Azure Key Vault support, for example, would mean:

1. Register the azurekv scheme
2. Implement AzureKeyVaultProvider with the above interface
3. Users can immediately use azurekv://myvault/my-secret

Implementation Details

• Package location: packages/internal/credential-store
• Resolution mechanism: top-level await resolveCredentials() in each service's
  environment/config module, before Zod validation
• Error handling: Collects all resolution errors and reports them together before failing
• No runtime cost for non-users: Provider code is only import()-ed if a matching URI is found. If no credential URIs exist in env vars, the call scans and returns with zero external API calls

---

Happy to discuss further or adjust the approach!

[mish-elle]

Hey @n1ru4l, after taking a look into the refresh/rotation, I think that dynamic rotation shouldn't be supported at this time - as the underlying code doesn't support credential rotations. For example, ENCRYPTION_SECRET uses a single key with no versioning in the ciphertext, so rotating it mid-flight would break decryption of existing data like OIDC client secrets.

Until the consuming code is rotation-aware, startup-time resolution + rolling restart is the best approach. Please let me know any feedback!

[n1ru4l]

@mish-elle That sounds reasonable. 👍 I agree with your proposal, it is well thought-through and future-proof!