feat(ai): add the ai resource with an OpenAI-compatible gateway#178
feat(ai): add the ai resource with an OpenAI-compatible gateway#178ItamarZand88 wants to merge 13 commits into
Conversation
Introduce built-in AI: an `ai` resource that provisions keyless, in-account model access through an embedded gateway. Adds the Rust gateway engine and napi addon, the TypeScript SDK wrapper, the per-cloud controllers and setup emitters (AWS Bedrock, GCP Vertex, Azure Foundry), the ai/invoke permission sets, and quickstart examples.
- The napi error envelope now carries httpStatusCode and hint, so a gateway startup failure (e.g. BindingConfigInvalid at http_status_code 400) is no longer flattened to a 500 when it crosses into JS. - parseSse preserves the malformed-chunk cause as a structured error source instead of folding it into the message string. - Name the condition the anthropic-beta merge test enforces rather than the old bug it guards against.
The ai() client resolved its napi addon by filesystem lookup, which cannot work inside a `bun build --compile` single-file binary — so a compiled Worker calling ai() failed while kv/storage/queue/vault worked. Mirror the bindings embedded-addon mechanism end to end: - The ai-gateway loader gains an embedded slot (registerEmbeddedAddon); loadAddon prefers it over the filesystem/prebuild resolution that can't run in a binary. - ai-gateway/native installEmbeddedAddon() registers the bun-embedded addon. - @alienplatform/sdk/native installs both the bindings and ai-gateway addons; the SDK keeps both external so the compiled binary shares one module for each. - alien-build stages the ai-gateway addon alongside bindings, best-effort: a Worker that resolves ai-gateway transitively through the SDK but never calls ai() still builds (a missing addon for the target is skipped, not an error). Verified by the package-layout compile-smoke (a compiled binary resolves the SDK's ai + storage after both staged .node files are removed) and a loader unit test that loadAddon prefers the embedded addon.
|
Too many files changed for review. ( Bypass the limit by tagging |
The Postgres runtime binding (packages/bindings/src/postgres.ts, getPostgresConnection, cloud secret-manager deps) and the AI+Postgres chatbot example rode in from the shared source branch. main already ships the Postgres resource and dials the DB directly, and PACKAGE_LAYOUT.md keeps getPostgresConnection off the SDK root, so this copy is superseded. Remove it so this PR stays AI-only.
| @@ -0,0 +1,26 @@ | |||
| import * as alien from "@alienplatform/core" | |||
|
|
|||
| // A model-less AI resource. The customer's cloud serves the inference; the | |||
There was a problem hiding this comment.
"model-less AI resource" - confusing
'embedded gateway workload ambient identity etc -- implementation details remove from example
| const api = new alien.Worker("api") | ||
| .code({ type: "source", src: "./", toolchain: { type: "typescript" } }) | ||
| .publicEndpoint("api") | ||
| // Linking injects ALIEN_LLM_BINDING and exposes the gateway as ALIEN_AI_GATEWAY_URL. |
There was a problem hiding this comment.
no need for this comment to be here
| if (!question) { | ||
| return c.json({ error: "pass a question as ?q=..." }, 400) | ||
| } | ||
| const llm = ai("llm") |
There was a problem hiding this comment.
ai("llm") is very confusing :) lets think of better name than "llm"
| const completion = (await llm.chat.completions.create({ | ||
| model, | ||
| messages: [{ role: "user", content: question }], | ||
| })) as { choices?: Array<{ message?: { content?: string } }> } |
There was a problem hiding this comment.
this code is a bit messy.. why is this necessary? need a much cleaner API
| @@ -0,0 +1,178 @@ | |||
| //! Embedded, protocol-agnostic AI gateway: a loopback HTTP server that injects the | |||
There was a problem hiding this comment.
Let's rename this crate to alien-ai-gateway
| @@ -0,0 +1,148 @@ | |||
| //! Live verification against real AWS Bedrock. Ignored by default — it makes a | |||
| //! real inference call and needs ambient AWS credentials in the environment. | |||
There was a problem hiding this comment.
Let's not ignore these tests by default, they're very important (same for foundry vertex etc). And lets run them and make sure they're all passing.
You can load our alien-test-target account credentials from ../env.test - see this pattern in the alien-bindings tests.
| //! real inference call and needs ambient AWS credentials in the environment. | ||
| //! | ||
| //! Run it with: | ||
| //! eval "$(aws configure export-credentials --profile <p> --format env)" |
There was a problem hiding this comment.
no need for this thing once we'll load ../env.test
| @@ -0,0 +1,52 @@ | |||
| //! Node-API addon for the alien AI gateway. | |||
There was a problem hiding this comment.
Hmmm.. let's think if we really need this crate. Maybe we can just start an ./alien-ai-gateway process from the typescript package?
discuss a little bit with claude what are the advantages / disadvantages of this
| @@ -0,0 +1,77 @@ | |||
| # `@alienplatform/ai-gateway` — package layout contract | |||
| @@ -0,0 +1,57 @@ | |||
| { | |||
There was a problem hiding this comment.
Let's think about it for a second. Do we really need a separate package for this? or should this just be inside @alienplatform/bindings?
There was a problem hiding this comment.
Looked at folding this into @alienplatform/bindings and decided to keep it separate.
Main reason: bindings is in every worker (kv/storage/queue/vault), so if the gateway lives there too, every worker ships the alien-ai-gateway binary even when it never calls ai(). Keeping it on its own means only AI workers carry it.
They're also different beasts now. bindings is a napi addon loaded in-process, the gateway is a standalone binary we spawn. Different build and staging (it's why alien-build has the Napi/Binary split). One package for both felt off.
Re binary size: same conclusion. There's only one napi addon left now (bindings), and the gateway binary only gets embedded into compiled workers that actually use ai(), so a normal worker doesn't grow. Merging is what would bloat everything.
Left it separate for now, lmk if you disagree.
| // module for each. They're real runtime dependencies, so the package stays | ||
| // self-contained. | ||
| external: [ | ||
| "@alienplatform/bindings", |
There was a problem hiding this comment.
do we really need both @alienplatform/bindings and @alienplatform/ai-gateway and not put them on the same thing? is there very large impact on the binary size?
There was a problem hiding this comment.
Kept them separate (more on the package.json thread). The size question actually argues for it: the gateway binary only lands in compiled workers that call ai(), so merging into bindings would put it in every worker instead. No size win from combining.
Apply Alon's #178 review: - Replace the alien-ai-gateway-node napi addon with spawning the standalone alien-ai-gateway binary. The SDK uses ALIEN_AI_GATEWAY_URL when a container launcher set it, else spawns the binary and reads the URL it prints; compiled Workers embed the binary and extract+spawn it at runtime. Deletes the addon crate + npm prebuilds; alien-build stages the binary (AddonKind::Binary). - Type chat.completions/responses.create via OpenAI's result types (types-only devDependency), dropping the caller-side casts. - Rename crate alien-gateway -> alien-ai-gateway. - Trim the ai-quickstart example; rename the binding llm -> assistant. - Live tests load .env.test (Bedrock un-ignored; Foundry/Vertex/Bedrock-Claude gated pending minted tokens + model grants); exclude the live binaries from the fast CI job.
getAvailableModels now returns only the models the deployment's cloud has actually enabled, probed at runtime, instead of the static catalog superset. The gateway sends a tiny max_tokens:1 request per catalog model on the first /v1/models (cached per process), signed with the workload's ambient credential, and keeps a model only if it authenticated (2xx/429/400); 401/403/404 drop it. No permission change: the probe rides the existing ai/invoke inference grant. Enriches AiModel with provider + displayName for a picker, adds a static activation field documenting each model's one-time enablement step, iterates the e2e over every listed model, and updates the example to discover-then-pick.
…uilds The release pipeline built the deleted alien-ai-gateway-node napi crate. It now builds the alien-ai-gateway binary in the addon matrix (darwin legs on the addon's targets; linux legs statically against musl, so one binary per arch serves both glibc and Alpine), generates the six per-triple prebuild manifests at publish time instead of keeping them checked in, and publishes them before the wrapper with exact-version optionalDependencies. The dry-run path covers the whole build and publish flow, matching publish-bindings.
…ration test /v1/models now probes every catalog model against the mock upstream, and an unmatched probe 404s and drops the model, so the Claude catalog assertion needs the InvokeModel path answered.
The deployment reconcile polls a deep async chain on a single stack: the executor drives each resource controller, which drives the cloud SDKs' tower/hyper/rustls futures. In debug builds those frames are un-inlined and the chain needs about 3MB, over tokio's 2MB default thread stack, so the alien-test cloud e2e (push_/pull_) overflows its libtest test thread and aborts during initial setup. Release binaries are unaffected (inlined frames fit the 2MB default, confirmed by running the same deploy in release), so this only sizes dev/test threads via a cargo [env] entry. 8MB matches the headroom alien-worker-runtime already gives its runtime for the same deep-async reason.
The availability probe counted 400 as "enabled", assuming it proved the endpoint authed and only rejected the minimal probe body. The cloud e2e disproved that: Bedrock answers "The provided model identifier is invalid" with 400 for a model the account cannot address, so qwen3-coder-next and claude-mythos-5 were listed by getAvailableModels and then failed on the first real call. The probe body is minimal and well formed, so a 400 is about the model rather than the request. Classifying it as unavailable keeps the contract the e2e asserts: every listed model is invocable. A model that only ever answers 400 now drops out of the list instead of being advertised and breaking at runtime.
lilienblum
left a comment
lilienblum
left a comment
There was a problem hiding this comment.
Did a deeper pass on the non-example parts. Below are the merge-blockers I found plus two cheap fixes; leaving the rest out. The gateway proxy core itself looks solid, the risk is concentrated in release wiring, process lifecycle, and one Frozen-vs-Live gap.
| fi | ||
|
|
||
| publish-ai-gateway: | ||
| needs: [prepare, build-addon, publish-npm] |
There was a problem hiding this comment.
🟠 High (not newly introduced by this PR). Release ordering can leave @alienplatform/sdk uninstallable.
publish-ai-gateway runs after publish-npm (this needs), and the SDK hard-depends on @alienplatform/ai-gateway (packages/sdk/package.json, workspace:^, rewritten to ^<version> by pnpm publish). So the SDK lands on npm before the gateway exists: npm i @alienplatform/sdk fails ETARGET during the build-addon window, and if a build-addon leg fails the gateway never publishes and that SDK version is orphaned (past npm's 72h unpublish limit).
To be clear, this isn't new. @alienplatform/bindings already has the identical shape (publish-bindings also needs: publish-npm) and has shipped that way across many releases, so it's a pre-existing release-infra weakness this PR extends rather than introduces. Still worth fixing at the root while it's fresh, especially since ai-gateway adds four fail-fast: false addon legs, which raises the odds of the permanent-orphan case.
Suggested fix. Publish both bindings and ai-gateway before the SDK (make publish-npm depend on both wrapper jobs), so the SDK is never on npm pointing at a dependency that isn't published yet.
| function providerBaseUrl(provider: string): string { | ||
| const override = process.env.ALIEN_AI_LOCAL_BASE_URL | ||
| if (override) return override.replace(/\/$/, "") | ||
| return provider === "anthropic" ? "https://api.anthropic.com" : "https://api.openai.com" |
There was a problem hiding this comment.
🟠 High. An unvalidated provider can silently send a BYO key to the wrong host.
provider is a free-form string at every layer (ExternalAiBinding.provider: String, binding.ts z.string(), no allowlist), and providerBaseUrl here falls back to https://api.openai.com for anything that isn't exactly "anthropic". _postSurface then attaches the BYO key as Authorization: Bearer to that base. So a binding with provider: "Anthropic" (casing), "google", "groq", or a typo, and no ALIEN_AI_LOCAL_BASE_URL override, ships the customer's key to OpenAI. Only the openai BYO path is tested, so these paths are unexercised.
Suggested fix. Validate provider against a known set at the resolve boundary and fail closed instead of defaulting to OpenAI.
(Correcting my earlier note: the "anthropic" branch itself is fine. https://api.anthropic.com/v1/chat/completions with Bearer is Anthropic's OpenAI-compat endpoint. The one real gap on that path is responses.create, since the compat layer doesn't serve /v1/responses, so a BYO-anthropic Responses call would 404 while chat/completions works.)
| child.on("exit", onGone) | ||
| child.stdout?.resume() | ||
| child.stderr?.resume() | ||
| child.unref() |
There was a problem hiding this comment.
🟠 High (scoped to one-shot processes). child.unref() here doesn't let a short-lived host process exit.
unref() drops only the process handle, but the resume() calls just above put stdout/stderr into flowing mode, which keeps those stream handles referenced, so the event loop stays alive for the child's whole lifetime. Long-running Workers (the shipped example is a Hono server) want to stay up anyway and are reaped on shutdown, so they're unaffected. The bite is on one-shot callers: a CLI/batch script or a vitest suite that calls ai() hangs until the gateway child dies, and since the child runs forever, that means it never returns.
Suggested fix. Unref the streams too: child.stdout?.unref(); child.stderr?.unref() after the resume()s. Confirmable with a one-shot script that calls ai() and checks whether it exits.
| * Read the gateway's `{"aiGatewayUrl":"..."}` line from stdout. Rejects (with the | ||
| * child's stderr as the reason) if the process exits or errors before printing it. | ||
| */ | ||
| function readReadyUrl(child: ChildProcess, binary: string): Promise<string> { |
There was a problem hiding this comment.
🟠 High (needs a silent startup stall to trigger). No startup timeout, so a wedged gateway wedges every ai() caller.
readReadyUrl settles only on the ready line, exit, or error, with no deadline, and createGateway memoizes the pending promise. So a child that starts but never prints its ready line (a stalled IMDS/STS/DNS lookup during credential resolution, which happens before the ready line) leaves this call and every later ai() caller awaiting forever. The exit/error paths are handled; only the silent-hang path isn't. Notably the container launcher already bounds this with wait_until_ready_blocking, but the SDK-spawn path here has no equivalent.
Suggested fix. Add a startup deadline that rejects and clears the memo, matching the container path's bound.
| client: reqwest::Client::new(), | ||
| models_cache, | ||
| }); | ||
| Router::new() |
There was a problem hiding this comment.
🟡 Should-fix (easy). Not merge-blocking, but it silently rejects valid large requests.
The handlers extract body: Bytes and the router never calls DefaultBodyLimit::disable(), so axum 0.8's default 2 MB limit 413s requests before they reach the upstream. That's reachable for vision/base64 images and long tool-heavy conversations.
Suggested fix. Since this is a pure proxy, disable the limit and let the upstream enforce its own: Router::new().layer(DefaultBodyLimit::disable()).
| //! Vertex AI endpoint without a cloud round-trip. | ||
| //! | ||
| //! The `aiplatform.googleapis.com` API enablement is handled by the | ||
| //! `GcpServiceActivationEmitter` when the preflight injects a |
There was a problem hiding this comment.
🟡 Should-fix (easy). Breaks the Frozen/Terraform GCP path only (the native controller is fine).
The doc here says aiplatform enablement is handled by GcpServiceActivationEmitter via an injected ServiceActivation, but get_required_services() in gcp_service_activation.rs has no "ai" arm, so nothing injects it. terraform apply still succeeds (the emitter just produces no google_project_service); the failure surfaces later at gateway runtime as SERVICE_DISABLED when Vertex is first called on a project that didn't already have the API on. The native controller enables it at runtime (ai/gcp.rs), so only Terraform-mode GCP deploys are affected.
Suggested fix. Add an "ai" arm to get_required_services().
3 participants
Summary
Adds an
airesource that gives a workload an OpenAI-compatible endpoint for model inference, in two modes. By default it's keyless: for the host cloud's own models — Bedrock on AWS, Vertex on GCP, Foundry on Azure — a loopback gateway signs each request with the workload's ambient cloud identity, so there are no model API keys to store. A workload can instead bring its own provider key (e.g. OpenAI), in which case the client talks to that provider directly with the vault-resolved key and the gateway isn't involved.What happens on the keyless path, when a workload declares an
aibinding for one of the host cloud's models:InvokeModel, VertexrawPredict, or the Foundry Anthropic endpoint — and signs it with the workload's ambient cloud credentials. ← the heart of the changeFor a cloud's own models, this changes AI access from managing model API keys to using the cloud account the workload already runs in.
What I did
airesource to the SDK, plus the controllers and per-cloud setup that grant a workload access to that cloud's models.aibinding can point at an external provider with the workload's own key (vault-resolved, redacted in logs), which bypasses the gateway and calls the provider directly.getAvailableModelsreport real availability: the gateway probes each catalog model with a one-token request under the workload's own inference credential and lists only what the cloud actually has enabled (a 429 counts as enabled but rate-limited; 401/403/404 drop the model). Each entry carriesprovideranddisplayNameso an app can build a model picker, and the example README documents each model's one-time enablement step per cloud.chat.completions.createandresponses.createreturn OpenAI's own result types (a types-only dev dependency), so callers read.choices[0].message.contentwith no cast.ai-quickstart-tsexample built around "list the available models, then pick one".Files touched
crates/alien-ai-gateway— the gateway engine (router, model catalog, availability probe, per-cloud translation + signing) and thealien-ai-gatewaylauncher binary.packages/ai-gateway— the TypeScript wrapper: resolves and spawns the gateway binary, and the ambient vs BYO-key connection resolution.crates/alien-core,packages/core,packages/sdk— theairesource type, the model catalog with per-model activation metadata, and the SDK surface (plus embedding the gateway binary into compiled Workers viapackages/package-layoutandcrates/alien-build).crates/alien-infra— the AI controllers and runtime wiring.crates/alien-terraform+crates/alien-cloudformation— the per-cloud setup emitters and their snapshots.crates/alien-permissions/permission-sets/ai/*— theai/invokegrant (plus heartbeat/management/provision).examples/ai-quickstart-ts— a runnable example.How I tested
/v1/models: 36 of the 42 AWS catalog models came back listed and enriched, and the 6 dropped were Claude ids that account/region cannot serve.bun build --compileof a Worker that callsai()— the embedded gateway binary is extracted and spawned from inside the single-file binary and serves a loopback URL (the package-layout compile smoke).terraform validateon the AI setup-emitter output and the CloudFormation snapshot suite; generated-schema drift check clean. The live Bedrock test reads the shared test env and runs in the credentialed job.getAvailableModelsreturns and fails if a listed model is not invocable, which is the contract the filter promises. Run it through the e2e-cloud workflow.terraform validate.I also ran a security review on the diff. What it checked:
ai/invokegrants inference only — AWSbedrock:InvokeModel*(+ mantleCreateInferencescoped toproject/default, notproject/*); a GCP custom role of justaiplatform.endpoints.predict/explain(notroles/aiplatform.user, which carries deploy + dataset-export); Azure "Cognitive Services OpenAI User" (data-plane only). No deployment writes or data reads.crates/alien-permissions/permission-sets/ai/invoke.jsonc./v1/modelsprobe sends a one-token inference request signed with that same inference-only grant; it adds no control-plane permission, cannot enable or subscribe anything, and its logs carry only the model id and status, never request or response bodies.crates/alien-ai-gateway/src/availability.rs.Debugimpl redacts it (crates/alien-core/src/bindings/ai.rs) and it's resolved into the worker environment, not logged or held in control-plane state.crates/alien-ai-gateway/src/config.rs.crates/alien-ai-gateway/src/lib.rs.crates/alien-ai-gateway/src/config.rs.Nothing turned up.