Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/create-chkit-example-prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"create-chkit": patch
---

Prompt for the example to scaffold from a bundled manifest instead of silently defaulting to `clickbench`. The list of examples ships with the package and is kept in sync with `examples/` at build time.
8 changes: 8 additions & 0 deletions .changeset/detect-streamed-clickhouse-exceptions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
"@chkit/clickhouse": patch
"chkit": patch
---

Detect ClickHouse exceptions that arrive in the `x-clickhouse-exception-code` response header on an HTTP 200 response. When `send_progress_in_http_headers=1` is set (chkit's default for long-running migrations), ClickHouse commits to a 200 status before the query completes; if the query then errors, the exception is reported via response headers rather than as an HTTP error code. `@clickhouse/client` does not surface this as a thrown error, so previously `chkit migrate` could record a failed INSERT migration as applied while the data never landed.

`@chkit/clickhouse` now inspects `result.response_headers` after every `command`/`query`/`queryJson`/`insert` call and throws a new `ClickHouseStreamedException` (with `code`, `exceptionTag`, and `query_id`) when a non-zero exception code is present. Migrations that fail this way now exit with a non-zero status and remain pending so the operator can fix the underlying issue and re-apply.
26 changes: 26 additions & 0 deletions .changeset/migration-mode-async.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
"chkit": patch
---

Add `mode=async` annotation for long-running migration operations.

Mark an operation as async by adding `mode=async` to its `-- operation:` header line, for example:

```sql
-- operation: load_table_data key=table:default.hits risk=caution mode=async
INSERT INTO default.hits SELECT * FROM s3(...);
```

When `chkit migrate --apply` encounters an async operation it:

1. Computes a deterministic `query_id` from `sha256(migration_filename + ':' + statement_index)`.
2. Checks `system.processes` / `system.query_log` for any prior attempt with that id.
3. Fires the INSERT via the existing `submit()` path without blocking on its HTTP response, and polls `queryStatus(query_id)` every 5 seconds — printing a one-line update (`written=N.NM rows (N.N GiB), elapsed Ns`) so the operator sees the load advance.
4. On `QueryFinish` → records the journal entry and proceeds. On `ExceptionWhileProcessing` → throws with the server's exception. On any prior run's failure → resubmits (retry semantics).

This unblocks two scenarios chkit could not previously handle:

- **Long INSERTs through a proxy/LB with an HTTP request-duration ceiling**: the operator sees progress, and a connection drop mid-poll no longer cancels the work — the deterministic id lets a re-run attach to the in-flight query on the server.
- **Transient client-side errors during a multi-minute load**: re-running chkit picks up where it left off rather than starting over.

Existing migrations without `mode=async` continue to use the synchronous path; the annotation is opt-in and forward-compatible (an unknown mode value falls back to sync).
6 changes: 6 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ This is a monorepo managed with Bun workspaces and Turborepo.
- Internal packages (`@chkit/clickhouse`, `@chkit/codegen`) are not meant to be installed directly by users.
- Plugins extend the CLI and are registered in `clickhouse.config.ts` via the `plugins` array.

## Examples

The `examples/` directory holds curated starter projects scaffolded by `create-chkit` (`bun create chkit@latest`). The list shown in the interactive prompt is sourced from `examples/manifest.json`, which is bundled into the `create-chkit` package at build time.

**When you add a new example under `examples/<name>/`, you MUST also add an entry to `examples/manifest.json`.** The `verify` job runs `scripts/check-examples-manifest.ts` (via `create-chkit`'s build step), which fails CI if any directory under `examples/` is missing from the manifest, any manifest entry has no matching directory, or the `default` field doesn't match a listed entry.

## CLI Commands

`init`, `generate`, `migrate`, `status`, `drift`, `check`, `codegen`, `pull`, `plugin`
Expand Down
13 changes: 13 additions & 0 deletions apps/docs/astro.config.mjs
Original file line number Diff line number Diff line change
@@ -1,15 +1,23 @@
// @ts-check
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
import react from '@astrojs/react';
import rawMarkdown from './src/integrations/raw-markdown';

const isDev = process.argv.includes('dev');

// https://astro.build/config
export default defineConfig({
integrations: [
starlight({
title: 'chkit Docs',
description: 'Public documentation for chkit, the ClickHouse schema and migration CLI.',
customCss: ['./src/styles/custom.css'],
...(isDev && {
components: {
Footer: './src/components/Footer.astro',
},
}),
sidebar: [
{
label: 'Overview',
Expand Down Expand Up @@ -41,12 +49,17 @@ export default defineConfig({
label: 'Schema',
autogenerate: { directory: 'schema' },
},
{
label: 'ObsessionDB',
autogenerate: { directory: 'obsessiondb' },
},
{
label: 'Plugins',
autogenerate: { directory: 'plugins' },
},
],
}),
...(isDev ? [react()] : []),
rawMarkdown(),
],
});
6 changes: 6 additions & 0 deletions apps/docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@
"sharp": "^0.34.2"
},
"devDependencies": {
"@astrojs/react": "^5.0.5",
"@types/react": "^19.2.15",
"@types/react-dom": "^19.2.3",
"agentation": "^3.0.2",
"react": "^19.2.6",
"react-dom": "^19.2.6",
"wrangler": "^4.65.0"
}
}
5 changes: 5 additions & 0 deletions apps/docs/src/components/AgentationDev.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { Agentation } from 'agentation';

export default function AgentationDev() {
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
return <Agentation />;
}
51 changes: 51 additions & 0 deletions apps/docs/src/components/Command.astro
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
interface Props {
command: string;
}

const { command } = Astro.props;
---

<div class="pkg-cmd pkg-cmd--plain" data-plain-cmd>
<div class="pkg-cmd-body">
<pre class="pkg-cmd-pre"><code><span class="pkg-cmd-prompt">$</span> {command}</code></pre>
</div>
<button type="button" class="pkg-cmd-copy pkg-cmd-copy--floating" aria-label="Copy command" data-command={command}>
<svg
width="14"
height="14"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
aria-hidden="true"
>
<rect x="9" y="9" width="13" height="13" rx="2"></rect>
<path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path>
</svg>
</button>
</div>

<script>
function init() {
document.querySelectorAll<HTMLElement>('[data-plain-cmd]').forEach((group) => {
group.querySelector<HTMLButtonElement>('.pkg-cmd-copy')?.addEventListener('click', (event) => {
const button = event.currentTarget as HTMLButtonElement;
const text = button.dataset.command;
if (!text) return;
void navigator.clipboard.writeText(text).then(() => {
button.classList.add('is-copied');
setTimeout(() => button.classList.remove('is-copied'), 1200);
});
});
});
}

if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', init);
} else {
init();
}
</script>
8 changes: 8 additions & 0 deletions apps/docs/src/components/Footer.astro
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
import Default from '@astrojs/starlight/components/Footer.astro';
import AgentationDev from './AgentationDev';
---

<Default><slot /></Default>

{import.meta.env.DEV && <AgentationDev client:only="react" />}
187 changes: 187 additions & 0 deletions apps/docs/src/components/PackagedCommand.astro
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
---
type PackageManager = 'bun' | 'npm' | 'pnpm' | 'yarn';

interface Props {
install?: string;
exec?: string;
create?: string;
args?: string;
dev?: boolean;
}

const { install, exec, create, args, dev } = Astro.props;

const pms: PackageManager[] = ['bun', 'npm', 'pnpm', 'yarn'];

function buildInstall(pm: PackageManager, packages: string, isDev: boolean): string {
const devFlag = isDev ? (pm === 'bun' ? '-d ' : '-D ') : '';
const verb = pm === 'npm' ? 'install' : 'add';
return `${pm} ${verb} ${devFlag}${packages}`;
}

function buildExec(pm: PackageManager, command: string): string {
switch (pm) {
case 'bun':
return `bunx ${command}`;
case 'npm':
return `npx ${command}`;
case 'pnpm':
return `pnpm dlx ${command}`;
case 'yarn':
return `yarn dlx ${command}`;
}
}

function buildCreate(pm: PackageManager, pkg: string, rawArgs?: string): string {
const yarnPkg = pkg.replace(/@latest$/, '');
const target = pm === 'yarn' ? yarnPkg : pkg;

if (!rawArgs) {
return `${pm} create ${target}`;
}

if (pm !== 'npm') {
return `${pm} create ${target} ${rawArgs}`;
}

const parts = rawArgs.split(/\s+/);
const firstFlagIdx = parts.findIndex((p) => p.startsWith('-'));
if (firstFlagIdx === -1) {
return `npm create ${target} ${rawArgs}`;
}
const positional = parts.slice(0, firstFlagIdx).join(' ');
const flags = parts.slice(firstFlagIdx).join(' ');
return positional
? `npm create ${target} ${positional} -- ${flags}`
: `npm create ${target} -- ${flags}`;
}

function commandFor(pm: PackageManager): string {
if (install !== undefined) return buildInstall(pm, install, dev ?? false);
if (exec !== undefined) return buildExec(pm, exec);
if (create !== undefined) return buildCreate(pm, create, args);
throw new Error('PackagedCommand requires one of: install, exec, create');
}

const entries = pms.map((pm) => ({ pm, command: commandFor(pm) }));
---

<div class="pkg-cmd" data-pkg-cmd>
<div class="pkg-cmd-header" role="tablist">
<div class="pkg-cmd-tabs">
{
entries.map(({ pm }) => (
<button
type="button"
role="tab"
class="pkg-cmd-tab"
data-pm={pm}
aria-selected={pm === 'bun' ? 'true' : 'false'}
>
{pm}
</button>
))
}
</div>
<button type="button" class="pkg-cmd-copy" aria-label="Copy command">
<svg
width="14"
height="14"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
aria-hidden="true"
>
<rect x="9" y="9" width="13" height="13" rx="2"></rect>
<path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path>
</svg>
</button>
</div>
<div class="pkg-cmd-body">
{
entries.map(({ pm, command }) => (
<pre class="pkg-cmd-pre" data-pm-pane={pm} hidden={pm !== 'bun'}><code><span class="pkg-cmd-prompt">$</span> {command}</code></pre>
))
}
</div>
</div>

<script>
const STORAGE_KEY = 'chkit:pm';
const DEFAULT_PM = 'bun';

function getActivePm(): string {
try {
return localStorage.getItem(STORAGE_KEY) ?? DEFAULT_PM;
} catch {
return DEFAULT_PM;
}
}

function setActivePm(pm: string) {
try {
localStorage.setItem(STORAGE_KEY, pm);
} catch {
/* ignore */
}
applyPm(pm);
}

function applyPm(pm: string) {
document.querySelectorAll<HTMLElement>('[data-pkg-cmd]').forEach((group) => {
group.querySelectorAll<HTMLButtonElement>('.pkg-cmd-tab').forEach((tab) => {
tab.setAttribute('aria-selected', tab.dataset.pm === pm ? 'true' : 'false');
});
let matched = false;
group.querySelectorAll<HTMLElement>('[data-pm-pane]').forEach((pane) => {
const isMatch = pane.dataset.pmPane === pm;
pane.hidden = !isMatch;
if (isMatch) matched = true;
});
// Fallback: if selected PM doesn't exist on this instance, show the first pane.
if (!matched) {
const first = group.querySelector<HTMLElement>('[data-pm-pane]');
if (first) first.hidden = false;
}
});
}

function init() {
document.querySelectorAll<HTMLElement>('[data-pkg-cmd]').forEach((group) => {
group.querySelectorAll<HTMLButtonElement>('.pkg-cmd-tab').forEach((tab) => {
tab.addEventListener('click', () => {
const pm = tab.dataset.pm;
if (pm) setActivePm(pm);
});
});
group.querySelector<HTMLButtonElement>('.pkg-cmd-copy')?.addEventListener('click', (event) => {
const button = event.currentTarget as HTMLButtonElement;
const activePane = group.querySelector<HTMLElement>('[data-pm-pane]:not([hidden])');
const code = activePane?.querySelector('code')?.textContent?.trim();
if (!code) return;
const text = code.replace(/^\$\s*/, '');
void navigator.clipboard.writeText(text).then(() => {
button.classList.add('is-copied');
setTimeout(() => button.classList.remove('is-copied'), 1200);
});
});
});

window.addEventListener('storage', (event) => {
if (event.key === STORAGE_KEY && event.newValue) {
applyPm(event.newValue);
}
});

applyPm(getActivePm());
}

if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', init);
} else {
init();
}
</script>
4 changes: 2 additions & 2 deletions apps/docs/src/content/docs/cli/query.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ The SQL must be a single positional argument. Wrap it in quotes if it contains s

No command-specific flags. See [global flags](/cli/overview/#global-flags).

When the [ObsessionDB plugin](/plugins/obsessiondb/) is loaded, `chkit query`
When the [ObsessionDB plugin](/obsessiondb/overview/) is loaded, `chkit query`
also accepts `--service <name-or-alias>` to route the query to a specific service for
this invocation (see [Per-command service override](/plugins/obsessiondb/#per-command-service-override)).
this invocation (see [Per-command override](/obsessiondb/services/#per-command-override)).

## Behavior

Expand Down
Loading
Loading