-
Notifications
You must be signed in to change notification settings - Fork 0
Prompt for create-chkit example and refactor docs install snippets #130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
ee203d6
Prompt for create-chkit example and replace docs PM tabs with Package…
KeKs0r 6378e5a
docs: promote ObsessionDB to dedicated section
KeKs0r 952256b
examples/clickbench: load dataset from ObsessionDB-hosted mirror
KeKs0r 218dda9
chore: include clickbench example in monorepo workspaces
KeKs0r 0667959
@chkit/clickhouse: detect streamed exceptions on 200 responses
KeKs0r 1a7f542
chkit: add mode=async migration operations with resumable polling
KeKs0r 3d780e1
Fix async migration review issues
KeKs0r 8479ba4
Guard async migration retries by checksum
KeKs0r File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "create-chkit": patch | ||
| --- | ||
|
|
||
| Prompt for the example to scaffold from a bundled manifest instead of silently defaulting to `clickbench`. The list of examples ships with the package and is kept in sync with `examples/` at build time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| "@chkit/clickhouse": patch | ||
| "chkit": patch | ||
| --- | ||
|
|
||
| Detect ClickHouse exceptions that arrive in the `x-clickhouse-exception-code` response header on an HTTP 200 response. When `send_progress_in_http_headers=1` is set (chkit's default for long-running migrations), ClickHouse commits to a 200 status before the query completes; if the query then errors, the exception is reported via response headers rather than as an HTTP error code. `@clickhouse/client` does not surface this as a thrown error, so previously `chkit migrate` could record a failed INSERT migration as applied while the data never landed. | ||
|
|
||
| `@chkit/clickhouse` now inspects `result.response_headers` after every `command`/`query`/`queryJson`/`insert` call and throws a new `ClickHouseStreamedException` (with `code`, `exceptionTag`, and `query_id`) when a non-zero exception code is present. Migrations that fail this way now exit with a non-zero status and remain pending so the operator can fix the underlying issue and re-apply. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| --- | ||
| "chkit": patch | ||
| --- | ||
|
|
||
| Add `mode=async` annotation for long-running migration operations. | ||
|
|
||
| Mark an operation as async by adding `mode=async` to its `-- operation:` header line, for example: | ||
|
|
||
| ```sql | ||
| -- operation: load_table_data key=table:default.hits risk=caution mode=async | ||
| INSERT INTO default.hits SELECT * FROM s3(...); | ||
| ``` | ||
|
|
||
| When `chkit migrate --apply` encounters an async operation it: | ||
|
|
||
| 1. Computes a deterministic `query_id` from `sha256(migration_filename + ':' + statement_index)`. | ||
| 2. Checks `system.processes` / `system.query_log` for any prior attempt with that id. | ||
| 3. Fires the INSERT via the existing `submit()` path without blocking on its HTTP response, and polls `queryStatus(query_id)` every 5 seconds — printing a one-line update (`written=N.NM rows (N.N GiB), elapsed Ns`) so the operator sees the load advance. | ||
| 4. On `QueryFinish` → records the journal entry and proceeds. On `ExceptionWhileProcessing` → throws with the server's exception. On any prior run's failure → resubmits (retry semantics). | ||
|
|
||
| This unblocks two scenarios chkit could not previously handle: | ||
|
|
||
| - **Long INSERTs through a proxy/LB with an HTTP request-duration ceiling**: the operator sees progress, and a connection drop mid-poll no longer cancels the work — the deterministic id lets a re-run attach to the in-flight query on the server. | ||
| - **Transient client-side errors during a multi-minute load**: re-running chkit picks up where it left off rather than starting over. | ||
|
|
||
| Existing migrations without `mode=async` continue to use the synchronous path; the annotation is opt-in and forward-compatible (an unknown mode value falls back to sync). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| import { Agentation } from 'agentation'; | ||
|
|
||
| export default function AgentationDev() { | ||
| return <Agentation />; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| --- | ||
| interface Props { | ||
| command: string; | ||
| } | ||
|
|
||
| const { command } = Astro.props; | ||
| --- | ||
|
|
||
| <div class="pkg-cmd pkg-cmd--plain" data-plain-cmd> | ||
| <div class="pkg-cmd-body"> | ||
| <pre class="pkg-cmd-pre"><code><span class="pkg-cmd-prompt">$</span> {command}</code></pre> | ||
| </div> | ||
| <button type="button" class="pkg-cmd-copy pkg-cmd-copy--floating" aria-label="Copy command" data-command={command}> | ||
| <svg | ||
| width="14" | ||
| height="14" | ||
| viewBox="0 0 24 24" | ||
| fill="none" | ||
| stroke="currentColor" | ||
| stroke-width="2" | ||
| stroke-linecap="round" | ||
| stroke-linejoin="round" | ||
| aria-hidden="true" | ||
| > | ||
| <rect x="9" y="9" width="13" height="13" rx="2"></rect> | ||
| <path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path> | ||
| </svg> | ||
| </button> | ||
| </div> | ||
|
|
||
| <script> | ||
| function init() { | ||
| document.querySelectorAll<HTMLElement>('[data-plain-cmd]').forEach((group) => { | ||
| group.querySelector<HTMLButtonElement>('.pkg-cmd-copy')?.addEventListener('click', (event) => { | ||
| const button = event.currentTarget as HTMLButtonElement; | ||
| const text = button.dataset.command; | ||
| if (!text) return; | ||
| void navigator.clipboard.writeText(text).then(() => { | ||
| button.classList.add('is-copied'); | ||
| setTimeout(() => button.classList.remove('is-copied'), 1200); | ||
| }); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| if (document.readyState === 'loading') { | ||
| document.addEventListener('DOMContentLoaded', init); | ||
| } else { | ||
| init(); | ||
| } | ||
| </script> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| import Default from '@astrojs/starlight/components/Footer.astro'; | ||
| import AgentationDev from './AgentationDev'; | ||
| --- | ||
|
|
||
| <Default><slot /></Default> | ||
|
|
||
| {import.meta.env.DEV && <AgentationDev client:only="react" />} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,187 @@ | ||
| --- | ||
| type PackageManager = 'bun' | 'npm' | 'pnpm' | 'yarn'; | ||
|
|
||
| interface Props { | ||
| install?: string; | ||
| exec?: string; | ||
| create?: string; | ||
| args?: string; | ||
| dev?: boolean; | ||
| } | ||
|
|
||
| const { install, exec, create, args, dev } = Astro.props; | ||
|
|
||
| const pms: PackageManager[] = ['bun', 'npm', 'pnpm', 'yarn']; | ||
|
|
||
| function buildInstall(pm: PackageManager, packages: string, isDev: boolean): string { | ||
| const devFlag = isDev ? (pm === 'bun' ? '-d ' : '-D ') : ''; | ||
| const verb = pm === 'npm' ? 'install' : 'add'; | ||
| return `${pm} ${verb} ${devFlag}${packages}`; | ||
| } | ||
|
|
||
| function buildExec(pm: PackageManager, command: string): string { | ||
| switch (pm) { | ||
| case 'bun': | ||
| return `bunx ${command}`; | ||
| case 'npm': | ||
| return `npx ${command}`; | ||
| case 'pnpm': | ||
| return `pnpm dlx ${command}`; | ||
| case 'yarn': | ||
| return `yarn dlx ${command}`; | ||
| } | ||
| } | ||
|
|
||
| function buildCreate(pm: PackageManager, pkg: string, rawArgs?: string): string { | ||
| const yarnPkg = pkg.replace(/@latest$/, ''); | ||
| const target = pm === 'yarn' ? yarnPkg : pkg; | ||
|
|
||
| if (!rawArgs) { | ||
| return `${pm} create ${target}`; | ||
| } | ||
|
|
||
| if (pm !== 'npm') { | ||
| return `${pm} create ${target} ${rawArgs}`; | ||
| } | ||
|
|
||
| const parts = rawArgs.split(/\s+/); | ||
| const firstFlagIdx = parts.findIndex((p) => p.startsWith('-')); | ||
| if (firstFlagIdx === -1) { | ||
| return `npm create ${target} ${rawArgs}`; | ||
| } | ||
| const positional = parts.slice(0, firstFlagIdx).join(' '); | ||
| const flags = parts.slice(firstFlagIdx).join(' '); | ||
| return positional | ||
| ? `npm create ${target} ${positional} -- ${flags}` | ||
| : `npm create ${target} -- ${flags}`; | ||
| } | ||
|
|
||
| function commandFor(pm: PackageManager): string { | ||
| if (install !== undefined) return buildInstall(pm, install, dev ?? false); | ||
| if (exec !== undefined) return buildExec(pm, exec); | ||
| if (create !== undefined) return buildCreate(pm, create, args); | ||
| throw new Error('PackagedCommand requires one of: install, exec, create'); | ||
| } | ||
|
|
||
| const entries = pms.map((pm) => ({ pm, command: commandFor(pm) })); | ||
| --- | ||
|
|
||
| <div class="pkg-cmd" data-pkg-cmd> | ||
| <div class="pkg-cmd-header" role="tablist"> | ||
| <div class="pkg-cmd-tabs"> | ||
| { | ||
| entries.map(({ pm }) => ( | ||
| <button | ||
| type="button" | ||
| role="tab" | ||
| class="pkg-cmd-tab" | ||
| data-pm={pm} | ||
| aria-selected={pm === 'bun' ? 'true' : 'false'} | ||
| > | ||
| {pm} | ||
| </button> | ||
| )) | ||
| } | ||
| </div> | ||
| <button type="button" class="pkg-cmd-copy" aria-label="Copy command"> | ||
| <svg | ||
| width="14" | ||
| height="14" | ||
| viewBox="0 0 24 24" | ||
| fill="none" | ||
| stroke="currentColor" | ||
| stroke-width="2" | ||
| stroke-linecap="round" | ||
| stroke-linejoin="round" | ||
| aria-hidden="true" | ||
| > | ||
| <rect x="9" y="9" width="13" height="13" rx="2"></rect> | ||
| <path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path> | ||
| </svg> | ||
| </button> | ||
| </div> | ||
| <div class="pkg-cmd-body"> | ||
| { | ||
| entries.map(({ pm, command }) => ( | ||
| <pre class="pkg-cmd-pre" data-pm-pane={pm} hidden={pm !== 'bun'}><code><span class="pkg-cmd-prompt">$</span> {command}</code></pre> | ||
| )) | ||
| } | ||
| </div> | ||
| </div> | ||
|
|
||
| <script> | ||
| const STORAGE_KEY = 'chkit:pm'; | ||
| const DEFAULT_PM = 'bun'; | ||
|
|
||
| function getActivePm(): string { | ||
| try { | ||
| return localStorage.getItem(STORAGE_KEY) ?? DEFAULT_PM; | ||
| } catch { | ||
| return DEFAULT_PM; | ||
| } | ||
| } | ||
|
|
||
| function setActivePm(pm: string) { | ||
| try { | ||
| localStorage.setItem(STORAGE_KEY, pm); | ||
| } catch { | ||
| /* ignore */ | ||
| } | ||
| applyPm(pm); | ||
| } | ||
|
|
||
| function applyPm(pm: string) { | ||
| document.querySelectorAll<HTMLElement>('[data-pkg-cmd]').forEach((group) => { | ||
| group.querySelectorAll<HTMLButtonElement>('.pkg-cmd-tab').forEach((tab) => { | ||
| tab.setAttribute('aria-selected', tab.dataset.pm === pm ? 'true' : 'false'); | ||
| }); | ||
| let matched = false; | ||
| group.querySelectorAll<HTMLElement>('[data-pm-pane]').forEach((pane) => { | ||
| const isMatch = pane.dataset.pmPane === pm; | ||
| pane.hidden = !isMatch; | ||
| if (isMatch) matched = true; | ||
| }); | ||
| // Fallback: if selected PM doesn't exist on this instance, show the first pane. | ||
| if (!matched) { | ||
| const first = group.querySelector<HTMLElement>('[data-pm-pane]'); | ||
| if (first) first.hidden = false; | ||
| } | ||
| }); | ||
| } | ||
|
|
||
| function init() { | ||
| document.querySelectorAll<HTMLElement>('[data-pkg-cmd]').forEach((group) => { | ||
| group.querySelectorAll<HTMLButtonElement>('.pkg-cmd-tab').forEach((tab) => { | ||
| tab.addEventListener('click', () => { | ||
| const pm = tab.dataset.pm; | ||
| if (pm) setActivePm(pm); | ||
| }); | ||
| }); | ||
| group.querySelector<HTMLButtonElement>('.pkg-cmd-copy')?.addEventListener('click', (event) => { | ||
| const button = event.currentTarget as HTMLButtonElement; | ||
| const activePane = group.querySelector<HTMLElement>('[data-pm-pane]:not([hidden])'); | ||
| const code = activePane?.querySelector('code')?.textContent?.trim(); | ||
| if (!code) return; | ||
| const text = code.replace(/^\$\s*/, ''); | ||
| void navigator.clipboard.writeText(text).then(() => { | ||
| button.classList.add('is-copied'); | ||
| setTimeout(() => button.classList.remove('is-copied'), 1200); | ||
| }); | ||
| }); | ||
| }); | ||
|
|
||
| window.addEventListener('storage', (event) => { | ||
| if (event.key === STORAGE_KEY && event.newValue) { | ||
| applyPm(event.newValue); | ||
| } | ||
| }); | ||
|
|
||
| applyPm(getActivePm()); | ||
| } | ||
|
|
||
| if (document.readyState === 'loading') { | ||
| document.addEventListener('DOMContentLoaded', init); | ||
| } else { | ||
| init(); | ||
| } | ||
| </script> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.