Learn from Abstract Wikipedia: implement Q/P → text generation service

[konard]

Summary

Issue #18 asked us to learn from Abstract Wikipedia. This PR does both halves:

1. Research — fresh 2025–2026 analysis of Abstract Wikipedia / Wikifunctions and a prioritised feature-gap map (research/).
2. Implementation — it acts on that research by building the single highest-leverage missing capability: reverse generation (Q/P → text), the mirror image of our existing Text → Q/P transformer. This closes the text → Q/P → text round-trip the README vision depends on.

Everything follows the project's established style (pure data layer, injectable renderer, offline-first tests, package exports, hand-written types, a unified-SPA mode, a legacy redirect shell, and e2e coverage).

What shipped

New: Generation service (Q/P → text)

• js/src/generation/constructors.js — typed constructors with named roles (instance_of, located_in, relation) and one templatic renderer per constructor-per-language across the six official UN languages (en, ar, es, fr, ru, zh). Includes English a/an phonotactics, negation patterns, and validateConstructor/buildConstructor/fillTemplate. Pure and dependency-free.
• js/src/generation/qp-to-text.js — QPRenderer with renderWithLabels (pure), render (resolves id-shaped roles via labels), and renderAll (UN 6). Label resolution is pluggable (labelProvider for offline use, or a lazily-constructed Wikidata client — browser vs Node, like the transformer).

Service-quality improvements (per the research)

• getLabels(ids, lang) — batch label resolution, and searchLexemes(term, lang) — Wikidata Lexeme lookup, added to both the Node and browser API clients.
• transformToConstructor(text) on the transformer — folds the flat Q/P sequence into a typed, role-labelled constructor and extracts negation and tense (present/past/future, incl. irregular pasts), addressing two documented limitations.

UI + packaging

• New Generation SPA mode (app.html#mode=generation) wired into routing, the shell tab bar, and app.html; plus a generation/index.html redirect shell and a landing-page card.
• Package surface: ./generate and ./generate/constructors subpath exports, index.js re-exports, index.d.ts declarations. Version bumped 0.1.0 → 0.2.0.

How to reproduce / verify

import { QPRenderer, buildConstructor } from 'human-language/generate';

const r = new QPRenderer();                 // offline: plain-text roles render verbatim
const c = buildConstructor('instance_of', { subject: 'Berlin', object: 'city' });
await r.renderAll(c);
// { en: 'Berlin is a city', es: 'Berlin es un city', fr: 'Berlin est un city',
//   ru: 'Berlin — city', zh: 'Berlin是city', ar: 'Berlin city' }

Round-trip via the transformer: transformToConstructor('Berlin is a city') → { type:'instance_of', subject, object } → renderAll → the table above.

Tests

• js/tests/unit/qp-to-text.test.mjs — 19 offline unit tests (phonotactics, validation, fillTemplate, multi-language render incl. negation/located_in/relation, async render/renderAll, transformer extractModifiers/toConstructor, and a text → Q/P → text round-trip). Added to the gating FAST_SUITES.
• js/tests/e2e/app.spec.mjs — Generation mode boots, the Generate button enables, and the default constructor renders across the UN 6 offline. Mode-count assertions updated (six → seven).
• All gating unit suites + the 45-file syntax check pass locally.

Docs

• docs/case-studies/issue-18/ — README, requirements, and architecture.
• research/missing-features-and-improvements.md — gap table updated with the now-closed items.
• README.md + index.html — Generation feature, demo, and architecture entries.

Files Changed

• js/src/generation/{constructors,qp-to-text}.js — new generation service.
• js/src/wikidata-api.js, js/src/wikidata-api-browser.js — getLabels + searchLexemes.
• js/src/transformation/text-to-qp-transformer.js — typed-constructor output.
• js/src/app/modes/generation.jsx, js/src/app/routing.js, js/src/app/shell.jsx, app.html, generation/index.html, index.html — SPA mode + redirect + landing.
• js/src/index.js, js/src/index.d.ts, package.json — exports, types, version.
• js/tests/unit/qp-to-text.test.mjs, js/tests/e2e/app.spec.mjs, js/tests/unit/routing.test.mjs, js/scripts/run-unit-tests.mjs — tests.
• docs/case-studies/issue-18/*, research/* — research + case study.

🤖 Generated with Claude Code

---

Resolves #18

---

Follow-up round: support all Abstract-Wikipedia test cases, expand features, fix wrong tests

Acting on the latest binding PR comment ("Double check again, and implement unimplemented or delayed, make sure we support all test cases … expand on our use cases and features … If you see clearly wrong tests … do it."):

New grammatical feature — Romance gender agreement

• romanceIndefiniteArticle(lang, gender) agrees the Spanish/French indefinite article (un/una, un/une) with the object noun's grammatical gender (the constructor's gender, ultimately a Wikidata Lexeme P5185 statement), defaulting to masculine. The instance_of es/fr templates now use the agreeing {article} token.
• Fixed clearly-wrong tests: the previous es/fr assertions were ungrammatical (una/une were missing). Corrected to Berlín es una ciudad, Berlin est une ville, etc.

New constructor — quantity (measurement)

• A quantity constructor (subject · value · unit) renders a measurement across all six UN languages (en is, es mide, fr mesure, ru/zh/ar copula) — the direct analogue of a Wikidata quantity claim (e.g. P2048 height). toConstructor now emits it whenever a number-with-unit is present, so Mount Everest is 8848 meters tall round-trips instead of losing its value.

Closed the documented analysis gaps

transformToConstructor now also:

• detectQuestion(text) — classifies wh-questions (who→entity, what→thing, when→time, where→place, why→reason, how→manner, how many/much→quantity) and polar questions.
• extractQuantities(text) — pulls out { value, unit } pairs.
• dedupeSequence(sequence) — collapses adjacent duplicate ids (never merging ambiguous), on by default in transform.

Limitation harness reflects reality

js/scripts/limitation-test.mjs now drives the structured transformToConstructor pipeline and scores against the fields that actually carry the signal. Documented problems dropped 12 → 4; the remaining four are genuine knowledge-base/data limits (Turkey disambiguation, ChatGPT/TikTok coverage, non-Latin script), not transformer logic. limitations-found.json regenerated.

UI

The Generation SPA mode is now role-aware: it reads each constructor's roles from CONSTRUCTORS, so the quantity constructor surfaces Value + Unit inputs and an Object gender selector appears for templates that use the Romance article. Added gender + quantity examples, demo runners, and an offline e2e test for the measurement path.

Tests

js/tests/unit/qp-to-text.test.mjs is now 36 offline tests (gender agreement, romanceIndefiniteArticle, detectQuestion, extractQuantities, dedupeSequence, quantity render across UN 6 + negation, quantity validation/toConstructor, and a measurement round-trip). All gating suites pass.

[konard]

Get latest fresh data about Abstract Wikipedia research, and improve quality of all our services, and also point to missing features we can also support.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-09T22:54:47.259Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

Addressed the feedback ("get latest fresh data… improve quality of all our services… point to missing features we can also support"):

Fresh research (June 2026) added to research/abstract-wikipedia-analysis.md with a cited Sources section — current Wikifunctions metrics, the 2025 NLG semantic fragments work (Z26039 across the UN 6 languages), the NLG SIG, the published roadmap (late-2025 sentence functions → 2026 soft launch), the 6-stage NLG pipeline (constructors → templatic renderers → UD trees → lexeme inflection → phonotactics → assembly), Ninai/Udiron, and the Jan-2023 Google.org risk-of-failure evaluation.

Missing features + service quality in new research/missing-features-and-improvements.md:

• Top gap: reverse generation Q/P → text — the missing counterpart to our transformer that the README vision depends on. Plus Wikidata Lexeme integration, typed/role-labelled constructors, UD parsing, and multi-language rendering.
• Per-service improvements mapped to real files: transformer (negation/tense/lexemes — directly targets the 12/31 failing cases in limitations-found.json), search (searchLexemes), entity/property viewers (auto example sentences), caching (ETag/batch), and a proposed generation/ service.

Commit: e13c141

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $1.290519

📊 Context and tokens usage:

Claude Opus 4.8:

• 61.5K / 1M (6%) input tokens, 14.3K / 128K (11%) output tokens

Total: (2.6K new + 71.1K cache writes + 678.2K cache reads) input tokens, 14.3K output tokens, $1.154360 cost

Claude Haiku 4.5:

• 91.8K / 200K (46%) input tokens, 2.9K / 64K (4%) output tokens

Total: 91.8K input tokens, 2.9K output tokens, $0.136159 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus
• Main model: Claude Opus 4.8 (claude-opus-4-8)
• Additional models:
  • Claude Haiku 4.5 (claude-haiku-4-5-20251001)

📎 Log file uploaded as Gist (770KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

Get latest changes from default branch, and implement everything in the established style our other features have.

Please plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-10T00:01:53.088Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

Working session summary

All work is complete. Here's a summary of what was accomplished on PR #19 (branch issue-18-ef8b353c):

What shipped

Issue #18 ("Learn from Abstract Wikipedia development") is now addressed in both halves — the research and a working implementation of the highest-leverage missing capability it identified: reverse generation (Q/P → text).

New: Generation service (Q/P → text)

• js/src/generation/constructors.js — pure, dependency-free data layer: typed constructors (instance_of, located_in, relation) with named roles and one templatic renderer per constructor-per-language across the six official UN languages (en, ar, es, fr, ru, zh), plus English a/an phonotactics and negation.
• js/src/generation/qp-to-text.js — QPRenderer with renderWithLabels (pure), render, and renderAll; label resolution is pluggable (offline labelProvider or a lazily-built Wikidata client, browser vs Node).

Service-quality improvements

• getLabels (batch) and searchLexemes (Lexeme lookup) added to both API clients.
• transformToConstructor on the transformer — typed, role-labelled output that detects negation and tense, enabling the text → Q/P → text round-trip.

Established-style wiring

• New Generation SPA mode (app.html#mode=generation) into routing, the shell tab bar, and app.html; generation/index.html redirect shell; landing-page card.
• Package surface: ./generate + ./generate/constructors exports, index.js re-exports, index.d.ts declarations; version bumped 0.1.0 → 0.2.0.
• Tests: 19 offline unit tests (gating) + Playwright e2e for the new mode; mode-count assertions updated.
• Docs: docs/case-studies/issue-18/ (README, requirements, architecture), research gap-table marked with closed items, README/index updates.

Verification

• All gating CI jobs green: syntax check, unit tests, E2E (Playwright), and the broken-link check (run 27244534766).
• Branch is up to date with main; working tree clean.
• PR Learn from Abstract Wikipedia: implement Q/P → text generation service #19 retitled, description rewritten, and marked ready for review: Learn from Abstract Wikipedia: implement Q/P → text generation service #19

One CI iteration was needed — the first run's only failure was the not-yet-deployed generation/index.html Pages URL (404), fixed by pointing the demo link at the already-deployed app.html#mode=generation (the page still ships and works after merge/deploy).

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $8.843501

📊 Context and tokens usage:

Claude Opus 4.8: (2 sub-sessions)

1. 115.9K / 1M (12%) input tokens, 38.7K / 128K (30%) output tokens
2. 114.1K / 1M (11%) input tokens, 37.6K / 128K (29%) output tokens

Total: (14.4K new + 265.6K cache writes + 10.0M cache reads) input tokens, 83.5K output tokens, $8.843502 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus
• Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (4006KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-10T00:28:34.481Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

🔎 Double-check pass — closed the one half-implemented gap

Reviewed every claimed capability against its implementation and test coverage. Everything held up except one real gap, now fixed:

Tense was extracted + exposed but silently dropped in rendering

Constructor.tense is a declared type, the transformer extracts it (extractModifiers), and the Generation UI has a tense selector — but the renderer ignored it. Caesar was a general round-tripped to the present Caesar is a general, and the UI's tense dropdown changed nothing.

Fix (js/src/generation/constructors.js): added past/future template variants that inflect the verb only, exactly where that stays grammatical without the noun-case morphology this renderer doesn't yet have (that's the Wikidata Lexeme work tracked in research/):

• instance_of copula tense — en/es/fr (was/era/était, will be/será/sera)
• located_in locative verb — en/es/fr/ru/ar (object case is stable across tenses)
• ru/ar instance_of past would need instrumental/accusative case → graceful fallback to present (no ungrammatical output)
• Chinese 是/在 don't inflect → present form is already correct for every tense

Present-tense output is byte-for-byte unchanged, so nothing regressed.

Tests

• js/tests/unit/qp-to-text.test.mjs: 24 → 28 tests — tense across en/es/fr/ru, tense+negation, explicit-present, a text → Q/P → text past-tense round-trip (Berlin was a city), plus two tests covering the previously-untested render() path through an injected apiClient.getLabels (id-shaped roles looked up; plain-text roles skip the network).

Other findings

• package-lock.json was left at 0.1.0 while package.json bumped to 0.2.0 (there's a check-package-version script) — synced to 0.2.0.
• Verified getLabels/searchLexemes exist and are consistent in both API clients with their helpers (fetchLabels/buildApiUrl).
• Docs updated (README known-limitations + architecture) to describe the now-real tense behaviour and its documented fallback.

All gating CI green on adfe86d (syntax + unit + e2e/Playwright, run 27245192866). Working tree clean; branch up to date with main.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $3.427067

📊 Context and tokens usage:

• 104.4K / 1M (10%) input tokens, 31.1K / 128K (24%) output tokens

Total: (3.3K new + 91.5K cache writes + 4.1M cache reads) input tokens, 31.1K output tokens, $3.427067 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus
• Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (1780KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

Double check again, and implement unimplemented or delayed, make sure we support all test cases abstract wikipedia project has. And also expand on our use cases and features, to make sure we support highest possible quality of everything.

If you see clearly wrong tests (in the entire codebase), that can be improved - do it.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-10T09:26:39.975Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

✅ Follow-up round complete

Addressed the latest request ("implement unimplemented or delayed, support all test cases Abstract Wikipedia has, expand features, fix clearly-wrong tests") in this same PR:

New grammatical feature — Romance gender agreement

• romanceIndefiniteArticle(lang, gender) agrees the es/fr indefinite article (un/una, un/une) with the object noun's grammatical gender (Wikidata Lexeme P5185), masculine default. instance_of es/fr templates now use the agreeing {article}.
• Fixed clearly-wrong tests: the prior es/fr assertions dropped the article entirely — corrected to Berlín es una ciudad, Berlin est une ville, etc.

New constructor — quantity (measurement)

• subject · value · unit renders across all six UN languages (the analogue of a Wikidata quantity claim, e.g. P2048 height). toConstructor emits it when a number-with-unit is present, so Mount Everest is 8848 meters tall round-trips instead of losing its value.

Closed the documented analysis gaps on transformToConstructor: detectQuestion (wh + polar), extractQuantities, dedupeSequence (adjacent-duplicate collapse, on by default).

Limitation harness now reflects reality — limitation-test.mjs scores the structured pipeline; documented problems 12 → 4 (remaining four are knowledge-base/data limits, not transformer logic). limitations-found.json regenerated.

UI — Generation mode is role-aware: quantity surfaces Value+Unit inputs, an Object-gender selector appears for Romance-article templates; added examples, demo runners, and an offline e2e test.

Tests — qp-to-text.test.mjs now 36/36 offline; all gating suites pass. CI green on a7b46ef (run).

Requirements R15–R20 in docs/case-studies/issue-18/requirements.md track this round.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $10.184543

📊 Context and tokens usage:

Claude Opus 4.8: (3 sub-sessions)

1. 114.7K / 1M (11%) input tokens, 13.5K / 128K (11%) output tokens
2. 116.5K / 1M (12%) input tokens, 34.4K / 128K (27%) output tokens
3. 94.4K / 1M (9%) input tokens, 30.5K / 128K (24%) output tokens

Total: (22.0K new + 292.0K cache writes + 12.0M cache reads) input tokens, 90.3K output tokens, $10.184543 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus
• Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (5211KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag

[konard]

Make sure our roadmap, vision and all requirements listed in the repository are fully and correctly implemented. Double check tests for obviously wrong cases and fix them also.

Please plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-06-10T23:35:25.710Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

[konard]

Working session summary

CI is green on the latest commit (34ebeb0 — JS / success). Everything is committed, pushed, and verified.

Done

The standing request — verify all requirements are correctly implemented and fix obviously-wrong test cases — is complete for this session. The one genuinely obviously-wrong case was the Russian locative output.

The fix (commit 34ebeb0 on issue-18-ef8b353c, PR #19):

• The ru located_in template emitted the nominative Wikidata label after the preposition «в» — «Берлин находится в Германия» — which is ungrammatical, since «в» governs the prepositional case. A unit test had even blessed that wrong form.
• Added a guarded russianPrepositional helper in js/src/generation/constructors.js, wired via a non-invasive per-role inflect map applied in fillTemplate. It handles the regular endings (-ия→-ии, -а/-я→-е, -й→-е, consonant→+е) and deliberately leaves indeclinable foreign vowel endings (Токио, Перу), all-caps abbreviations (США), gender-ambiguous soft-sign nouns, and non-Cyrillic labels untouched — so it can only improve output, never regress it.
• Corrected the test assertion and added coverage: the helper across 19 inputs, plus present/past/future/negated renderer output.

Verification:

• All tense/polarity variants now render correctly: Берлин находится/находился/будет/не находится в Германии.
• test:syntax (45 files) and test:unit (all gating suites) pass; qp-to-text is 38/38.
• A medium-thorough Explore sweep of the entire test suite found no other obviously-wrong cases — the multi-language seeds (Arabic/Chinese nominal templates, plain-text label passthrough) are documented simplifications, not bugs.
• Docs updated to match: README.md, and the case-study README.md/requirements.md (R11, R21)/architecture.md.
• CI on the branch: green.

The documented round-trip examples were checked against live code output and match exactly.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $5.436513

📊 Context and tokens usage:

Claude Opus 4.8: (2 sub-sessions)

1. 116.9K / 1M (12%) input tokens, 30.5K / 128K (24%) output tokens
2. 42.9K / 1M (4%) input tokens, 5.0K / 128K (4%) output tokens

Total: (11.1K new + 218.6K cache writes + 5.3M cache reads) input tokens, 41.2K output tokens, $5.113437 cost

Claude Haiku 4.5:

• 72.7K / 200K (36%) input tokens, 7.2K / 64K (11%) output tokens

Total: (100 new + 72.6K cache writes + 2.0M cache reads) input tokens, 7.2K output tokens, $0.323076 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus
• Main model: Claude Opus 4.8 (claude-opus-4-8)
• Additional models:
  • Claude Haiku 4.5 (claude-haiku-4-5-20251001)

📎 Log file uploaded as Gist (3516KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

✅ Ready to merge

This pull request is now ready to be merged:

• All CI checks have passed
• No merge conflicts
• No pending changes

---

Monitored by hive-mind with --auto-restart-until-mergeable flag