refactor: Restructure training examples pipeline and remove automatic Firestore updates

  - Add download-training-examples.ts script to fetch examples from Firestore
  - Replace update-embeddings.ts with update-embeddings-from-md.ts for markdown-based processing
  - Remove automatic Firestore writes from code-generation-service:
    - Remove addTrainingExample() exported function
    - Remove storeSuccessfulGeneration() function and all its calls
    - Firestore updates now only via explicit scripts (no automatic writes)
  - Standardize on "lang" field name (was inconsistently "language" in some places)
  - Update embedding-service vectorSearch to filter by "lang" field
  - Transform retrieved examples to match new standardized format with proper field mapping
  - Update training markdown files with improved structure
  - Add new npm scripts: download-training-examples, update-embeddings-from-md

  BREAKING: Removed addTrainingExample() export from code-generation-service

Author: jeffdyer

AGENTS.md

@@ -1,37 +1,39 @@
 # Repository Guidelines
 
 ## Project Structure & Module Organization
-- `src/pages/`: Next.js routes and API handlers (`src/pages/api`).
-- `src/components/`: React UI components; `src/hooks/` custom hooks.
-- `src/lib/` core services (auth, API, embeddings); `src/utils/` helpers.
-- `src/styles/` global and editor styles; `public/` static assets.
-- `docs/`, `scripts/` (utility scripts), `training/` (example and data assets).
+- `src/pages/`: Next.js routes and API (`src/pages/api`).
+- `src/components/`: React components; `src/hooks/` custom hooks.
+- `src/lib/`: Core services (auth, API, embeddings); `src/utils/` helpers.
+- `src/styles/`: Global/editor styles; `public/`: static assets.
+- `docs/`, `scripts/` (utility scripts), `training/` (examples/data).
 
 ## Build, Test, and Development Commands
-- `npm run dev`: Start the Next.js dev server on `http://localhost:3000`.
-- `npm run build`: Create a production build.
+- `npm run dev`: Start dev server at `http://localhost:3000`.
+- `npm run build`: Create production build.
 - `npm run start`: Serve the production build.
-- `npm run lint` / `npm run lint:fix`: Lint (and auto-fix) the codebase.
+- `npm run lint` / `npm run lint:fix`: Lint the codebase (and auto-fix).
 - `npm run typecheck`: Run TypeScript checks.
-- `npm run refresh`: Install local `graffiticode-*.tgz` language packages (dev convenience).
-- Requirements: Node `>=18.17`. Environment vars via `.env.local` (see `.env*` for hints).
+- `npm run refresh`: Install local `graffiticode-*.tgz` language packages.
+- Requirements: Node `>=18.17`. Configure env in `.env.local` (see `.env*`).
 
 ## Coding Style & Naming Conventions
-- Language: TypeScript, React 18, Next.js 15, TailwindCSS for styling.
-- Linting: ESLint (extends `next` and `next/core-web-vitals`). Keep code lint‑clean and type‑safe.
-- Indentation: 2 spaces; prefer concise, functional components and hooks.
-- Naming: PascalCase for React components/files in `src/components` (e.g., `NewAPIKeyDialog.tsx`); camelCase for variables/functions; lower-case file names for pages (e.g., `src/pages/items.tsx`).
-- Styling: Prefer Tailwind utility classes; group related classes logically.
+- Language: TypeScript, React 18, Next.js 15; TailwindCSS for styling.
+- Linting: ESLint (extends `next`, `next/core-web-vitals`). Keep lint- and type-clean.
+- Indentation: 2 spaces; prefer concise functional components and hooks.
+- Naming: PascalCase for components in `src/components` (e.g., `NewAPIKeyDialog.tsx`);
+  camelCase for variables/functions; page files lower-case in `src/pages` (e.g., `items.tsx`).
+- Styling: Use Tailwind utility classes; group related utilities logically.
 
 ## Testing Guidelines
-- No formal test runner is configured. When adding complex logic, include lightweight component or unit tests (e.g., Jest + React Testing Library) in the PR if feasible.
-- At minimum, ensure `npm run lint` and `npm run typecheck` pass and exercise critical paths manually.
+- No formal runner configured. For complex logic, include lightweight Jest + RTL tests in PRs if feasible.
+- Minimum bar: `npm run lint` and `npm run typecheck` must pass; manually exercise critical paths.
+- Place component/unit tests near sources or under `__tests__` when added; name files `*.test.ts(x)`.
 
 ## Commit & Pull Request Guidelines
-- Commits: Use clear, imperative messages (e.g., "fix: address build error", "feat: add editor timeline"). Keep changes focused.
-- PRs: Provide a summary, linked issues, screenshots/GIFs for UI changes, and steps to verify locally.
-- Checks: PRs should be lint-clean, type-clean, and build successfully. Note any config or data prerequisites.
+- Commits: Clear, imperative messages (e.g., `fix: address build error`, `feat: add editor timeline`). Keep changes focused.
+- PRs: Provide a summary, linked issues, and screenshots/GIFs for UI changes. Include steps to verify locally and any config/data prerequisites.
+- Checks: PRs should lint, typecheck, and build successfully.
 
 ## Security & Configuration Tips
-- Store secrets in `.env.local` (do not commit). Typical keys: NextAuth secret/URL, Stripe keys, provider credentials. Client-side Firebase config is embedded; server credentials must remain private.
-- Avoid logging secrets; scrub PII in debug output.
+- Store secrets in `.env.local` only. Typical keys: NextAuth secret/URL, Stripe, provider creds.
+- Avoid logging secrets; scrub PII in debug output. Client-side Firebase config is public; server creds must remain private.

package.json

@@ -13,7 +13,10 @@
     "start": "cross-env NODE_NO_WARNINGS=1 next start",
     "lint": "next lint",
     "lint:fix": "eslint --fix \"src/**/*.{ts,tsx}\"",
-    "typecheck": "tsc"
+    "typecheck": "tsc",
+    "download-training-examples": "node --experimental-strip-types scripts/download-training-examples.ts",
+    "update-embeddings": "node --experimental-strip-types scripts/update-embeddings.ts --collection training_examples",
+    "update-embeddings-from-md": "node --experimental-strip-types scripts/update-embeddings-from-md.ts"
   },
   "dependencies": {
     "@codemirror/autocomplete": "^6.18.6",