Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 44 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
# Interview Coach

AI-powered interview coaching platform. Users select an interviewer, listen to a spoken question, record their answer, and receive structured feedback on delivery, tone, and answer quality.
AI-powered interview coaching platform. Users select an interviewer, choose feedback timing, listen to a spoken question, record their answer, and receive structured feedback on delivery, tone, and answer quality.

## How it works

1. User selects an interviewer voice and presses **Start Interview**
1. User selects an interviewer voice and feedback timing, then presses **Start Interview**
2. Interviewer introduces themselves via TTS (`POST /speech/tts`)
3. Interviewer reads the question aloud; recording starts automatically
4. User answers and presses **I'm Done Answering**
5. Audio → Groq Whisper → timestamped transcript segments (`POST /speech/transcribe`)
6. Each segment → local wav2vec2 model → arousal / dominance / valence scores
7. Scores + transcript displayed per segment on the interview screen
8. *(Coming)* Transcript + scores + question → LLM → structured feedback (`POST /feedback/generate`)
9. *(Coming)* Frontend renders full scorecard
8. Transcript + scores + question → Groq LLM → structured feedback (`POST /feedback/generate`)
9. Scorecard renders per-question feedback (strengths, improvements, delivery notes, model answer, transcript scores) or full-session summary at end of interview (`POST /feedback/generate-session`)

## Prerequisites

Install these once at the OS level before running setup:

- **Python 3.9+**
- **Node.js 18+** — download from https://nodejs.org (LTS release)
- **pnpm** — `npm install -g pnpm`
- **ffmpeg** — required to decode audio files:

Expand Down Expand Up @@ -54,8 +55,8 @@ cp backend/.env.example backend/.env
Open `backend/.env` and fill in your keys:

```
GROQ_API_KEY=your_key_here # https://console.groq.com (speech-to-text + TTS)
ANTHROPIC_API_KEY=your_key_here # https://console.anthropic.com (LLM feedback, coming)
GROQ_API_KEY=your_key_here # https://console.groq.com (TTS + speech-to-text + LLM feedback)
ANTHROPIC_API_KEY=your_key_here # https://console.anthropic.com (reserved)
OPENAI_API_KEY=your_key_here # Reserved
```

Expand Down Expand Up @@ -116,16 +117,26 @@ backend/
requirements.txt — Python dependencies
services/
tone_delivery_analyzer/ — local wav2vec2 emotion model (arousal/dominance/valence)
speech_to_text/ — Groq Whisper transcription + TTS
llm/ — LLM feedback generation (coming)
text_analysis/ — transcript scoring (coming)
speech_to_text/ — Groq Whisper transcription + Orpheus TTS
llm/ — Groq LLM feedback generation (/feedback/generate, /feedback/generate-session)
text_analysis/ — transcript scoring (reserved)

frontend/
app/
page.tsx — interview UI (home page)
InterviewClient.tsx — full interview state machine
InterviewClient.tsx — full interview state machine (recording, VAD, TTS, feedback)
layout.tsx / globals.css — root layout and styles
scorecard/ — scorecard display components
components/
MicWaveform/ — live mic waveform visualizer
scorecard/
ScorecardPanel.tsx — per-question scorecard (calls /feedback/generate, renders result)
SessionScorecardPanel.tsx — full-session scorecard (calls /feedback/generate-session)
components/
Scorecard/ — top-level scorecard layout
DeliveryScores/ — arousal/dominance/valence score display
TranscriptFeedbackScores/ — clarity/structure/relevance/conciseness display
QualitativeFeedback/ — strengths, improvements, delivery notes
ModelAnswer/ — example answer display
dev/
flow/ — pipeline visualization dev page
transcribe/ — transcription dev/debug page
Expand All @@ -137,6 +148,11 @@ frontend/
types.ts — shared TypeScript types
mocks.ts — mock data for UI development
pipelineStages.ts — pipeline stage definitions
sessionAdapter.ts — builds ReviewContextPayload from question + segments
feedbackSpeech.ts — converts feedback responses to TTS scripts
aggregateReviewPayload.ts — aggregates per-question answers into session payload
speech/
tts.ts — frontend helper for POST /speech/tts

docs/
architecture.md — system architecture and data flow
Expand All @@ -155,11 +171,25 @@ docs/
| Layer | Technology |
| ---------------- | -------------------------------------------------------------- |
| Frontend | Next.js 16, React 19, TypeScript, Tailwind CSS 4 |
| Component docs | Storybook 10 |
| Backend | FastAPI (Python) |
| Text-to-speech | Groq Orpheus (`canopylabs/orpheus-v1-english`) |
| Speech-to-text | Groq Whisper (`whisper-large-v3-turbo`) |
| Tone/delivery | `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` (local)|
| LLM feedback | Anthropic Claude (coming) |
| LLM feedback | Groq (`openai/gpt-oss-20b`) |
| Package manager | pnpm |
| Frontend deploy | Vercel (planned) |
| Backend deploy | Render or Fly.io (planned) |
| Frontend deploy | TBD |
| Backend deploy | TBD |

## Backend endpoints

| Method | Path | Status | Purpose |
| ------ | --------------------------- | -------- | -------------------------------------------------------------- |
| GET | `/health` | done | Liveness check |
| GET | `/emotion/health` | done | Confirms emotion model is loaded |
| POST | `/speech/tts` | done | Text → WAV audio (Groq Orpheus TTS) |
| POST | `/speech/transcribe` | done | Audio → per-segment transcript + emotion scores |
| POST | `/emotion/analyze` | done | Audio → single arousal / dominance / valence score |
| POST | `/feedback/generate` | done | Transcript + scores + question → per-answer LLM feedback |
| POST | `/feedback/generate-session`| done | Multiple Q&A pairs → holistic session feedback + per-question reviews |
| POST | `/analysis/*` | reserved | Transcript text analysis (reserved) |
2 changes: 1 addition & 1 deletion backend/.env.example
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Copy this file to .env and fill in your keys before running the backend.
# .env is gitignored — never commit it.
# All three keys are required for the full pipeline to function.
# GROQ_API_KEY is required. ANTHROPIC_API_KEY and OPENAI_API_KEY are reserved and not needed for the full pipeline.

# Required for speech-to-text (Groq Whisper) AND LLM feedback (Groq gpt-oss-20b).
# https://console.groq.com
Expand Down
77 changes: 74 additions & 3 deletions backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ For setup and running instructions see the [root README](../README.md).
| POST | `/speech/tts` | done | Text → WAV audio via Groq Orpheus TTS |
| POST | `/speech/transcribe` | done | Audio file → per-segment transcript + emotion scores |
| POST | `/emotion/analyze` | done | Audio file → single arousal / dominance / valence score |
| POST | `/feedback/generate` | todo | Transcript + scores + question → structured LLM feedback |
| POST | `/feedback/generate` | done | Transcript + scores + question → per-answer LLM feedback (Groq) |
| POST | `/feedback/generate-session` | done | Multiple Q&A pairs → holistic session feedback + per-question reviews |

## Structure

Expand All @@ -26,9 +27,10 @@ services/
emotion_model.py — model class definitions (do not modify)
run_emotion.py — standalone CLI for testing the model directly
llm/
router.py — POST /feedback/generate (not yet implemented)
router.py — POST /feedback/generate and POST /feedback/generate-session (Groq LLM feedback)
schemas.py — Pydantic request/response models for feedback endpoints
text_analysis/
router.py — reserved for transcript scoring (not yet implemented)
router.py — reserved — no endpoints yet
```

## POST /speech/tts
Expand Down Expand Up @@ -93,6 +95,75 @@ curl -X POST http://localhost:8000/emotion/analyze \
-F "file=@path/to/audio.mp3"
```

## POST /feedback/generate

Generates per-answer feedback using Groq (`openai/gpt-oss-20b`). Accepts a question and full transcript with per-segment delivery scores; returns transcript quality scores, qualitative feedback, and a model answer.

**Request body (JSON):**
```json
{
"question": { "id": "1", "text": "Tell me about yourself." },
"transcript": {
"text": "I have three years of experience...",
"segments": [
{ "start": 0.0, "end": 2.4, "text": "I have three years...", "arousal": 0.61, "dominance": 0.55, "valence": 0.32 }
]
}
}
```

**Response:**
```json
{
"transcriptScores": { "clarity": 0.8, "structure": 0.7, "relevance": 0.9, "conciseness": 0.75 },
"feedback": {
"summary": "You answered confidently and stayed on topic.",
"strengths": ["Strong opening", "Specific examples"],
"improvements": ["Tighten the closing", "Vary your pace"],
"deliveryNotes": "Your arousal stayed high throughout — good energy."
},
"modelAnswer": { "text": "I have three years of experience in..." }
}
```

Requires `GROQ_API_KEY` in `backend/.env`.

## POST /feedback/generate-session

Generates holistic session feedback across all answers in a single Groq call. Accepts an array of question + transcript pairs; returns session-level summary and per-question reviews.

**Request body (JSON):**
```json
{
"answers": [
{
"question": { "id": "1", "text": "Tell me about yourself." },
"transcript": { "text": "...", "segments": [ ... ] }
}
]
}
```

**Response:**
```json
{
"overallSummary": "You showed consistent confidence across questions.",
"overallStrengths": ["Clear communication", "Strong examples"],
"overallImprovements": ["Improve conciseness", "Vary sentence structure"],
"overallDeliveryNotes": "Arousal trended high — sustained energy throughout.",
"questionReviews": [
{
"question": { "id": "1", "text": "Tell me about yourself." },
"transcriptScores": { "clarity": 0.8, "structure": 0.7, "relevance": 0.9, "conciseness": 0.75 },
"feedback": { "summary": "...", "strengths": [], "improvements": [], "deliveryNotes": "..." },
"modelAnswer": { "text": "..." }
}
]
}
```

Requires `GROQ_API_KEY` in `backend/.env`.

## Score interpretation

| Dimension | Low (→ 0) | High (→ 1) |
Expand Down
10 changes: 6 additions & 4 deletions docs/agent-workflows/branch-change-impact-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,18 @@ Check for:
- mixed old/new flows that prevent a clean fallback
- changes that partially switch a feature without fully gating it

### 3. Feature-flag involvement
### 3. Env-var or behavioral mode changes

If the diff touches anything related to feature flags or env-gated behavior (`featureFlag`, `featureGate`, `NEXT_PUBLIC_*` toggles, `flags`, or similar):
If the diff introduces or modifies `NEXT_PUBLIC_*` env vars used for behavioral gating (not just API base URL configuration), or adds new behavioral mode branches with distinct execution paths:

- Note which files and call sites are involved
- Ask the user: "Feature flag changes detected. Would you like me to run the feature flag gating review before continuing the audit?"
- Ask the user: "Behavioral gating changes detected. Would you like me to run the feature flag gating review before continuing the audit?"
- If the user says yes, read and run `docs/agent-workflows/feature-flag-gating-review.md` and include its output as a subsection before continuing
- If the user says no, note it and continue the audit without the flag review

If no flag involvement is detected, state that explicitly and continue.
This project has no formal feature flag registry. Simple UI state variables like `feedbackMode` do not require this review unless their branching logic is complex.

If no env-var or behavioral mode changes are detected, state that explicitly and continue.

### 4. Integration consistency

Expand Down
Loading
Loading