Skip to content

README.md

Latest commit

 

History

History
208 lines (151 loc) · 10.5 KB

README.md

File metadata and controls

208 lines (151 loc) · 10.5 KB
Raw

VibeAudit

VibeAudit is a GitHub App that automates code audits using LLM agents. It runs as a standalone web service where users connect their GitHub account, select repositories, and receive audit reports by email.

  • Connect & select — install the GitHub App and grant access to specific repositories.
  • Audit — VibeAudit clones a selected repo and runs LLM agents over the code.
  • Report — findings are delivered as a report by email.
  • Access is short-lived — the GitHub App uninstalls itself automatically once your code has been cloned; it never keeps standing access to your account.
  • Sources aren't retained — the local clone is deleted once the audit finishes (default; staff can opt to keep it).

Important

Disclaimer — no warranty, advisory output. VibeAudit is provided "as is", without warranty of any kind, to the extent permitted by law (see LICENSE). It is an aid, not a substitute for professional security review. Audit reports are generated by LLMs and may contain false positives, miss real vulnerabilities, or be otherwise incorrect — do not rely on them as your sole assurance of code safety. Always have a qualified human review findings before acting on them. VibeAudit clones and sends the contents of the repositories you point it at to a third-party LLM provider (Anthropic by default); only run it on code you are authorized to share under that provider's terms.

Development

Prerequisites: uv, just, docker

Tasks are just recipes — run just to see them all.

Option A — Docker Compose (recommended)

Starts Traefik, Postgres, Redis, Django, Celery worker, and Vite — all with hot reload.

just dc-up      # alias: just dcu

First run — apply migrations, then (optionally) create an admin user:

just dc-migrate
just dc-shell   # shell into the backend container, e.g. to run createsuperuser
  • App: http://localhost:3000 (same port as the frontend dev server, so an ngrok session can be reused)
  • Admin: http://localhost:8000/admin

For the GitHub integration, place your .pem key at backend/secrets/github-app.pem and set GITHUB_APP_ID, GITHUB_APP_SLUG (see GitHub integration).

Option B — manual

cp backend/.env.example backend/.env  # set DEBUG=True, SECRET_KEY at minimum
just i        # install all dependencies
just migrate  # run DB migrations
just dbe      # Terminal 1 — backend on :8000
just dfe      # Terminal 2 — frontend on :3000 (proxies /api to backend)

GitHub integration (optional)

Required only to test the GitHub App OAuth flow locally. Needs a tunnel to expose localhost publicly — ngrok or equivalent.

1. Expose a public URL and note the HTTPS URL (used as <NGROK> below):

ngrok http 3000   # e.g. https://9647-46-205-207-234.ngrok-free.app

2. Create a GitHub App under Settings → Developer settings → GitHub Apps → New GitHub App:

Field Value
Homepage URL <NGROK>
Setup URL <NGROK>/api/github/setup
Redirect after installation checked
Webhook disabled
Permissions Repository: Contents → Read-only

Generate a private key (.pem), save it as backend/secrets/github-app.pem (gitignored), and note the App ID and App slug (github.com/apps/<slug>).

To pass the key as an environment variable instead (e.g. SOPS or .env), convert it to a single line with escaped newlines:

python3 backend/scripts/pem_to_env.py backend/secrets/github-app.pem | pbcopy

3. Configure — add to backend/.env:

GITHUB_APP_ID=<your app id>
GITHUB_APP_PRIVATE_KEY_PATH=secrets/github-app.pem
GITHUB_APP_SLUG=<your app slug>
ALLOWED_HOSTS=localhost,127.0.0.1,<NGROK host>
CSRF_TRUSTED_ORIGINS=<NGROK>
ALLOW_UNAUTHENTICATED_AUDIT=True

and to frontend/.env:

VITE_EXTRA_HOST=<NGROK host>

Open <NGROK>/connect to test the install flow.

Deployment

Build and push a production image, then run it:

IMAGE_REPO=ghcr.io/ulamlabs/vibeaudit IMAGE_TAG=main just build
just run   # runs the production image on port 8080

Runtime tuning (env)

  • LITELLM_LOCAL_MODEL_COST_MAP — True makes litellm use its bundled model-cost snapshot instead of fetching it from GitHub on import. The fetch is safe by default (httpx 5s timeout, then falls back to the bundled map — it never hangs), so leave it unset. Only set True on egress-restricted hosts to skip the network attempt entirely.
  • WEB_CONCURRENCY (default 2) / GUNICORN_THREADS (default 4) — gunicorn worker processes and threads per worker (the web command runs with --preload).
  • CELERY_CONCURRENCY (default 2) / CELERY_LOGLEVEL (default info) — Celery prefork children and log level (the worker command). Each child imports the AI stack, so this bounds worker memory and avoids over-forking to host-CPU-count on a CPU-limited host.

Sentry (optional)

  • Backend — set BACKEND_SENTRY_DSN in the runtime environment.

  • Frontend — pass FRONTEND_SENTRY_DSN at image build time (embedded as VITE_SENTRY_DSN):

    docker build --build-arg FRONTEND_SENTRY_DSN=https://<key>@o0.ingest.sentry.io/<project> -t ghcr.io/ulamlabs/vibeaudit:latest .

Templates

VibeAudit ships PDF-report and transactional-email templates. To author or restyle template content, see TEMPLATING.md — the authoritative reference for template variables, MJML/Django syntax, and WeasyPrint CSS. This section covers only what an operator needs to run and override them.

Email sources are MJML (backend/audit/templates/email/src/*.mjml); recompile after editing:

just compile-email   # alias: just ce

The transactional emails:

Template Sent when Sent to
new_submission_email.html A clone finished — job awaiting approval Staff (audit_notifications)
report_approval_email.html A run completed — report held for approval Staff (audit_notifications)
report_email.html A staff member approved the report Submitter
run_failure_email.html A run failed Staff (audit_notifications)
failure_email.html Cloning failed Staff (audit_notifications)

Submitters are never emailed automatically: the report is delivered only after a staff member approves it, and failures are reported to staff, who decide what to tell the submitter.

Approval flow

An audit passes two human gates:

  1. Job approval — a submitted repo is cloned, then held at awaiting_approval until staff approve it in the admin. Approving starts a run.
  2. Report approval — a completed run holds its report at awaiting_approval. Staff can edit the report body, then Approve & send or Reject report. Approving does not send synchronously: it transitions the report to approved and enqueues delivery. A worker then claims the run (approved -> sending) before emailing the submitter, so two racing workers (e.g. an impatient Resend) cannot both deliver the same report. On success the run moves to sent; on a delivery failure it rolls back to approved so Resend report can retry. approved therefore always means "queued or retryable," never "in flight." Reject report sends nothing, ever.

A run is flagged Needs attention in the admin (the same lazy-detection idiom used for overdue clones) under any of four conditions, each with its own remedy:

  • Stuck running — running past the suite's hard time limit (plus grace); the worker likely died without reaching failure handling. Remedy: Terminate execution.
  • Stranded send — a worker died mid-send, leaving the run at sending with a stale sending_since past a generous window. Remedy: Reset to approved, then Resend report.
  • Failed send — approved with sending_since still set (the failed-send rollback preserves it instead of clearing it); a send was attempted and failed, and nothing retries it automatically. Remedy: Resend report.
  • Send never claimed — approved with sending_since never set, but approved_at past that same window; the queue message that should have triggered the send was lost. Remedy: Resend report.

Cloned sources survive until every report on the job is resolved, so a rejected report can be followed by a new run with a different suite — add one from the AuditRun admin while the job is still Ready. Automatic close/cleanup only happens once a report is successfully sent and nothing else is outstanding (maybe_cleanup_sources runs solely from send_approved_report). A run that failed, or a report that was rejected, deliberately leaves the job Ready holding its clone — staff can re-run with another suite or click Delete sources manually — and such a job is flagged Needs attention in the meantime.

Operator notes:

  • Overriding — set EXTRA_EMAIL_TEMPLATES_DIR to an absolute path; matching filenames there take precedence over bundled ones, unmatched ones fall back to bundled.

  • Report subject — set via REPORT_EMAIL_SUBJECT (default: "VibeAudit Report").

  • Staff notifications — recipients must be is_staff=True and members of the audit_notifications group. The group is not created automatically — create it in Django admin and add staff to it, or no staff notifications are sent (the app otherwise works normally).

  • MJML gotcha — MJML strips bare {% %} tags during compilation; wrap Django tags in <mj-raw> so they survive:

    <mj-raw>{% if site_url %}</mj-raw>
<mj-button href="{{ site_url }}/...">...</mj-button>
<mj-raw>{% endif %}</mj-raw>

Contributing

Contributions are welcome. For anything non-trivial, open an issue to discuss it before sending a PR. Run just test (backend + frontend checks) and just lint locally first — CI runs the same. Unless you state otherwise, contributions you submit are licensed under the project's Apache-2.0 license (per section 5 of the license).

License

VibeAudit is licensed under the Apache License 2.0; see LICENSE for the full text. It is a permissive license: you're free to use, modify, and distribute VibeAudit — including in closed-source and commercial products — provided you retain the license and copyright notices and state any significant changes. It also includes an express grant of patent rights from contributors.