prompt-git-manager

Git-native prompt version control & CI guardrail tool

Installation • Quick Start • Commands • CI Integration • Docs • Contributing • 中文

Why prompt-git-manager?

The Problem

Prompt engineering is becoming critical to AI applications, but managing prompts is chaotic:

🔀 No version control: Prompts scattered in code, docs, and chat logs
📊 No metrics: Can't measure if changes improve or degrade performance
🚫 No guardrails: Breaking changes ship without detection
🔍 No diff tools: Text diff is meaningless for structured prompts

The Solution

prompt-git-manager brings software engineering best practices to prompt management:

Feature	Traditional Approach	prompt-git-manager
Version Control	Copy-paste in docs	Git-native commits
Change Detection	Manual review	Semantic diff
Quality Gates	Hope for the best	Automated evaluation
Rollback	"What was the old prompt?"	`git checkout`

Key Differentiators

Zero Infrastructure: No servers, no databases, no SaaS dependencies
Git Native: Prompts are files, versions are commits
CI First: Built for GitHub Actions, pre-commit, and PR workflows
Offline Capable: Works without LLM API access (rule-based evaluation)
LLM Enhanced: Optional LLM-as-judge evaluation with multi-provider support (OpenAI, Anthropic, local models)

Documentation

📖 Full Documentation

Document	Description
Quick Start	Get started in 5 minutes
CLI Reference	All commands detailed
Prompt Schema	Prompt file format spec
Evaluation Guide	Evaluation complete reference
Dataset Guide	Create evaluation datasets
Configuration	Configuration reference
Python API	Programmatic usage
Architecture	Internal implementation
Best Practices	Prompt management tips
Migration Guide	Migrate from other tools
Troubleshooting	Common issues & solutions
Roadmap	Future plans

Installation

Using uv (Recommended)

uv pip install prompt-git-manager

Using pip

pip install prompt-git-manager

From Source

git clone https://github.com/ChanChiChoi/prompt-git-manager.git
cd prompt-git-manager
uv sync

Verify Installation

pg --version
# prompt-git-manager 0.1.0

Quick Start

1. Initialize a Project

cd your-project
pg init

This creates:

.prompts/
├── config.json      # Project settings
└── .gitignore       # Internal files

2. Add Your Prompts

# Create a prompt file
cat > qa_prompt.yaml << 'EOF'
name: qa-assistant
version: "1.0.0"
system_prompt: "You are a helpful assistant."
user_template: "Answer: {{question}}"
variables:
  question:
    type: string
    default: "What is Python?"
constraints:
  - Be concise
  - Use examples
metadata:
  author: your-name
EOF

# Add to tracking
pg add qa_prompt.yaml

3. Commit Changes

pg commit -m "Initial QA prompt"

4. Review Changes

# Make some changes to the prompt...
vim .prompts/qa_prompt.yaml

# See semantic diff
pg diff --semantic

5. Evaluate Against Dataset

# Create a test dataset
cat > fixtures/dataset.jsonl << 'EOF'
{"input": "What is Python?", "expected_output": "Python is a programming language"}
{"input": "What is Git?", "expected_output": "Git is a version control system"}
EOF

# Run evaluation
pg eval --dataset fixtures/dataset.jsonl --threshold 0.05

Commands

`pg init`

Initialize prompt-git-manager in your repository.

pg init [--dry-run]

`pg add`

Add a prompt file to version tracking.

pg add <file> [--dry-run]

Supported formats: YAML (.yaml, .yml), JSON (.json)

Required fields:

name: Prompt identifier
system_prompt: System message
user_template: User message template with {{variables}}

`pg commit`

Commit prompt changes with structured metadata.

pg commit -m "message" [--dry-run]

Generates commit record:

{
  "hash": "abc123",
  "timestamp": "2026-05-04T10:30:00",
  "changed_files": [".prompts/qa_prompt.yaml"],
  "validation_status": "pass",
  "message": "Update QA prompt"
}

`pg diff`

Show differences between prompt versions.

pg diff [file] [--semantic] [--json]

Semantic Analysis:

Variable changes ({{old_var}} → {{new_var}})
Constraint changes (added/removed rules)
Tone shifts (formal ↔ casual)
Role shifts (assistant persona changes)

Risk Levels:

🟢 LOW: Minor changes, no semantic impact
🟡 MEDIUM: Constraint or tone changes
🔴 HIGH: Role or variable removal

`pg eval`

Evaluate prompts against a dataset.

# Rule-based evaluation (offline, no LLM dependency)
pg eval --dataset <file.jsonl> [--threshold 0.05] [--json]

# LLM-enhanced evaluation
pg eval --dataset <file.jsonl> --provider openai --model gpt-4

# LLM-as-judge evaluation (more accurate scoring)
pg eval --dataset <file.jsonl> --provider openai --model gpt-4 --judge

# Compare multiple models
pg eval --dataset <file.jsonl> --compare-models gpt-3.5,gpt-4

Dataset Format:

{"input": "question", "expected_output": "answer", "metadata": {}}

Metrics:

accuracy_delta: Change in accuracy (-1 to +1)
token_cost_delta: Change in token usage
consistency_score: Agreement between versions (0-1)

LLM Providers:

OpenAI (gpt-3.5, gpt-4, etc.)
Anthropic (claude-2, claude-3, etc.)
Ollama (local models)
Any LiteLLM-supported provider

`pg ci init`

Generate CI/CD configuration files.

pg ci init [--dry-run]

Generates:

.github/workflows/prompt-guard.yml - GitHub Actions workflow
.pre-commit-config.yaml - Pre-commit hooks
scripts/bump_version.sh - Version management

CI Integration

GitHub Actions

Automatic Setup

pg ci init

Manual Setup

Create .github/workflows/prompt-guard.yml:

name: Prompt Guard

on:
  pull_request:
    paths:
      - '.prompts/**'

jobs:
  prompt-guard:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'

      - name: Install prompt-git-manager
        run: pip install prompt-git-manager

      - name: Run diff
        run: pg diff --semantic --json > diff.json

      - name: Run evaluation
        run: pg eval --dataset fixtures/dataset.jsonl --threshold 0.05

      - name: Comment PR
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const diff = fs.readFileSync('diff.json', 'utf8');
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## ❌ Prompt Guard Failed\n\n\`\`\`json\n${diff}\n\`\`\``
            });

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: prompt-diff
        name: Prompt Diff Check
        entry: pg diff --fail-on=high
        language: system
        files: '\.prompts/.*\.ya?ml$'
        pass_filenames: false

Install hooks:

pre-commit install

Local CI Script

#!/bin/bash
# scripts/ci_check.sh

set -e

echo "Running prompt checks..."

# Run diff
pg diff --semantic --json > diff.json

# Run evaluation
pg eval --dataset fixtures/dataset.jsonl --threshold 0.05 --json > eval.json

echo "All checks passed!"

Configuration

Project Config

.prompts/config.json:

{
  "version": "0.1.0",
  "eval_threshold": 0.05,
  "model_provider": "openai",
  "default_model": "gpt-3.5-turbo",
  "auto_validate": true
}

Environment Variables

Variable	Description	Default
`PROMPT_GIT_MODEL`	LLM model for evaluation	`none`
`PROMPT_GIT_THRESHOLD`	Default eval threshold	`0.05`
`OPENAI_API_KEY`	OpenAI API key	-
`ANTHROPIC_API_KEY`	Anthropic API key	-

Benchmark

Performance

Operation	Time	Notes
`pg init`	<100ms	Creates directory structure
`pg add`	<200ms	Validates + copies file
`pg commit`	<500ms	Git commit + record
`pg diff`	<300ms	Structured diff analysis
`pg eval` (20 samples)	<1s	Rule-based evaluation
`pg eval` (100 samples)	<5s	Rule-based evaluation

Test Coverage

Module	Coverage
cli.py	42%
schema.py	90%
diff_engine.py	90%
evaluator.py	99%
ci_gen.py	-
Total	74%

Test Count

Suite	Tests
test_cli.py	14
test_diff.py	29
test_eval.py	33
test_ci_gen.py	40+
Total	116+

Architecture

prompt-git-manager/
├── src/promptgit/
│   ├── __init__.py          # Version
│   ├── cli.py               # Typer CLI entry point
│   ├── schema.py            # Pydantic models
│   ├── diff_engine.py       # Semantic diff engine
│   ├── evaluator.py         # Dataset evaluation
│   ├── ci_gen.py            # CI/CD generator
│   └── utils.py             # Git + Rich helpers
├── tests/
│   ├── conftest.py          # Fixtures
│   ├── test_cli.py
│   ├── test_diff.py
│   ├── test_eval.py
│   └── test_ci_gen.py
├── fixtures/
│   ├── dataset.jsonl        # Test dataset
│   └── prompts/             # Edge case prompts
├── examples/
│   ├── customer_service.yaml
│   ├── code_generation.yaml
│   └── data_extraction.yaml
├── docs/
│   ├── cli_reference.md
│   └── architecture.md
└── .github/
    └── workflows/
        ├── prompt-guard.yml
        └── publish.yml

Contributing

Development Setup

# Clone repository
git clone https://github.com/ChanChiChoi/prompt-git-manager.git
cd prompt-git-manager

# Install with dev dependencies
uv sync --extra dev

# Run tests
uv run pytest

# Run with coverage
uv run pytest --cov=promptgit --cov-report=html

Code Style

Python 3.10+ with type hints
Pydantic for data validation
Typer for CLI
Rich for terminal output
pytest for testing

Pull Request Process

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open a Pull Request

Running Checks

# Type checking
uv run mypy src/

# Linting
uv run ruff check src/

# Format
uv run ruff format src/

# All tests
uv run pytest -v

Releasing

# Bump version
./scripts/bump_version.sh patch  # or minor, major

# Push with tags
git push && git push --tags

# GitHub Action will publish to PyPI automatically

Quick PR Demo with gh CLI

Create a PR with Prompt Changes

# 1. Create feature branch
git checkout -b feature/update-qa-prompt

# 2. Make prompt changes
vim .prompts/qa_prompt.yaml

# 3. Commit with prompt-git-manager
pg commit -m "Improve QA prompt accuracy"

# 4. Push branch
git push -u origin feature/update-qa-prompt

# 5. Create PR with gh CLI
gh pr create \
  --title "Improve QA prompt accuracy" \
  --body "$(cat <<'EOF'
## Summary
- Updated system prompt for better context understanding
- Added few-shot examples to user template
- Adjusted constraints for more consistent outputs

## Prompt Diff
$(pg diff --semantic)

## Evaluation Results
$(pg eval --dataset fixtures/dataset.jsonl --json)

## Checklist
- [x] Semantic diff reviewed
- [x] Evaluation passed (threshold: 5%)
- [ ] Team review
EOF
)"

# 6. View PR
gh pr view --web

Check PR Status

# List open PRs
gh pr list

# View specific PR
gh pr view 42

# Check CI status
gh pr checks 42

# Merge when ready
gh pr merge 42 --squash

FAQ

Q: Why not use Langfuse/Weights & Biases?

A: Those are great runtime monitoring tools. prompt-git-manager focuses on development-time workflow:

Git-native (no new tool to learn)
CI-first (catches issues before deploy)
Zero infrastructure (no servers to maintain)

Q: Can I use this with private prompts?

A: Yes! Prompts stay in your private Git repo. No data is sent externally unless you enable LLM evaluation.

Q: How does rule-based evaluation work?

A: Without LLM APIs, we use keyword matching and text similarity as heuristics. It's less accurate but:

Works offline
No API costs
Fast execution
Deterministic results

Q: What prompt formats are supported?

A: YAML and JSON with this structure:

name: string
version: string
system_prompt: string
user_template: string  # with {{variables}}
messages: []           # optional, multi-turn history [{role, content}]
variables: {}
constraints: []
metadata: {}

License

MIT License - see LICENSE for details.

Acknowledgments

Typer - CLI framework
Pydantic - Data validation
GitPython - Git integration
Rich - Terminal formatting

Made with ❤️ for the AI engineering community

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
.github/workflows		.github/workflows
docs		docs
examples		examples
fixtures		fixtures
scripts		scripts
src/promptgit		src/promptgit
templates		templates
tests		tests
.gitignore		.gitignore
.helper.md		.helper.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README_zh.md		README_zh.md
SECURITY.md		SECURITY.md
pyproject.toml		pyproject.toml
spec.md		spec.md

Folders and files

Latest commit

History

Repository files navigation

prompt-git-manager

Why prompt-git-manager?

The Problem

The Solution

Key Differentiators

Documentation

Installation

Using uv (Recommended)

Using pip

From Source

Verify Installation

Quick Start

1. Initialize a Project

2. Add Your Prompts

3. Commit Changes

4. Review Changes

5. Evaluate Against Dataset

Commands

pg init

pg add

pg commit

pg diff

pg eval

pg ci init

CI Integration

GitHub Actions

Automatic Setup

Manual Setup

Pre-commit Hooks

Local CI Script

Configuration

Project Config

Environment Variables

Benchmark

Performance

Test Coverage

Test Count

Architecture

Contributing

Development Setup

Code Style

Pull Request Process

Running Checks

Releasing

Quick PR Demo with gh CLI

Create a PR with Prompt Changes

Check PR Status

FAQ

Q: Why not use Langfuse/Weights & Biases?

Q: Can I use this with private prompts?

Q: How does rule-based evaluation work?

Q: What prompt formats are supported?

License

Acknowledgments

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`pg init`

`pg add`

`pg commit`

`pg diff`

`pg eval`

`pg ci init`