SkillCycle

Automatically improve AI agent performance by learning from validation errors and generating skill updates.

What It Does

1. Tests applications - Validates apps against criteria
2. Detects patterns - Identifies recurring errors automatically
3. Generates skills - Creates improvement guidelines
4. Measures impact - Tracks effectiveness of skills
5. Evolves continuously - Never stops learning

How It Works

Test Apps → Collect Errors → Detect Patterns → Generate Skills → Apply to AI → Test Again → Measure

Example

# 1. Test applications
python3 automate_test.py --evaluate test-apps/app1 --app-id APP001
python3 automate_test.py --evaluate test-apps/app2 --app-id APP002

# 2. Analyze patterns
python3 skill_updater.py analyze
# Output: Found 4 patterns (json_syntax: 5, security: 3, tests: 2)

# 3. Generate skill
python3 skill_updater.py generate
# Creates: skills/app-quality-<patterns>/SKILL.md

# 4. Apply skill to your AI agent
cp skills/app-quality-*/SKILL.md ~/.cursor/skills/

# 5. Test with improved agent
python3 automate_test.py --evaluate test-apps/app3 --app-id APP003

# 6. Measure effectiveness
python3 skill_updater.py measure --version v1
# Output: 75% effective (3/4 patterns resolved)

Quick Start

Setup Test Criteria

python3 setup_test.py MY_APP

# Paste criteria (plain text or JSON):
Requirements:
- Feature 1
- Feature 2

Expected Files:
- config.json
- main.py

Type 'END' and press Enter

Evaluate Existing App

# Copy your app
cp -r /path/to/app test-apps/MY_APP/

# Evaluate
python3 automate_test.py --evaluate test-apps/MY_APP --app-id MY_APP

# With saved criteria
python3 automate_test.py --evaluate test-apps/MY_APP --requirements test-criteria/MY_APP-criteria.json

Commands

skill_updater.py

# Analyze error patterns
python3 skill_updater.py analyze

# Generate skill update
python3 skill_updater.py generate

# Measure effectiveness
python3 skill_updater.py measure --version v1

# View evolution report
python3 skill_updater.py report

# List all skills
python3 skill_updater.py list

automate_test.py

# Evaluate app
python3 automate_test.py --evaluate APP_PATH --app-id APP_ID

# With requirements
python3 automate_test.py --evaluate APP_PATH --app-id APP_ID --requirements "Feature 1,Feature 2"

# With criteria file
python3 automate_test.py --evaluate APP_PATH --app-id APP_ID --requirements criteria.json

Features

Built-in Validators (8)

• JSON syntax validation
• README quality checks
• Code quality metrics
• Security scanning (hardcoded secrets, credentials)
• Documentation quality
• Dependency management (Python, Node.js, Ruby, Java)
• Test coverage analysis
• Project structure validation

Skill Learning

• Automatic pattern detection from errors
• Severity classification (critical, high, medium, low)
• Skill generation with actionable guidelines
• Effectiveness measurement
• Evolution tracking

Pluggable Architecture

Add custom validators by creating validators/your_validator.py:

class YourValidator:
    def validate(self, app_path, criteria):
        # Your validation logic
        return {
            'passed': True,
            'message': 'All checks passed',
            'details': {}
        }

Scoring

• 100-point scale with letter grades (A-F)
• File Structure (40 points): Required files present
• Custom Validators (60 points): Validation checks
• Grade Scale: A (90-100) • B (80-89) • C (70-79) • D (60-69) • F (<60)

Project Structure

skill-cycle/
├── automate_test.py           # Main testing framework
├── skill_updater.py           # Skill learning engine
├── setup_test.py              # Test criteria setup
├── error_learner.py           # Error pattern detection
├── convert_criteria.py        # Criteria converter
├── validators/                # Pluggable validators (8)
│   ├── json_validator.py
│   ├── readme_validator.py
│   ├── code_quality_validator.py
│   ├── security_validator.py
│   ├── documentation_validator.py
│   ├── dependency_validator.py
│   ├── test_validator.py
│   └── structure_validator.py
├── test-apps/                 # Applications to test
├── test-criteria/             # Validation criteria
├── results/                   # Test results (JSON)
├── skills/                    # Generated skills
└── .dev/                      # Error learning data
    ├── comparison/error_database.json
    └── learning/skill_history.json

Use Cases

1. Improve AI Code Generation

• Test AI-generated code
• Learn from errors
• Generate skills
• Apply to AI agent
• Generate better code

2. Continuous Quality Improvement

• Run tests continuously
• Detect patterns automatically
• Generate improvements
• Measure effectiveness

3. Compliance Automation

• Define compliance criteria
• Test existing apps
• Learn from failures
• Generate compliance skills

Requirements

• Python 3.7+
• No external dependencies

Tips

See SKILL_UPDATER.md for detailed tips on skill generation and improvement.

License

MIT License - see LICENSE file