[FEATURE] Unify validator's three validation flows to reduce duplication

[epappas]

[FEATURE] Unify validator's three validation flows to reduce duplication

Problem Statement

The validator currently performs validation through three separate flows that have significant duplication:

1. Discovery Flow - Fetches miners from Bittensor metagraph
2. Scoring Flow - Verifies miners and calculates scores
3. Weight Setting Flow - Sets weights based on scores

Each flow independently fetches metagraph data, processes miner information, and manages state. This leads to:

• Redundant network calls to Bittensor
• Duplicate miner data processing logic
• Inconsistent state management
• Higher resource usage
• Maintenance burden

Proposed Solution

Create a unified validation pipeline with shared components:

1. Unified Metagraph Service - Single source for miner data with caching
2. Centralized Miner Repository - One place for all miner state
3. Event-Driven Architecture - Components communicate via events
4. Shared Validation Utilities - Common logic extracted
5. Single Scheduler - Manages all timed operations

Component

Validator

Priority Level

High

Checklist

Phase 1: Extract Shared Components

• Create MinerRegistryService in validator
  • Cache metagraph data with configurable TTL
  • Provide consistent miner filtering (validators vs miners)
  • Handle stake and UID lookups
  • Emit metagraph update events
• Extract validation utilities
  • Move is_invalid_endpoint() to common location
  • Create EndpointValidator struct
  • Add IP parsing utilities
  • Implement threshold validators
• Create shared data models
  • Define UnifiedMinerProfile struct
  • Standardize executor information format
  • Create consistent verification result type

Phase 2: Implement Event System

• Create event bus infrastructure
  • Define event types (MinerDiscovered, VerificationComplete, etc.)
  • Implement pub/sub mechanism
  • Add event persistence for recovery
• Convert discovery to event producer
  • Emit MinerDiscovered events
  • Remove direct database writes
  • Add event batching for efficiency
• Convert verification to event consumer/producer
  • Subscribe to MinerDiscovered events
  • Emit VerificationComplete events
  • Handle backpressure
• Convert weight setter to event consumer
  • Subscribe to VerificationComplete events
  • Remove polling of database
  • Add event aggregation logic

Phase 3: Unify Data Layer

• Create MinerRepository trait
  • Define CRUD operations
  • Add query methods for scores/profiles
  • Include transaction support
• Implement unified persistence
  • Consolidate database schemas
  • Create single source of truth
  • Add proper indexes
• Migrate existing code
  • Update discovery to use repository
  • Update verification to use repository
  • Update weight setter to use repository

Phase 4: Simplify Scheduling

• Create SchedulerService
  • Manage all timers in one place
  • Implement priority queue
  • Add rate limiting
  • Support dynamic intervals
• Define scheduling policies
  • Discovery interval (configurable)
  • Verification concurrency limits
  • Weight setting by block height
• Add monitoring
  • Track task execution times
  • Monitor queue depths
  • Alert on backlogs

Phase 5: Testing and Migration

• Write comprehensive tests
  • Unit tests for each component
  • Integration tests for event flow
  • Performance benchmarks
• Create migration plan
  • Feature flag for gradual rollout
  • Backwards compatibility layer
  • Rollback procedures
• Update documentation
  • Architecture diagrams
  • Event flow documentation
  • Configuration guide

Implementation Ideas

Example unified flow:

1. SchedulerService triggers discovery every N minutes
2. MinerRegistryService fetches/caches metagraph
3. Discovery emits batch of MinerDiscovered events
4. Verification consumes events, performs checks
5. Verification emits VerificationComplete events
6. Weight setter aggregates results
7. On block interval, weight setter submits to chain

Additional Context

Current duplication examples:

• Metagraph fetched 3+ times per cycle
• Endpoint validation logic in 2 files
• Miner data transformed multiple times
• Database queries overlap significantly

Expected benefits:

• 50% reduction in network calls
• Better resource utilization
• Easier to add new validation types
• Improved debugging and monitoring
• Cleaner, more maintainable code

Related Files

• crates/validator/src/miner_prover/discovery.rs
• crates/validator/src/miner_prover/verification.rs
• crates/validator/src/bittensor_core/weight_setter.rs
• crates/validator/src/miner_prover/scheduler.rs