From cb796401afe8be64eb311a45e566fc315a13a00c Mon Sep 17 00:00:00 2001 From: vib-tools Date: Sat, 25 Jul 2026 17:54:36 -0700 Subject: [PATCH 1/2] feat: acquire GitHub repositories for analysis --- CHANGELOG.md | 11 + PROJECT_STATUS.md | 6 +- REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md | 194 ++++---- VERSION.json | 18 +- .../connected_accounts_module/README.md | 3 + .../internal/service.py | 145 +++++- .../connected_accounts_module/public.py | 31 +- .../connected_accounts_module/schemas.py | 12 +- .../repository_analysis_engine/README.md | 5 + backend/engines/repository_engine/README.md | 4 +- backend/engines/repository_engine/errors.py | 8 + .../repository_engine/internal/service.py | 116 ++++- backend/engines/repository_engine/schemas.py | 4 + backend/providers/github_provider/client.py | 377 +++++++++++++--- backend/providers/github_provider/errors.py | 11 + backend/providers/github_provider/schemas.py | 20 + ...b_repository_acquisition_analyzer_input.py | 420 ++++++++++++++++++ ...repository_analysis_current_state_audit.py | 40 +- ...CQUISITION_ANALYZER_INPUT_SPECIFICATION.md | 75 ++++ 19 files changed, 1259 insertions(+), 241 deletions(-) create mode 100644 backend/tests/test_github_repository_acquisition_analyzer_input.py create mode 100644 docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 149e94f..f5df044 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -408,3 +408,14 @@ Implemented Notification Engine with in-app notifications, unread count, mark-re - No new migration. - No new external dependency. - No real provider deployment execution in sandbox. + +## Update 1 — GitHub Repository Acquisition + Analyzer Input + +- Added ephemeral GitHub App installation credential acquisition through Connected Accounts public API. +- Added authenticated GitHub repository metadata retrieval. +- Added default-branch commit pinning and recursive Git tree acquisition. +- Added fail-closed handling for truncated or oversized tree responses. +- Added bounded UTF-8 acquisition for analysis-critical repository files. +- Persisted normalized snapshots in existing Repository Engine JSON fields; no migration added. +- Added commit-pinned analyzer input fields for tree SHA, snapshot completeness, and file contents. +- Preserved Cloudflare provider execution disabled. diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md index a633f5c..15079d5 100644 --- a/PROJECT_STATUS.md +++ b/PROJECT_STATUS.md @@ -46,13 +46,15 @@ Business logic remains outside the Dashboard. Providers are not imported directl ## Engine Matrix +Repository acquisition status: **commit-pinned snapshot implemented**. + | Engine/component | Status | Important remaining work | |---|---|---| | Auth Engine | Implemented | Controlled production identity/session validation | | Connected Accounts Module | Implemented | Live account/credential lifecycle validation under production configuration | | Project Engine | Implemented | No critical foundation gap identified | -| Repository Engine | Metadata persistence implemented; AG-002 standalone provider gate foundation added but not runtime-wired; live tree acquisition incomplete | GitHub App installation-token metadata/tree fetch, commit pinning, and private repository validation | -| Repository Analysis Engine | Quick-analysis contract implemented; current live input is incomplete | Actual repository tree acquisition, deep-analysis execution, Project reattachment after recalculation, and broader framework validation | +| Repository Engine | GitHub App installation-authenticated metadata, commit pinning, complete recursive tree snapshot, and analyzer-input handoff implemented; AG-002 standalone provider gate remains not runtime-wired | Controlled live private-repository validation and future provider-gate integration | +| Repository Analysis Engine | Quick-analysis contract now receives a commit-pinned complete snapshot and bounded text evidence | Strong analyzer rules, deep-analysis execution, Project reattachment after recalculation, and Cloudflare readiness expansion | | Deploy Engine | Implemented; AG-001 provider gate foundation added but not runtime-wired | Provider result reconciliation and reviewed future gate integration | | Deploy Pipeline | Concrete Cloudflare orchestration and trusted policy-to-binding handoff implemented; default runtime disabled | Controlled live provider validation and operational hardening | | Deployment History Engine | Pipeline intent, provider-result, terminal-failure, and retry-safe replay persistence integrated | Controlled PostgreSQL validation and operational monitoring | diff --git a/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md b/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md index 5bf6205..c51beba 100644 --- a/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md +++ b/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md @@ -1,176 +1,138 @@ # YGIT Repository Analysis Current-State Audit -Version: 1.0 -Status: Approved Current-State Audit -Date: 2026-07-22 +Version: 1.1 +Status: Implemented Current-State Record +Date: 2026-07-25 Owner: Repository Analysis Engine -Implementation Change: None +Implementation Change: Update 1 — GitHub Repository Acquisition + Analyzer Input ## Purpose -This document records the current Repository Analysis execution path and the exact gaps that prevent a repository with incomplete metadata from becoming deploy-ready. - -This patch documents current behavior only. It does not implement repository acquisition, deep-analysis execution, Project mutation, provider calls, or deployment. +This document records the repository acquisition and analyzer-input state after Update 1. It replaces the earlier metadata-only gap record while preserving the remaining deep-analysis and Project-reattachment gaps. ## Current Data Flow ```text Project Create - -> Repository Engine fetches GitHub repository metadata - -> Repository metadata is persisted - -> Repository Analysis Engine prepares analysis input - -> Quick Analysis reads file_tree_snapshot - -> framework, package manager, build command, output directory, - render mode, warnings, readiness, and score are calculated + -> Repository Engine requests an ephemeral GitHub installation credential + from Connected Accounts public API + -> GitHub Provider reads repository metadata with the installation token + -> GitHub Provider resolves the default branch head commit + -> GitHub Provider resolves the exact Git tree + -> the complete recursive tree and bounded analysis-critical text files + are normalized into a versioned snapshot + -> Repository Engine persists the commit-pinned snapshot + -> Repository Analysis Engine prepares a complete analyzer input + -> Quick Analysis reads the snapshot file paths -> Analysis result is stored -> Project is attached to Repository and Analysis - -> Project status reflects Analysis deploy readiness ``` ## Verified Current Behavior -### Project creation attaches the initial Analysis +### GitHub acquisition uses the connected GitHub App installation -Project creation fetches repository metadata, runs Quick Analysis with the new Project ID, and attaches the resulting Repository and Analysis records to the Project. +Repository Engine calls Connected Accounts through its public service boundary. Connected Accounts validates the stored installation reference and creates a short-lived installation access token server-side. -The Project becomes deploy-ready only when the stored Analysis reports `deploy_ready=true`. +The token is passed only to GitHub Provider. It is not persisted in Repository Engine fields, returned through API responses, written to job payloads, or included in the repository snapshot. -### Quick Analysis depends on repository file paths +### Repository acquisition is commit-pinned -Quick Analysis obtains its file list exclusively from: +GitHub Provider now resolves: ```text -RepositoryAnalysisInput.file_tree_snapshot +default branch +-> branch head commit SHA +-> Git commit tree SHA +-> recursive Git tree ``` -The file index recognizes path-like values under: +The snapshot stores the exact commit SHA and tree SHA. Repository Analysis rejects a snapshot when the persisted commit SHA and snapshot commit SHA do not match. -```text -path -name -file -filename -files -tree -items -children -paths -``` +### Tree acquisition fails closed -Framework, package-manager, static/dynamic, build-command, output-directory, readiness, score, warnings, and recommendations are derived from those extracted paths. +A truncated GitHub recursive-tree response is not accepted. Oversized tree responses are also rejected through the same incomplete-snapshot boundary. -### Current GitHub metadata is incomplete for real Analysis - -The current repository metadata path calls the GitHub repository metadata endpoint without an installation-token authorization header. - -It currently stores: +The engine returns: ```text -latest_commit_sha = None -file_tree_snapshot = {"default_branch": ""} +REPOSITORY_SNAPSHOT_INCOMPLETE ``` -The default branch name is metadata, not a repository file path. It does not provide `package.json`, framework configuration, source files, or build-output evidence. +Partial repository analysis is not treated as deploy-ready evidence. -Therefore the current Quick Analysis path can legitimately return unknown framework/output/build information and keep the Project blocked. +### Analyzer input includes bounded text evidence -### Deploy readiness is fail-closed +The snapshot contains: -Readiness is blocked when any applicable condition is present: +- the complete bounded file-path inventory; +- exact Git object identifiers; +- UTF-8 contents for selected analysis-critical configuration files; +- acquisition limits and skipped-file reasons. -- framework is unknown; -- output directory is unknown; -- dynamic server behavior is detected; -- a non-HTML framework has no known build command. +Repository Analysis input exposes the commit SHA, tree SHA, snapshot version, completeness flag, file-tree snapshot, and bounded file contents. The file index remains compatible with the documented keys: `path`, `name`, `file`, `filename`, `files`, `tree`, `items`, `children`, and `paths`. -This behavior is correct for the current incomplete input. The Dashboard must display blockers; it must not bypass them. +Quick Analysis still uses file paths for its current v0.1.0 detection rules. Update 2 will consume the acquired text evidence for stronger framework, monorepo, build-command, output-directory, environment-reference, and Cloudflare-readiness detection. -### Deep Analysis is currently a queue boundary - -The current Deep Analysis layer creates or enqueues a `repository_analysis_deep` job reference. +### Project creation attaches the initial Analysis -This audit does not establish evidence of a worker implementation that acquires repository contents, executes deep detection, stores a completed deep result, and updates the Project. +Project creation fetches repository metadata and the commit-pinned snapshot, runs Quick Analysis with the Project ID, and attaches the Repository and Analysis records to the Project. -### Recalculation does not reattach the new result to the Project +The Project becomes deploy-ready only when the stored Analysis reports `deploy_ready=true`. -The current recalculation path loads an existing Analysis and runs Quick Analysis again using the Repository ID. +### Deploy readiness remains fail-closed -It does not pass the previous Project ID and does not call the Project attachment boundary. A new Analysis result therefore does not automatically become the Project's active Analysis through this path. +Readiness remains blocked when applicable evidence is unknown or incompatible. Update 1 does not relax readiness rules. -## Current Blocking Chain +### Deep Analysis remains a queue boundary -```text -GitHub metadata without installation-token repository acquisition - -> no pinned commit SHA - -> no actual file tree - -> Quick Analysis receives insufficient paths - -> framework/output/build may remain unknown - -> deploy_ready=false - -> Deploy Engine and Dashboard correctly block deployment -``` +Deep Analysis is currently a queue boundary. -## Required Future Implementation Sequence +Deep Analysis still creates or enqueues a `repository_analysis_deep` job reference. This update does not implement the final deep-analysis worker execution path. -The future implementation should remain engine-owned and proceed in separate approved patches: +### Recalculation still does not reattach the new result to the Project -1. Resolve the user's GitHub App installation credential through Connected Accounts. -2. Acquire repository metadata using an installation token. -3. Resolve and persist the selected branch's commit SHA. -4. Acquire a normalized Git tree or equivalent bounded file-path snapshot. -5. Support private repositories through the GitHub App installation boundary. -6. Run Quick Analysis against the real, commit-pinned file tree. -7. Define and implement the deep-analysis worker execution contract. -8. Persist the completed Analysis result. -9. Reattach the latest approved Analysis to the Project through the Project Engine. -10. Re-evaluate Deploy Engine readiness and expose the result to the Dashboard. -11. Run controlled live GitHub and Cloudflare deployment validation. +The recalculation path still runs Quick Analysis using the Repository ID without reattaching the new result through Project Engine. This remains separate approved work. -## Security and Ownership Requirements +## Remaining Implementation Sequence -Future repository acquisition must: - -- use the user's GitHub App installation authorization; -- never expose installation tokens to the Dashboard or job payload; -- keep provider calls inside the Provider Layer; -- keep Repository and Analysis business logic inside their engines; -- pin analysis to a commit SHA; -- bound tree size and API pagination; -- fail closed on missing or inconsistent repository data; -- preserve repository ownership and user-access checks; -- avoid logging credentials or private source contents. - -## Explicit Non-Goals of This Patch - -This audit patch does not: - -- change Repository Analysis source; -- call GitHub or Cloudflare; -- clone a repository; -- acquire a Git tree; -- implement a deep-analysis worker; -- update a Project's active Analysis; -- change Deploy readiness rules; -- create a database migration; -- change Project Open or Deploy UI; -- change provider execution policy; -- integrate AG-001. +1. Strengthen detection using the acquired text evidence. +2. Add monorepo/root-directory resolution. +3. Resolve actual package scripts, install command, build command, and output directory. +4. Detect required environment-variable references without storing values. +5. Produce explicit Cloudflare readiness states and blocking reasons. +6. Implement deep-analysis worker completion. +7. Reattach the latest approved Analysis through Project Engine. +8. Run controlled live private-repository validation. +9. Run controlled live Cloudflare Pages deployment validation. -## Verification Boundary +## Security and Ownership Requirements -The audit contract tests verify that this document matches the current source at the audited commit. +- Connected Accounts owns installation references and ephemeral token acquisition. +- GitHub Provider owns GitHub HTTP calls only. +- Repository Engine owns normalized metadata and snapshots. +- Repository Analysis Engine owns detection and readiness decisions. +- Raw GitHub tokens must never be persisted or returned. +- Private source contents must never be logged. +- Cloudflare provider execution remains disabled. -Passing tests do not prove: +## Explicit Non-Goals of Update 1 -- live private-repository access; -- installation-token repository acquisition; -- real Git tree retrieval; +- broader framework detection; - deep-analysis worker completion; - Project reattachment after recalculation; -- live PostgreSQL or Redis behavior; -- a real Cloudflare Pages deployment. +- Cloudflare readiness expansion; +- Cloudflare provider execution; +- database migration; +- UI-V2 changes. + +## Verification Boundary + +Passing tests prove the deterministic provider, engine, snapshot, analyzer-input, and security contracts under mocks and local fixtures. They do not by themselves prove live GitHub or Cloudflare API behavior. ## Revision History | Date | Version | Summary | |---|---|---| -| 2026-07-22 | 1.0 | Recorded the current metadata, Quick Analysis, readiness, deep-queue, recalculation, and Project-attachment boundaries | +| 2026-07-25 | 1.1 | Implemented installation-authenticated metadata, commit pinning, complete tree snapshot, bounded text acquisition, and analyzer-input handoff | +| 2026-07-22 | 1.0 | Recorded the metadata-only gap and remaining analysis boundaries | diff --git a/VERSION.json b/VERSION.json index 63812de..36665a4 100644 --- a/VERSION.json +++ b/VERSION.json @@ -216,7 +216,10 @@ "provider_status": { "github": { "live_api_verified": false, - "repository_metadata_foundation": true + "repository_metadata_foundation": true, + "repository_acquisition": "installation_authenticated_commit_pinned", + "recursive_tree_snapshot": true, + "analysis_text_input": true }, "cloudflare": { "account_oauth_foundation": true, @@ -361,5 +364,18 @@ "compose_path": "/docker-compose.yml", "provider_mode_default": "disabled", "migration_head": "0012_notification_engine" + }, + "updates": { + "github_repository_acquisition_analyzer_input": { + "version": "1.0.0", + "status": "implemented_for_verification", + "baseline_commit": "d2ad741c0ee48bf3eb3887a3357f6f450825c99a", + "github_app_installation_credential": "ephemeral", + "commit_pinned_snapshot": true, + "recursive_tree_required_complete": true, + "analysis_critical_text_files_bounded": true, + "database_migration": false, + "cloudflare_provider_execution_changed": false + } } } diff --git a/backend/engines/auth_engine/connected_accounts_module/README.md b/backend/engines/auth_engine/connected_accounts_module/README.md index 8efc15f..b26b7e7 100644 --- a/backend/engines/auth_engine/connected_accounts_module/README.md +++ b/backend/engines/auth_engine/connected_accounts_module/README.md @@ -32,6 +32,7 @@ connected_accounts_service.connect_provider(...) connected_accounts_service.handle_provider_callback(...) connected_accounts_service.disconnect_provider(...) connected_accounts_service.require_provider_connected(...) +connected_accounts_service.acquire_github_repository_credential(...) connected_accounts_service.check_provider_health(...) connected_accounts_service.mark_provider_error(...) ``` @@ -52,3 +53,5 @@ DELETE /api/v1/connected-accounts/{provider} - Admin-visible responses must only expose provider state and safe metadata. - Token storage uses safe token references in this module baseline. - Provider modules remain external API wrappers only. + +- Installation access tokens are generated ephemerally for approved engine calls and are never persisted. diff --git a/backend/engines/auth_engine/connected_accounts_module/internal/service.py b/backend/engines/auth_engine/connected_accounts_module/internal/service.py index 843a237..62b3f0b 100644 --- a/backend/engines/auth_engine/connected_accounts_module/internal/service.py +++ b/backend/engines/auth_engine/connected_accounts_module/internal/service.py @@ -1,8 +1,7 @@ from __future__ import annotations import hmac - -from datetime import datetime, timezone +from datetime import UTC, datetime from sqlalchemy.ext.asyncio import AsyncSession @@ -10,10 +9,10 @@ from backend.engines.auth_engine.connected_accounts_module.errors import ( ConnectedAccountNotFoundError, ProviderConnectionFailedError, + ProviderCredentialExpiredError, ProviderNotConnectedError, ProviderNotSupportedError, ProviderOAuthFailedError, - ProviderCredentialExpiredError, ProviderTokenInvalidError, ) from backend.engines.auth_engine.connected_accounts_module.internal.credential_vault import ( @@ -24,17 +23,20 @@ ConnectedAccountOAuthState, TokenReferenceFactory, ) -from backend.engines.auth_engine.connected_accounts_module.repository import ConnectedAccountRepository +from backend.engines.auth_engine.connected_accounts_module.repository import ( + ConnectedAccountRepository, +) from backend.engines.auth_engine.connected_accounts_module.schemas import ( - ConnectProviderResult, ConnectedAccountRecord, + ConnectedAccountsList, ConnectedAccountStatus, ConnectedAccountSummary, - ConnectedAccountsList, + ConnectProviderResult, DisconnectProviderResult, ProviderCallbackResult, ProviderConnectionHealth, ProviderName, + ResolvedGitHubInstallationCredential, ResolvedProviderCredential, ) from backend.providers.cloudflare_provider.client import CloudflareProviderClient @@ -44,6 +46,7 @@ CloudflareProviderUnavailableError, ) from backend.providers.github_provider.client import GitHubProviderClient +from backend.providers.github_provider.errors import GitHubProviderError SUPPORTED_PROVIDERS: tuple[ProviderName, ...] = ("github", "cloudflare") @@ -88,12 +91,20 @@ def parse_provider(self, provider: str) -> ProviderName: raise ProviderNotSupportedError() return normalized # type: ignore[return-value] - async def list_connected_accounts(self, db: AsyncSession, *, user_id: str) -> ConnectedAccountsList: - records = [self.repository.to_record(model) for model in await self.repository.list_by_user(db, user_id=user_id)] + async def list_connected_accounts( + self, + db: AsyncSession, + *, + user_id: str, + ) -> ConnectedAccountsList: + models = await self.repository.list_by_user(db, user_id=user_id) + records = [self.repository.to_record(model) for model in models] by_provider = {record.provider: record for record in records} - return ConnectedAccountsList( - accounts=[self.to_summary(by_provider.get(provider), provider=provider) for provider in SUPPORTED_PROVIDERS] - ) + accounts = [ + self.to_summary(by_provider.get(provider), provider=provider) + for provider in SUPPORTED_PROVIDERS + ] + return ConnectedAccountsList(accounts=accounts) async def start_provider_connect( self, @@ -110,7 +121,11 @@ async def start_provider_connect( install_url=str(settings.github_app_install_url), state=state, ) - return ConnectProviderResult(provider=provider_name, authorization_url=authorization_url, state=state) + return ConnectProviderResult( + provider=provider_name, + authorization_url=authorization_url, + state=state, + ) state = ConnectedAccountOAuthState.new_state(user_id=user_id, provider=provider_name) if settings.cloudflare_oauth_client_id: @@ -122,8 +137,16 @@ async def start_provider_connect( ) else: redirect_base = str(settings.app_base_url).rstrip("/") - authorization_url = f"{redirect_base}{settings.api_prefix}/connected-accounts/{provider_name}/callback?state={state}" - return ConnectProviderResult(provider=provider_name, authorization_url=authorization_url, state=state) + callback_path = ( + f"{settings.api_prefix}/connected-accounts/" + f"{provider_name}/callback?state={state}" + ) + authorization_url = f"{redirect_base}{callback_path}" + return ConnectProviderResult( + provider=provider_name, + authorization_url=authorization_url, + state=state, + ) async def handle_provider_callback( self, @@ -345,7 +368,11 @@ async def disconnect_provider( provider: str, ) -> DisconnectProviderResult: provider_name = self.parse_provider(provider) - existing = await self.repository.get_by_user_provider(db, user_id=user_id, provider=provider_name) + existing = await self.repository.get_by_user_provider( + db, + user_id=user_id, + provider=provider_name, + ) if existing is None: raise ConnectedAccountNotFoundError() @@ -353,9 +380,17 @@ async def disconnect_provider( if provider_name == "github": await self._delete_github_app_installation(existing_record.token_secret_ref) - record = await self.repository.mark_disconnected(db, user_id=user_id, provider=provider_name) + record = await self.repository.mark_disconnected( + db, + user_id=user_id, + provider=provider_name, + ) await db.commit() - return DisconnectProviderResult(provider=record.provider, connected=False, status=record.status) + return DisconnectProviderResult( + provider=record.provider, + connected=False, + status=record.status, + ) async def require_provider_connected( self, @@ -365,7 +400,11 @@ async def require_provider_connected( provider: str, ) -> ConnectedAccountRecord: provider_name = self.parse_provider(provider) - model = await self.repository.get_by_user_provider(db, user_id=user_id, provider=provider_name) + model = await self.repository.get_by_user_provider( + db, + user_id=user_id, + provider=provider_name, + ) if model is None: raise ConnectedAccountNotFoundError() record = self.repository.to_record(model) @@ -373,6 +412,61 @@ async def require_provider_connected( raise ProviderNotConnectedError() return record + async def acquire_github_repository_credential( + self, + db: AsyncSession, + *, + user_id: str, + ) -> ResolvedGitHubInstallationCredential: + record = await self.require_provider_connected( + db, + user_id=user_id, + provider="github", + ) + token_reference = (record.token_secret_ref or "").strip() + prefix = "github_app_installation:" + installation_value = token_reference.removeprefix(prefix).strip() + + if ( + not token_reference.startswith(prefix) + or not installation_value.isdigit() + or not record.provider_account_id + or not hmac.compare_digest( + record.provider_account_id.strip(), + installation_value, + ) + ): + raise ProviderTokenInvalidError() + + settings = get_settings() + try: + app_jwt = self.github_provider.create_app_jwt( + app_id=settings.github_app_id, + private_key_pem=settings.github_app_private_key.get_secret_value(), + ) + credential = await self.github_provider.create_installation_access_token( + installation_id=installation_value, + app_jwt=app_jwt, + ) + except GitHubProviderError as exc: + raise ProviderConnectionFailedError( + "GitHub repository credential acquisition failed." + ) from exc + + permissions = { + str(name): str(access) + for name, access in (credential.permissions or {}).items() + if name is not None and access is not None + } + return ResolvedGitHubInstallationCredential( + token_secret_ref=token_reference, + installation_id=int(installation_value), + access_token=credential.token, + expires_at=credential.expires_at, + permissions=permissions, + repository_selection=credential.repository_selection, + ) + async def resolve_cloudflare_credential( self, db: AsyncSession, @@ -680,7 +774,7 @@ async def check_provider_health( provider=record.provider, status="connected", healthy=True, - checked_at=datetime.now(timezone.utc), + checked_at=datetime.now(UTC), ) async def mark_provider_error( @@ -693,7 +787,12 @@ async def mark_provider_error( ) -> None: provider_name = self.parse_provider(provider) error_code = error_summary.strip().upper().replace(" ", "_")[:128] or "PROVIDER_ERROR" - await self.repository.mark_provider_error(db, user_id=user_id, provider=provider_name, error_code=error_code) + await self.repository.mark_provider_error( + db, + user_id=user_id, + provider=provider_name, + error_code=error_code, + ) await db.commit() async def _delete_github_app_installation(self, token_secret_ref: str | None) -> None: @@ -722,7 +821,11 @@ async def _delete_github_app_installation(self, token_secret_ref: str | None) -> except Exception as exc: raise ProviderConnectionFailedError() from exc - async def _validate_provider_account(self, provider: ProviderName, token_ref: str) -> dict[str, str]: + async def _validate_provider_account( + self, + provider: ProviderName, + token_ref: str, + ) -> dict[str, str]: if provider == "github": return await self.github_provider.validate_account(token_ref) return await self.cloudflare_provider.validate_account(token_ref) diff --git a/backend/engines/auth_engine/connected_accounts_module/public.py b/backend/engines/auth_engine/connected_accounts_module/public.py index adfa60b..cdbb259 100644 --- a/backend/engines/auth_engine/connected_accounts_module/public.py +++ b/backend/engines/auth_engine/connected_accounts_module/public.py @@ -2,14 +2,17 @@ from sqlalchemy.ext.asyncio import AsyncSession -from backend.engines.auth_engine.connected_accounts_module.internal.service import ConnectedAccountsInternalService +from backend.engines.auth_engine.connected_accounts_module.internal.service import ( + ConnectedAccountsInternalService, +) from backend.engines.auth_engine.connected_accounts_module.schemas import ( - ConnectProviderResult, ConnectedAccountRecord, ConnectedAccountsList, + ConnectProviderResult, DisconnectProviderResult, ProviderCallbackResult, ProviderConnectionHealth, + ResolvedGitHubInstallationCredential, ResolvedProviderCredential, ) @@ -24,7 +27,12 @@ class ConnectedAccountsPublicService: def __init__(self, internal: ConnectedAccountsInternalService | None = None) -> None: self._internal = internal or ConnectedAccountsInternalService() - async def get_connected_accounts(self, db: AsyncSession, *, user_id: str) -> ConnectedAccountsList: + async def get_connected_accounts( + self, + db: AsyncSession, + *, + user_id: str, + ) -> ConnectedAccountsList: return await self._internal.list_connected_accounts(db, user_id=user_id) async def connect_provider(self, *, user_id: str, provider: str) -> ConnectProviderResult: @@ -69,7 +77,22 @@ async def require_provider_connected( user_id: str, provider: str, ) -> ConnectedAccountRecord: - return await self._internal.require_provider_connected(db, user_id=user_id, provider=provider) + return await self._internal.require_provider_connected( + db, + user_id=user_id, + provider=provider, + ) + + async def acquire_github_repository_credential( + self, + db: AsyncSession, + *, + user_id: str, + ) -> ResolvedGitHubInstallationCredential: + return await self._internal.acquire_github_repository_credential( + db, + user_id=user_id, + ) async def resolve_cloudflare_credential( self, diff --git a/backend/engines/auth_engine/connected_accounts_module/schemas.py b/backend/engines/auth_engine/connected_accounts_module/schemas.py index 79c136d..5e49144 100644 --- a/backend/engines/auth_engine/connected_accounts_module/schemas.py +++ b/backend/engines/auth_engine/connected_accounts_module/schemas.py @@ -33,11 +33,21 @@ class ResolvedProviderCredential(BaseModel): token_secret_ref: str access_token: SecretStr refresh_token: SecretStr | None = None - token_type: str = "bearer" + token_type: str = Field(default="bearer") expires_at: datetime | None = None scopes: list[str] = Field(default_factory=list) +class ResolvedGitHubInstallationCredential(BaseModel): + provider: Literal["github"] = "github" + token_secret_ref: str + installation_id: int + access_token: SecretStr + expires_at: str + permissions: dict[str, str] = Field(default_factory=dict) + repository_selection: str | None = None + + class ConnectedAccountSummary(BaseModel): provider: ProviderName connected: bool diff --git a/backend/engines/repository_analysis_engine/README.md b/backend/engines/repository_analysis_engine/README.md index e29c961..d39ad1f 100644 --- a/backend/engines/repository_analysis_engine/README.md +++ b/backend/engines/repository_analysis_engine/README.md @@ -50,3 +50,8 @@ backend.engines.repository_analysis_engine.internal.* backend.engines.repository_analysis_engine.repository backend.engines.repository_analysis_engine.models ``` + + +## Analyzer Input Contract + +Quick and deep analysis receive a commit-pinned, complete Repository Engine snapshot containing file paths and bounded configuration-file contents. The analysis engine does not call GitHub directly and rejects incomplete snapshots through the Repository Engine public boundary. diff --git a/backend/engines/repository_engine/README.md b/backend/engines/repository_engine/README.md index 331140a..e545158 100644 --- a/backend/engines/repository_engine/README.md +++ b/backend/engines/repository_engine/README.md @@ -2,7 +2,7 @@ ## Responsibility -Repository Engine owns GitHub repository URL validation, repository metadata normalization, default branch, visibility, and repository metadata snapshots. +Repository Engine owns GitHub repository URL validation, authenticated repository metadata acquisition, exact commit pinning, normalized file-tree snapshots, default branch, visibility, and analyzer input preparation. ## Contract Version @@ -37,4 +37,4 @@ repository_metadata ## Boundary Rules -Repository Engine may call GitHub Provider. It must not call Cloudflare Provider, Deploy Pipeline, Deploy Engine mutation services, or Repository Analysis internal services. +Repository Engine may call the Connected Accounts public API for an ephemeral GitHub installation credential and may call GitHub Provider. It must not persist raw credentials, call Cloudflare Provider, call Deploy Pipeline, call Deploy Engine mutation services, or import Repository Analysis internals. diff --git a/backend/engines/repository_engine/errors.py b/backend/engines/repository_engine/errors.py index 746ae26..f8c7a4d 100644 --- a/backend/engines/repository_engine/errors.py +++ b/backend/engines/repository_engine/errors.py @@ -59,3 +59,11 @@ def __init__(self) -> None: message="Repository default branch is missing.", status_code=422, ) + +class RepositorySnapshotIncompleteError(RepositoryError): + def __init__(self) -> None: + super().__init__( + code="REPOSITORY_SNAPSHOT_INCOMPLETE", + message="Repository snapshot could not be acquired completely.", + status_code=422, + ) diff --git a/backend/engines/repository_engine/internal/service.py b/backend/engines/repository_engine/internal/service.py index ebc47da..6f4a06b 100644 --- a/backend/engines/repository_engine/internal/service.py +++ b/backend/engines/repository_engine/internal/service.py @@ -2,11 +2,16 @@ from sqlalchemy.ext.asyncio import AsyncSession +from backend.engines.auth_engine.connected_accounts_module.public import ( + ConnectedAccountsPublicService, + connected_accounts_service, +) from backend.engines.repository_engine.errors import ( RepositoryAccessDeniedError, RepositoryDefaultBranchMissingError, RepositoryMetadataFetchFailedError, RepositoryNotFoundError, + RepositorySnapshotIncompleteError, ) from backend.engines.repository_engine.internal.validators import parse_github_repository_url from backend.engines.repository_engine.repository import RepositoryMetadataRepository @@ -22,7 +27,11 @@ RepositoryValidationResult, ) from backend.providers.github_provider.client import GitHubProviderClient -from backend.providers.github_provider.errors import GitHubProviderError, GitHubRepositoryNotFoundError +from backend.providers.github_provider.errors import ( + GitHubProviderError, + GitHubRepositoryNotFoundError, + GitHubRepositoryTreeTruncatedError, +) class RepositoryInternalService: @@ -30,14 +39,21 @@ def __init__( self, repository: RepositoryMetadataRepository | None = None, github_provider: GitHubProviderClient | None = None, + connected_accounts_public_service: ConnectedAccountsPublicService | None = None, ) -> None: self.repository = repository or RepositoryMetadataRepository() self.github_provider = github_provider or GitHubProviderClient() + self.connected_accounts_service = ( + connected_accounts_public_service or connected_accounts_service + ) async def parse_repository_url(self, repository_url: str) -> ParsedRepositoryUrl: return parse_github_repository_url(repository_url) - async def validate_repository_url(self, input_data: RepositoryValidateInput) -> RepositoryValidationResult: + async def validate_repository_url( + self, + input_data: RepositoryValidateInput, + ) -> RepositoryValidationResult: parsed = await self.parse_repository_url(input_data.repository_url) return RepositoryValidationResult( valid=True, @@ -55,13 +71,46 @@ async def fetch_repository_metadata( input_data: RepositoryMetadataInput, ) -> RepositoryDetail: parsed = await self.parse_repository_url(input_data.repository_url) + credential = ( + await self.connected_accounts_service.acquire_github_repository_credential( + db, + user_id=user_id, + ) + ) + access_value = credential.access_token.get_secret_value() + try: - provider_result = await self.github_provider.get_repository_metadata(owner=parsed.owner, repo=parsed.repo) + provider_result = await self.github_provider.get_repository_metadata( + owner=parsed.owner, + repo=parsed.repo, + installation_credential=access_value, + ) + if provider_result.default_branch is None: + raise RepositoryDefaultBranchMissingError() + snapshot = await self.github_provider.get_repository_snapshot( + owner=provider_result.owner, + repo=provider_result.name, + branch=provider_result.default_branch, + installation_credential=access_value, + ) except GitHubRepositoryNotFoundError as exc: raise RepositoryNotFoundError() from exc + except GitHubRepositoryTreeTruncatedError as exc: + raise RepositorySnapshotIncompleteError() from exc + except RepositoryDefaultBranchMissingError: + raise except GitHubProviderError as exc: raise RepositoryMetadataFetchFailedError() from exc + metadata = dict(provider_result.metadata or {}) + metadata["snapshot"] = { + "version": snapshot.version, + "branch": snapshot.branch, + "commit_sha": snapshot.commit_sha, + "tree_sha": snapshot.tree_sha, + "file_count": len(snapshot.files), + "text_file_count": len(snapshot.text_files), + } provider_metadata = RepositoryProviderMetadata( provider="github", repository_url=parsed.normalized_url, @@ -69,17 +118,25 @@ async def fetch_repository_metadata( name=provider_result.name, default_branch=provider_result.default_branch, visibility=provider_result.visibility, - latest_commit_sha=provider_result.latest_commit_sha, - file_tree_snapshot=provider_result.file_tree_snapshot, - metadata=provider_result.metadata, + latest_commit_sha=snapshot.commit_sha, + file_tree_snapshot=snapshot.model_dump(mode="json"), + metadata=metadata, + ) + record = await self.repository.upsert_metadata( + db, + user_id=user_id, + provider_metadata=provider_metadata, ) - if provider_metadata.default_branch is None: - raise RepositoryDefaultBranchMissingError() - record = await self.repository.upsert_metadata(db, user_id=user_id, provider_metadata=provider_metadata) await db.commit() return self.to_detail(record) - async def get_repository_metadata(self, db: AsyncSession, *, user_id: str, repository_id: str) -> RepositoryDetail: + async def get_repository_metadata( + self, + db: AsyncSession, + *, + user_id: str, + repository_id: str, + ) -> RepositoryDetail: model = await self.repository.get_active_by_id(db, repository_id) if model is None: raise RepositoryNotFoundError() @@ -87,8 +144,37 @@ async def get_repository_metadata(self, db: AsyncSession, *, user_id: str, repos raise RepositoryAccessDeniedError() return self.to_detail(self.repository.to_record(model)) - async def prepare_analysis_input(self, db: AsyncSession, *, user_id: str, repository_id: str) -> RepositoryAnalysisInput: - detail = await self.get_repository_metadata(db, user_id=user_id, repository_id=repository_id) + async def prepare_analysis_input( + self, + db: AsyncSession, + *, + user_id: str, + repository_id: str, + ) -> RepositoryAnalysisInput: + detail = await self.get_repository_metadata( + db, + user_id=user_id, + repository_id=repository_id, + ) + snapshot = detail.file_tree_snapshot or {} + snapshot_commit = str(snapshot.get("commit_sha") or "") + truncated = bool(snapshot.get("truncated")) + if ( + not detail.latest_commit_sha + or snapshot_commit != detail.latest_commit_sha + or truncated + or not isinstance(snapshot.get("files"), list) + ): + raise RepositorySnapshotIncompleteError() + + raw_text_files = snapshot.get("text_files") + file_contents = { + str(path): str(content) + for path, content in ( + raw_text_files.items() if isinstance(raw_text_files, dict) else [] + ) + if path and isinstance(content, str) + } return RepositoryAnalysisInput( repository_id=detail.id, provider=detail.provider, @@ -98,7 +184,11 @@ async def prepare_analysis_input(self, db: AsyncSession, *, user_id: str, reposi default_branch=detail.default_branch, visibility=detail.visibility, latest_commit_sha=detail.latest_commit_sha, - file_tree_snapshot=detail.file_tree_snapshot, + tree_sha=str(snapshot.get("tree_sha") or "") or None, + snapshot_version=int(snapshot.get("version") or 1), + snapshot_complete=True, + file_tree_snapshot=snapshot, + file_contents=file_contents, ) def to_summary(self, record: RepositoryRecord) -> RepositorySummary: diff --git a/backend/engines/repository_engine/schemas.py b/backend/engines/repository_engine/schemas.py index a6479fd..453f884 100644 --- a/backend/engines/repository_engine/schemas.py +++ b/backend/engines/repository_engine/schemas.py @@ -97,4 +97,8 @@ class RepositoryAnalysisInput(BaseModel): default_branch: str | None = None visibility: RepositoryVisibility latest_commit_sha: str | None = None + tree_sha: str | None = None + snapshot_version: int = 1 + snapshot_complete: bool = False file_tree_snapshot: dict[str, Any] | None = None + file_contents: dict[str, str] = Field(default_factory=dict) diff --git a/backend/providers/github_provider/client.py b/backend/providers/github_provider/client.py index 5016ff6..5448427 100644 --- a/backend/providers/github_provider/client.py +++ b/backend/providers/github_provider/client.py @@ -1,8 +1,10 @@ from __future__ import annotations +import base64 +import binascii import time from typing import Any -from urllib.parse import urlencode +from urllib.parse import quote, urlencode import httpx import jwt @@ -12,12 +14,53 @@ GitHubAppConfigurationError, GitHubInstallationValidationError, GitHubProviderUnavailableError, + GitHubRepositoryContentError, GitHubRepositoryNotFoundError, + GitHubRepositoryTreeTruncatedError, ) from backend.providers.github_provider.schemas import ( GitHubAppInstallation, GitHubInstallationAccessToken, GitHubRepository, + GitHubRepositorySnapshot, + GitHubRepositoryTreeEntry, +) + +_ANALYSIS_TEXT_FILE_NAMES = frozenset( + { + ".nvmrc", + "_config.yml", + "angular.json", + "astro.config.js", + "astro.config.mjs", + "astro.config.ts", + "bun.lock", + "bun.lockb", + "cargo.toml", + "docusaurus.config.js", + "docusaurus.config.ts", + "gatsby-config.js", + "gatsby-config.ts", + "go.mod", + "hugo.toml", + "index.html", + "next.config.js", + "next.config.mjs", + "next.config.ts", + "nuxt.config.js", + "nuxt.config.ts", + "package.json", + "pnpm-workspace.yaml", + "svelte.config.js", + "svelte.config.ts", + "tsconfig.json", + "turbo.json", + "vite.config.js", + "vite.config.mjs", + "vite.config.ts", + "vue.config.js", + "webpack.config.js", + } ) @@ -31,9 +74,17 @@ def __init__( *, timeout_seconds: float = 10.0, api_base_url: str = "https://api.github.com", + max_tree_entries: int = 25_000, + max_text_files: int = 128, + max_text_file_bytes: int = 256 * 1024, + max_total_text_bytes: int = 2 * 1024 * 1024, ) -> None: self.timeout_seconds = timeout_seconds self.api_base_url = api_base_url.rstrip("/") + self.max_tree_entries = max_tree_entries + self.max_text_files = max_text_files + self.max_text_file_bytes = max_text_file_bytes + self.max_total_text_bytes = max_total_text_bytes def build_app_installation_url(self, *, install_url: str, state: str) -> str: if not install_url: @@ -51,11 +102,7 @@ def create_app_jwt(self, *, app_id: str, private_key_pem: str) -> str: issued_at = int(time.time()) - 60 expires_at = issued_at + 540 - payload = { - "iat": issued_at, - "exp": expires_at, - "iss": app_id.strip(), - } + payload = {"iat": issued_at, "exp": expires_at, "iss": app_id.strip()} try: token = jwt.encode(payload, private_key_pem, algorithm="RS256") @@ -75,21 +122,12 @@ async def get_app_installation( if not app_jwt: raise GitHubAppAuthenticationError("GitHub App JWT is missing.") - url = f"{self.api_base_url}/app/installations/{installation_id}" - headers = self._app_headers(app_jwt) - - try: - async with httpx.AsyncClient(timeout=self.timeout_seconds, headers=headers) as client: - response = await client.get(url) - except httpx.HTTPError as exc: - raise GitHubProviderUnavailableError("GitHub installation validation request failed.") from exc - - if response.status_code == 404: - raise GitHubInstallationValidationError("GitHub App installation was not found.") - if response.status_code >= 400: - raise GitHubInstallationValidationError("GitHub App installation validation failed.") - - payload: dict[str, Any] = response.json() + payload = await self._get_json( + f"{self.api_base_url}/app/installations/{installation_id}", + headers=self._app_headers(app_jwt), + not_found_message="GitHub App installation was not found.", + error_message="GitHub App installation validation failed.", + ) return self._installation_from_payload(payload) async def create_installation_access_token( @@ -102,23 +140,29 @@ async def create_installation_access_token( raise GitHubAppAuthenticationError("GitHub App JWT is missing.") url = f"{self.api_base_url}/app/installations/{installation_id}/access_tokens" - headers = self._app_headers(app_jwt) - try: - async with httpx.AsyncClient(timeout=self.timeout_seconds, headers=headers) as client: + async with httpx.AsyncClient( + timeout=self.timeout_seconds, + headers=self._app_headers(app_jwt), + ) as client: response = await client.post(url) except httpx.HTTPError as exc: - raise GitHubProviderUnavailableError("GitHub installation token request failed.") from exc + raise GitHubProviderUnavailableError( + "GitHub installation token request failed." + ) from exc if response.status_code >= 400: - raise GitHubInstallationValidationError("GitHub installation token request failed.") + raise GitHubInstallationValidationError( + "GitHub installation token request failed." + ) payload: dict[str, Any] = response.json() token = str(payload.get("token") or "") expires_at = str(payload.get("expires_at") or "") - if not token or not expires_at: - raise GitHubInstallationValidationError("GitHub installation token response was invalid.") + raise GitHubInstallationValidationError( + "GitHub installation token response was invalid." + ) return GitHubInstallationAccessToken( token=token, @@ -137,50 +181,53 @@ async def delete_app_installation( raise GitHubAppAuthenticationError("GitHub App JWT is missing.") url = f"{self.api_base_url}/app/installations/{installation_id}" - headers = self._app_headers(app_jwt) - try: - async with httpx.AsyncClient(timeout=self.timeout_seconds, headers=headers) as client: + async with httpx.AsyncClient( + timeout=self.timeout_seconds, + headers=self._app_headers(app_jwt), + ) as client: response = await client.delete(url) except httpx.HTTPError as exc: - raise GitHubProviderUnavailableError("GitHub App installation uninstall request failed.") from exc + raise GitHubProviderUnavailableError( + "GitHub App installation uninstall request failed." + ) from exc if response.status_code in {204, 404}: return if response.status_code >= 400: - raise GitHubInstallationValidationError("GitHub App installation uninstall failed.") - + raise GitHubInstallationValidationError( + "GitHub App installation uninstall failed." + ) async def validate_account(self, token_ref: str) -> dict[str, str]: if not token_ref: return {"provider": self.provider, "status": "invalid"} - return {"provider": self.provider, "status": "validated", "account_name": "github-account"} + return { + "provider": self.provider, + "status": "validated", + "account_name": "github-account", + } async def get_repository_metadata( self, *, owner: str, repo: str, - token_ref: str | None = None, + installation_credential: str | None = None, ) -> GitHubRepository: - headers = {"Accept": "application/vnd.github+json", "User-Agent": "YGIT"} - # token_ref is intentionally not dereferenced by the provider. Token resolution belongs to Connected Accounts. - url = f"{self.api_base_url}/repos/{owner}/{repo}" - try: - async with httpx.AsyncClient(timeout=self.timeout_seconds, headers=headers) as client: - response = await client.get(url) - except httpx.HTTPError as exc: - raise GitHubProviderUnavailableError("GitHub API request failed.") from exc - - if response.status_code == 404: - raise GitHubRepositoryNotFoundError("GitHub repository was not found.") - if response.status_code >= 400: - raise GitHubProviderUnavailableError("GitHub API returned an error.") + headers = self._installation_headers(installation_credential) + payload = await self._get_json( + f"{self.api_base_url}/repos/{owner}/{repo}", + headers=headers, + not_found_message="GitHub repository was not found.", + error_message="GitHub repository metadata request failed.", + ) - payload: dict[str, Any] = response.json() owner_login = str(payload.get("owner", {}).get("login") or owner) name = str(payload.get("name") or repo) - visibility = "private" if payload.get("private") else "public" + visibility = str(payload.get("visibility") or "") + if visibility not in {"public", "private", "internal"}: + visibility = "private" if payload.get("private") else "public" default_branch = payload.get("default_branch") pushed_at = payload.get("pushed_at") return GitHubRepository( @@ -190,18 +237,216 @@ async def get_repository_metadata( default_branch=default_branch, visibility=visibility, latest_commit_sha=None, - file_tree_snapshot={"default_branch": default_branch} if default_branch else None, + file_tree_snapshot=None, metadata={ "id": payload.get("id"), + "node_id": payload.get("node_id"), "full_name": payload.get("full_name"), "html_url": payload.get("html_url"), "description": payload.get("description"), "pushed_at": pushed_at, "archived": payload.get("archived"), "disabled": payload.get("disabled"), + "fork": payload.get("fork"), }, ) + async def get_repository_snapshot( + self, + *, + owner: str, + repo: str, + branch: str, + installation_credential: str, + ) -> GitHubRepositorySnapshot: + if not installation_credential.strip(): + raise GitHubAppAuthenticationError( + "GitHub installation access token is missing." + ) + headers = self._installation_headers(installation_credential) + branch_payload = await self._get_json( + f"{self.api_base_url}/repos/{owner}/{repo}/branches/{quote(branch, safe='')}", + headers=headers, + not_found_message="GitHub repository branch was not found.", + error_message="GitHub repository branch resolution failed.", + ) + commit_sha = str((branch_payload.get("commit") or {}).get("sha") or "") + if not commit_sha: + raise GitHubRepositoryContentError( + "GitHub repository branch response did not include a commit." + ) + + commit_payload = await self._get_json( + f"{self.api_base_url}/repos/{owner}/{repo}/git/commits/{commit_sha}", + headers=headers, + not_found_message="GitHub repository commit was not found.", + error_message="GitHub repository commit resolution failed.", + ) + tree_sha = str((commit_payload.get("tree") or {}).get("sha") or "") + if not tree_sha: + raise GitHubRepositoryContentError( + "GitHub repository commit response did not include a tree." + ) + + tree_payload = await self._get_json( + f"{self.api_base_url}/repos/{owner}/{repo}/git/trees/{tree_sha}", + headers=headers, + params={"recursive": "1"}, + not_found_message="GitHub repository tree was not found.", + error_message="GitHub repository tree acquisition failed.", + ) + if bool(tree_payload.get("truncated")): + raise GitHubRepositoryTreeTruncatedError( + "GitHub repository tree response was truncated." + ) + + raw_tree = tree_payload.get("tree") or [] + if not isinstance(raw_tree, list) or len(raw_tree) > self.max_tree_entries: + raise GitHubRepositoryTreeTruncatedError( + "GitHub repository tree exceeds the supported snapshot size." + ) + + entries: list[GitHubRepositoryTreeEntry] = [] + for raw_entry in raw_tree: + if not isinstance(raw_entry, dict): + continue + path = str(raw_entry.get("path") or "").strip().lstrip("/") + sha = str(raw_entry.get("sha") or "").strip() + raw_type = str(raw_entry.get("type") or "unknown") + entry_type = raw_type if raw_type in {"blob", "tree", "commit"} else "unknown" + if not path or not sha: + continue + raw_size = raw_entry.get("size") + entries.append( + GitHubRepositoryTreeEntry( + path=path, + mode=str(raw_entry.get("mode") or "") or None, + type=entry_type, + sha=sha, + size=int(raw_size) if isinstance(raw_size, int) else None, + ) + ) + + entries.sort(key=lambda item: item.path.lower()) + text_files, skipped_files = await self._fetch_analysis_text_files( + owner=owner, + repo=repo, + entries=entries, + headers=headers, + ) + return GitHubRepositorySnapshot( + branch=branch, + commit_sha=commit_sha, + tree_sha=tree_sha, + files=entries, + text_files=text_files, + metadata={ + "file_count": len(entries), + "text_file_count": len(text_files), + "skipped_text_files": skipped_files, + "max_tree_entries": self.max_tree_entries, + "max_text_files": self.max_text_files, + "max_text_file_bytes": self.max_text_file_bytes, + "max_total_text_bytes": self.max_total_text_bytes, + }, + ) + + async def _fetch_analysis_text_files( + self, + *, + owner: str, + repo: str, + entries: list[GitHubRepositoryTreeEntry], + headers: dict[str, str], + ) -> tuple[dict[str, str], list[dict[str, str]]]: + candidates = [ + entry + for entry in entries + if entry.type == "blob" and self._is_analysis_text_candidate(entry) + ][: self.max_text_files] + text_files: dict[str, str] = {} + skipped: list[dict[str, str]] = [] + total_bytes = 0 + + for entry in candidates: + if entry.size is not None and entry.size > self.max_text_file_bytes: + skipped.append({"path": entry.path, "reason": "file_too_large"}) + continue + payload = await self._get_json( + f"{self.api_base_url}/repos/{owner}/{repo}/git/blobs/{entry.sha}", + headers=headers, + not_found_message="GitHub repository blob was not found.", + error_message="GitHub repository blob acquisition failed.", + ) + if str(payload.get("encoding") or "").lower() != "base64": + skipped.append({"path": entry.path, "reason": "unsupported_encoding"}) + continue + encoded = str(payload.get("content") or "").replace("\n", "") + try: + content_bytes = base64.b64decode(encoded, validate=True) + content = content_bytes.decode("utf-8") + except (binascii.Error, UnicodeDecodeError): + skipped.append({"path": entry.path, "reason": "not_utf8_text"}) + continue + if len(content_bytes) > self.max_text_file_bytes: + skipped.append({"path": entry.path, "reason": "file_too_large"}) + continue + if total_bytes + len(content_bytes) > self.max_total_text_bytes: + skipped.append({"path": entry.path, "reason": "total_text_limit"}) + continue + text_files[entry.path] = content + total_bytes += len(content_bytes) + + return text_files, skipped + + def _is_analysis_text_candidate(self, entry: GitHubRepositoryTreeEntry) -> bool: + path = entry.path.lower() + basename = path.rsplit("/", 1)[-1] + sensitive_markers = ( + ".env", + "credential", + "secret", + "id_rsa", + ".pem", + ".key", + ".p12", + ".pfx", + ".jks", + ) + if any(marker in basename for marker in sensitive_markers): + return False + return basename in _ANALYSIS_TEXT_FILE_NAMES + + async def _get_json( + self, + url: str, + *, + headers: dict[str, str], + not_found_message: str, + error_message: str, + params: dict[str, str] | None = None, + ) -> dict[str, Any]: + try: + async with httpx.AsyncClient( + timeout=self.timeout_seconds, + headers=headers, + ) as client: + response = await client.get(url, params=params) + except httpx.HTTPError as exc: + raise GitHubProviderUnavailableError(error_message) from exc + + if response.status_code == 404: + raise GitHubRepositoryNotFoundError(not_found_message) + if response.status_code >= 400: + raise GitHubProviderUnavailableError(error_message) + try: + payload = response.json() + except ValueError as exc: + raise GitHubProviderUnavailableError(error_message) from exc + if not isinstance(payload, dict): + raise GitHubProviderUnavailableError(error_message) + return payload + def _app_headers(self, app_jwt: str) -> dict[str, str]: return { "Accept": "application/vnd.github+json", @@ -210,17 +455,30 @@ def _app_headers(self, app_jwt: str) -> dict[str, str]: "X-GitHub-Api-Version": "2022-11-28", } + def _installation_headers(self, access_token: str | None) -> dict[str, str]: + headers = { + "Accept": "application/vnd.github+json", + "User-Agent": "YGIT", + "X-GitHub-Api-Version": "2022-11-28", + } + if access_token: + headers["Authorization"] = f"Bearer {access_token}" + return headers + def _installation_from_payload(self, payload: dict[str, Any]) -> GitHubAppInstallation: raw_id = payload.get("id") if raw_id is None: - raise GitHubInstallationValidationError("GitHub installation payload is missing id.") + raise GitHubInstallationValidationError( + "GitHub installation payload is missing id." + ) account = payload.get("account") or {} account_login = str(account.get("login") or account.get("slug") or "") account_id = str(account.get("id") or "") - if not account_login or not account_id: - raise GitHubInstallationValidationError("GitHub installation payload is missing account metadata.") + raise GitHubInstallationValidationError( + "GitHub installation payload is missing account metadata." + ) return GitHubAppInstallation( installation_id=int(raw_id), @@ -231,12 +489,7 @@ def _installation_from_payload(self, payload: dict[str, Any]) -> GitHubAppInstal repository_selection=payload.get("repository_selection"), permissions={ str(name): str(access) - for name, access in ( - payload.get("permissions") or {} - ).items() - if ( - name is not None - and access is not None - ) + for name, access in (payload.get("permissions") or {}).items() + if name is not None and access is not None }, ) diff --git a/backend/providers/github_provider/errors.py b/backend/providers/github_provider/errors.py index 6c49106..e5ddeda 100644 --- a/backend/providers/github_provider/errors.py +++ b/backend/providers/github_provider/errors.py @@ -23,3 +23,14 @@ class GitHubAppAuthenticationError(GitHubProviderError): class GitHubInstallationValidationError(GitHubProviderError): pass + +class GitHubRepositorySnapshotError(GitHubProviderError): + pass + + +class GitHubRepositoryTreeTruncatedError(GitHubRepositorySnapshotError): + pass + + +class GitHubRepositoryContentError(GitHubRepositorySnapshotError): + pass diff --git a/backend/providers/github_provider/schemas.py b/backend/providers/github_provider/schemas.py index 0697d23..6cb8c55 100644 --- a/backend/providers/github_provider/schemas.py +++ b/backend/providers/github_provider/schemas.py @@ -17,6 +17,26 @@ class GitHubRepository(BaseModel): metadata: dict[str, Any] | None = None +class GitHubRepositoryTreeEntry(BaseModel): + path: str + mode: str | None = None + type: Literal["blob", "tree", "commit", "unknown"] = "unknown" + sha: str + size: int | None = None + + +class GitHubRepositorySnapshot(BaseModel): + version: Literal[1] = 1 + provider: Literal["github"] = "github" + branch: str + commit_sha: str + tree_sha: str + truncated: bool = False + files: list[GitHubRepositoryTreeEntry] = Field(default_factory=list) + text_files: dict[str, str] = Field(default_factory=dict) + metadata: dict[str, Any] = Field(default_factory=dict) + + class ProviderResponse(BaseModel): provider: str ok: bool diff --git a/backend/tests/test_github_repository_acquisition_analyzer_input.py b/backend/tests/test_github_repository_acquisition_analyzer_input.py new file mode 100644 index 0000000..afce5a0 --- /dev/null +++ b/backend/tests/test_github_repository_acquisition_analyzer_input.py @@ -0,0 +1,420 @@ +# ruff: noqa: S105, S106 +from __future__ import annotations + +from datetime import UTC, datetime +from types import SimpleNamespace +from typing import Any + +import pytest +from pydantic import SecretStr + +from backend.engines.auth_engine.connected_accounts_module.internal.service import ( + ConnectedAccountsInternalService, +) +from backend.engines.auth_engine.connected_accounts_module.schemas import ( + ConnectedAccountRecord, + ResolvedGitHubInstallationCredential, +) +from backend.engines.repository_engine.errors import RepositorySnapshotIncompleteError +from backend.engines.repository_engine.internal.service import RepositoryInternalService +from backend.engines.repository_engine.schemas import ( + RepositoryDetail, + RepositoryMetadataInput, + RepositoryRecord, +) +from backend.providers.github_provider.client import GitHubProviderClient +from backend.providers.github_provider.errors import GitHubRepositoryTreeTruncatedError +from backend.providers.github_provider.schemas import ( + GitHubInstallationAccessToken, + GitHubRepository, + GitHubRepositorySnapshot, + GitHubRepositoryTreeEntry, +) + + +class SnapshotGitHubClient(GitHubProviderClient): + def __init__(self, *, truncated: bool = False) -> None: + super().__init__(api_base_url="https://api.github.test") + self.truncated = truncated + self.seen_headers: list[dict[str, str]] = [] + + async def _get_json( + self, + url: str, + *, + headers: dict[str, str], + not_found_message: str, + error_message: str, + params: dict[str, str] | None = None, + ) -> dict[str, Any]: + _ = not_found_message, error_message, params + self.seen_headers.append(headers) + if "/branches/" in url: + return {"commit": {"sha": "commit-123"}} + if "/git/commits/" in url: + return {"tree": {"sha": "tree-456"}} + if "/git/trees/" in url: + return { + "truncated": self.truncated, + "tree": [ + { + "path": "package.json", + "mode": "100644", + "type": "blob", + "sha": "blob-package", + "size": 40, + }, + { + "path": "src/main.ts", + "mode": "100644", + "type": "blob", + "sha": "blob-source", + "size": 20, + }, + { + "path": ".env", + "mode": "100644", + "type": "blob", + "sha": "blob-env", + "size": 20, + }, + ], + } + if url.endswith("/git/blobs/blob-package"): + return { + "encoding": "base64", + "content": "eyJzY3JpcHRzIjp7ImJ1aWxkIjoidml0ZSBidWlsZCJ9fQ==", + } + raise AssertionError(f"Unexpected GitHub URL: {url}") + + +@pytest.mark.asyncio +async def test_github_snapshot_is_commit_pinned_and_uses_installation_token() -> None: + client = SnapshotGitHubClient() + + result = await client.get_repository_snapshot( + owner="vibtools", + repo="ygit", + branch="main", + installation_credential="installation-secret", + ) + + assert result.commit_sha == "commit-123" + assert result.tree_sha == "tree-456" + assert result.truncated is False + assert [entry.path for entry in result.files] == [".env", "package.json", "src/main.ts"] + assert ".env" not in result.text_files + assert "vite build" in result.text_files["package.json"] + assert all( + headers.get("Authorization") == "Bearer installation-secret" + for headers in client.seen_headers + ) + payload = result.model_dump(mode="json") + assert "installation-secret" not in str(payload) + + +@pytest.mark.asyncio +async def test_github_snapshot_rejects_truncated_tree() -> None: + client = SnapshotGitHubClient(truncated=True) + + with pytest.raises(GitHubRepositoryTreeTruncatedError): + await client.get_repository_snapshot( + owner="vibtools", + repo="ygit", + branch="main", + installation_credential="installation-secret", + ) + + +class FakeGitHubCredentialProvider: + def create_app_jwt(self, *, app_id: str, private_key_pem: str) -> str: + assert app_id == "app-123" + assert private_key_pem == "private-key" + return "app-jwt" + + async def create_installation_access_token( + self, + *, + installation_id: int | str, + app_jwt: str, + ) -> GitHubInstallationAccessToken: + assert str(installation_id) == "456" + assert app_jwt == "app-jwt" + return GitHubInstallationAccessToken( + token="ephemeral-token", + expires_at="2026-07-25T23:00:00Z", + permissions={"contents": "read", "metadata": "read"}, + repository_selection="selected", + ) + + +@pytest.mark.asyncio +async def test_connected_accounts_acquires_ephemeral_github_credential(monkeypatch) -> None: + service = ConnectedAccountsInternalService( + repository=object(), # type: ignore[arg-type] + github_provider=FakeGitHubCredentialProvider(), # type: ignore[arg-type] + ) + record = ConnectedAccountRecord( + id="acct-1", + user_id="user-1", + provider="github", + status="connected", + provider_account_id="456", + provider_account_name="vibtools", + token_secret_ref="github_app_installation:456", + token_key_version="v1", + ) + + async def require_provider_connected(*args: Any, **kwargs: Any) -> ConnectedAccountRecord: + _ = args, kwargs + return record + + service.require_provider_connected = require_provider_connected # type: ignore[method-assign] + settings = SimpleNamespace( + github_app_id="app-123", + github_app_private_key=SecretStr("private-key"), + ) + monkeypatch.setattr( + "backend.engines.auth_engine.connected_accounts_module.internal.service.get_settings", + lambda: settings, + ) + + result = await service.acquire_github_repository_credential( + object(), # type: ignore[arg-type] + user_id="user-1", + ) + + assert result.installation_id == 456 + assert result.token_secret_ref == "github_app_installation:456" + assert result.access_token.get_secret_value() == "ephemeral-token" + assert result.permissions == {"contents": "read", "metadata": "read"} + + +class FakeConnectedAccountsService: + async def acquire_github_repository_credential( + self, + db: Any, + *, + user_id: str, + ) -> ResolvedGitHubInstallationCredential: + _ = db + assert user_id == "user-1" + return ResolvedGitHubInstallationCredential( + token_secret_ref="github_app_installation:456", + installation_id=456, + access_token=SecretStr("ephemeral-token"), + expires_at="2026-07-25T23:00:00Z", + permissions={"contents": "read"}, + repository_selection="selected", + ) + + +class FakeRepositoryProvider: + seen_token: str | None = None + + async def get_repository_metadata( + self, + *, + owner: str, + repo: str, + installation_credential: str | None = None, + ) -> GitHubRepository: + assert owner == "vibtools" + assert repo == "ygit" + self.seen_token = installation_credential + return GitHubRepository( + repository_url="https://github.com/vibtools/ygit", + owner="vibtools", + name="ygit", + default_branch="main", + visibility="private", + metadata={"id": 100}, + ) + + async def get_repository_snapshot( + self, + *, + owner: str, + repo: str, + branch: str, + installation_credential: str, + ) -> GitHubRepositorySnapshot: + assert (owner, repo, branch) == ("vibtools", "ygit", "main") + assert installation_credential == "ephemeral-token" + return GitHubRepositorySnapshot( + branch="main", + commit_sha="commit-123", + tree_sha="tree-456", + files=[ + GitHubRepositoryTreeEntry( + path="package.json", + mode="100644", + type="blob", + sha="blob-1", + size=40, + ) + ], + text_files={"package.json": '{"scripts":{"build":"vite build"}}'}, + ) + + +class FakeMetadataRepository: + stored_metadata: Any = None + + async def upsert_metadata( + self, + db: Any, + *, + user_id: str, + provider_metadata: Any, + ) -> RepositoryRecord: + _ = db + assert user_id == "user-1" + self.stored_metadata = provider_metadata + now = datetime.now(UTC) + return RepositoryRecord( + id="repo-1", + user_id=user_id, + provider="github", + repository_url=provider_metadata.repository_url, + owner=provider_metadata.owner, + name=provider_metadata.name, + default_branch=provider_metadata.default_branch, + visibility=provider_metadata.visibility, + latest_commit_sha=provider_metadata.latest_commit_sha, + file_tree_snapshot=provider_metadata.file_tree_snapshot, + metadata=provider_metadata.metadata, + fetched_at=now, + created_at=now, + updated_at=now, + ) + + +class FakeDb: + committed = False + + async def commit(self) -> None: + self.committed = True + + +@pytest.mark.asyncio +async def test_repository_engine_persists_snapshot_without_raw_token() -> None: + metadata_repository = FakeMetadataRepository() + provider = FakeRepositoryProvider() + service = RepositoryInternalService( + repository=metadata_repository, # type: ignore[arg-type] + github_provider=provider, # type: ignore[arg-type] + connected_accounts_public_service=FakeConnectedAccountsService(), # type: ignore[arg-type] + ) + db = FakeDb() + + result = await service.fetch_repository_metadata( + db, # type: ignore[arg-type] + user_id="user-1", + input_data=RepositoryMetadataInput( + repository_url="https://github.com/vibtools/ygit" + ), + ) + + assert db.committed is True + assert provider.seen_token == "ephemeral-token" + assert result.latest_commit_sha == "commit-123" + assert result.file_tree_snapshot is not None + assert result.file_tree_snapshot["tree_sha"] == "tree-456" + stored = metadata_repository.stored_metadata.model_dump(mode="json") + assert "ephemeral-token" not in str(stored) + + +@pytest.mark.asyncio +async def test_prepare_analysis_input_exposes_commit_tree_and_text_files() -> None: + service = RepositoryInternalService( + repository=object(), # type: ignore[arg-type] + github_provider=object(), # type: ignore[arg-type] + connected_accounts_public_service=object(), # type: ignore[arg-type] + ) + now = datetime.now(UTC) + detail = RepositoryDetail( + id="repo-1", + user_id="user-1", + provider="github", + repository_url="https://github.com/vibtools/ygit", + owner="vibtools", + name="ygit", + default_branch="main", + visibility="private", + latest_commit_sha="commit-123", + file_tree_snapshot={ + "version": 1, + "branch": "main", + "commit_sha": "commit-123", + "tree_sha": "tree-456", + "truncated": False, + "files": [{"path": "package.json", "type": "blob", "sha": "blob-1"}], + "text_files": {"package.json": '{"scripts":{"build":"vite build"}}'}, + }, + metadata={}, + fetched_at=now, + created_at=now, + updated_at=now, + ) + + async def get_repository_metadata(*args: Any, **kwargs: Any) -> RepositoryDetail: + _ = args, kwargs + return detail + + service.get_repository_metadata = get_repository_metadata # type: ignore[method-assign] + result = await service.prepare_analysis_input( + object(), # type: ignore[arg-type] + user_id="user-1", + repository_id="repo-1", + ) + + assert result.latest_commit_sha == "commit-123" + assert result.tree_sha == "tree-456" + assert result.snapshot_complete is True + assert result.file_contents["package.json"].startswith("{") + + +@pytest.mark.asyncio +async def test_prepare_analysis_input_rejects_commit_mismatch() -> None: + service = RepositoryInternalService( + repository=object(), # type: ignore[arg-type] + github_provider=object(), # type: ignore[arg-type] + connected_accounts_public_service=object(), # type: ignore[arg-type] + ) + now = datetime.now(UTC) + detail = RepositoryDetail( + id="repo-1", + user_id="user-1", + provider="github", + repository_url="https://github.com/vibtools/ygit", + owner="vibtools", + name="ygit", + default_branch="main", + visibility="private", + latest_commit_sha="commit-new", + file_tree_snapshot={ + "version": 1, + "commit_sha": "commit-old", + "tree_sha": "tree-456", + "truncated": False, + "files": [], + "text_files": {}, + }, + metadata={}, + fetched_at=now, + created_at=now, + updated_at=now, + ) + + async def get_repository_metadata(*args: Any, **kwargs: Any) -> RepositoryDetail: + _ = args, kwargs + return detail + + service.get_repository_metadata = get_repository_metadata # type: ignore[method-assign] + with pytest.raises(RepositorySnapshotIncompleteError): + await service.prepare_analysis_input( + object(), # type: ignore[arg-type] + user_id="user-1", + repository_id="repo-1", + ) diff --git a/backend/tests/test_repository_analysis_current_state_audit.py b/backend/tests/test_repository_analysis_current_state_audit.py index b09872c..6fa2d92 100644 --- a/backend/tests/test_repository_analysis_current_state_audit.py +++ b/backend/tests/test_repository_analysis_current_state_audit.py @@ -40,9 +40,9 @@ def compact(source: str) -> str: def test_audit_document_has_version_status_and_revision_history() -> None: source = AUDIT.read_text(encoding="utf-8") - assert "Version: 1.0" in source - assert "Status: Approved Current-State Audit" in source - assert "Implementation Change: None" in source + assert "Version: 1.1" in source + assert "Status: Implemented Current-State Record" in source + assert "Implementation Change: Update 1" in source assert "## Revision History" in source @@ -79,25 +79,25 @@ def test_file_index_contract_matches_documented_keys() -> None: assert key in audit -def test_github_metadata_current_gap_is_documented() -> None: +def test_github_acquisition_is_commit_pinned_and_authenticated() -> None: source = GITHUB_CLIENT.read_text(encoding="utf-8") - metadata = source.split( - "async def get_repository_metadata(", + acquisition = source.split( + "async def get_repository_snapshot(", 1, )[1].split( - "def _app_headers(", + "async def _fetch_analysis_text_files(", 1, )[0] - compact_metadata = compact(metadata) audit = AUDIT.read_text(encoding="utf-8") - assert "latest_commit_sha=None" in compact_metadata - assert "file_tree_snapshot=" in metadata - assert '{"default_branch": default_branch}' in metadata - assert "if default_branch else None" in compact_metadata - assert '"Authorization"' not in metadata - assert "no pinned commit SHA" in audit - assert "no actual file tree" in audit + assert "installation_credential" in acquisition + assert 'headers = self._installation_headers(installation_credential)' in acquisition + assert 'commit_sha = str((branch_payload.get("commit")' in acquisition + assert 'tree_sha = str((commit_payload.get("tree")' in acquisition + assert 'if bool(tree_payload.get("truncated"))' in acquisition + assert "GitHubRepositorySnapshot(" in acquisition + assert "Raw GitHub tokens must never be persisted or returned" in audit + assert "Repository acquisition is commit-pinned" in audit def test_repository_analysis_input_passes_stored_snapshot() -> None: @@ -113,9 +113,11 @@ def test_repository_analysis_input_passes_stored_snapshot() -> None: assert "latest_commit_sha=detail.latest_commit_sha" in compact( prepared ) - assert "file_tree_snapshot=detail.file_tree_snapshot" in compact( - prepared - ) + prepared_compact = compact(prepared) + assert "snapshot_complete=True" in prepared_compact + assert "file_tree_snapshot=snapshot" in prepared_compact + assert "file_contents=file_contents" in prepared_compact + assert 'snapshot_commit != detail.latest_commit_sha' in prepared_compact def test_readiness_is_fail_closed_for_unknown_or_dynamic_input() -> None: @@ -174,7 +176,7 @@ def test_project_status_links_audit_and_states_remaining_work() -> None: source = PROJECT_STATUS.read_text(encoding="utf-8") assert "REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md" in source - assert "live tree acquisition incomplete" in source + assert "commit-pinned snapshot implemented" in source assert "Project reattachment after recalculation" in source assert "Full suite:" in source assert "passed, 1 warning" in source diff --git a/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md b/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md new file mode 100644 index 0000000..ac6a504 --- /dev/null +++ b/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md @@ -0,0 +1,75 @@ +# GitHub Repository Acquisition and Analyzer Input Specification + +**Version:** 1.0.0
+**Status:** Implemented for Verification
+**Product:** YGIT
+**Update:** Update 1 — GitHub Repository Acquisition + Analyzer Input
+**Baseline Commit:** `d2ad741c0ee48bf3eb3887a3357f6f450825c99a` + +## Purpose + +This update establishes the authenticated, commit-pinned GitHub repository acquisition path required before strengthening framework and Cloudflare readiness analysis. + +## Runtime Flow + +```text +Project Engine +→ Repository Engine public API +→ Connected Accounts public API +→ ephemeral GitHub App installation access token +→ GitHub Provider +→ repository metadata +→ default branch head commit +→ exact Git tree +→ bounded analysis-critical text files +→ Repository Engine persistence +→ Repository Analysis Engine input +``` + +## Security Contract + +- The GitHub App installation reference remains stored as a safe reference. +- Installation access tokens are generated server-side only when needed. +- Raw GitHub tokens are never stored in `repository_metadata`. +- Raw GitHub tokens are never returned by API routes. +- Repository snapshots contain only repository paths, Git object identifiers, bounded text content, and safe metadata. +- Provider execution for Cloudflare remains disabled. + +## Snapshot Contract + +A complete snapshot contains: + +- schema version; +- default branch; +- exact commit SHA; +- exact root tree SHA; +- non-truncated recursive file inventory; +- bounded UTF-8 contents for analysis-critical configuration files; +- acquisition limits and skipped-file reasons. + +Truncated GitHub tree responses fail closed with `REPOSITORY_SNAPSHOT_INCOMPLETE`. Partial repository analysis is not accepted. + +## Analyzer Input Contract + +Repository Analysis receives: + +- repository identity; +- exact commit SHA; +- exact tree SHA; +- snapshot version; +- `snapshot_complete=true`; +- complete file-path inventory; +- bounded analysis-critical file contents. + +Update 1 does not expand framework detection rules. Strong framework/build/output/readiness detection belongs to Update 2. + +## Database Impact + +No migration is required. Existing `repository_metadata.latest_commit_sha`, `file_tree_snapshot`, and `metadata` fields store the normalized acquisition result. + +## Provider Impact + +- GitHub App read access is used. +- No GitHub write operation is added. +- No Cloudflare API execution is added. +- `WORKER_PROVIDER_EXECUTION_MODE=disabled` remains unchanged. From 65a6958cd75cb098cc98f5dea3a908e4cc5cf9ca Mon Sep 17 00:00:00 2001 From: vib-tools Date: Sat, 25 Jul 2026 19:10:02 -0700 Subject: [PATCH 2/2] docs: complete Update 01 records and PR package --- CHANGELOG.md | 56 +++ GITHUB_APP_INTEGRATION.md | 58 ++- GITHUB_APP_PERMISSION_CAPTURE_SPEC.md | 33 +- PROJECT_ENGINE_DELIVERY_NOTE.md | 16 + PROJECT_STATUS.md | 71 ++-- README.md | 67 ++-- REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md | 22 ++ REPOSITORY_ANALYSIS_ENGINE_DELIVERY_NOTE.md | 20 + REPOSITORY_ENGINE_DELIVERY_NOTE.md | 50 +++ VERSION.json | 42 +- .../connected_accounts_module/README.md | 22 ++ .../repository_analysis_engine/README.md | 18 + backend/engines/repository_engine/README.md | 43 +++ ...CQUISITION_ANALYZER_INPUT_SPECIFICATION.md | 358 +++++++++++++++--- docs/updates/README.md | 14 + ...ITHUB_REPOSITORY_ACQUISITION_COMPLETION.md | 81 ++++ docs/updates/UPDATE_01_PR_CHECKLIST.md | 75 ++++ .../UPDATE_01_VERIFICATION_EVIDENCE.md | 58 +++ 18 files changed, 978 insertions(+), 126 deletions(-) create mode 100644 docs/updates/README.md create mode 100644 docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md create mode 100644 docs/updates/UPDATE_01_PR_CHECKLIST.md create mode 100644 docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f5df044..62901e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,61 @@ # Changelog +## Unreleased — Update 01: GitHub Repository Acquisition + Analyzer Input — 2026-07-25 + +### Added + +- Ephemeral GitHub App installation-credential acquisition through the Connected Accounts public boundary. +- Installation-authenticated repository metadata acquisition for public and private repositories authorized to the installation. +- Default-branch head resolution to an exact commit SHA. +- Exact commit-tree resolution and complete recursive Git-tree acquisition. +- Bounded UTF-8 contents for analysis-critical repository files. +- Versioned, complete, commit-pinned repository snapshot persistence. +- Repository Analysis input containing commit SHA, tree SHA, snapshot version, completeness state, full path inventory, and bounded text evidence. +- Fail-closed errors for inaccessible repositories, missing default branches, incomplete/truncated trees, and invalid installation credentials. +- Update 01 architecture, completion, verification, and PR-control documentation. + +### Changed + +- Repository Engine now obtains the GitHub installation credential through Connected Accounts instead of relying on unauthenticated metadata-only acquisition. +- GitHub Provider now accepts an installation access token for repository reads and resolves metadata, commit, tree, inventory, and selected text content. +- Repository metadata persistence now records the exact latest commit and normalized snapshot required by Repository Analysis. +- Project and Repository Analysis documentation now distinguish completed Update 01 acquisition from Update 02 detection/readiness work. +- Current-state documents now identify the verified implementation branch and commit. + +### Verified + +- Implementation branch: `feature/github-repository-acquisition-analyzer-input-v1`. +- Implementation commit: `cb796401afe8be64eb311a45e566fc315a13a00c`. +- Base commit: `d2ad741c0ee48bf3eb3887a3357f6f450825c99a`. +- Exact implementation scope: 19 files. +- Ruff `0.15.20`: PASS. +- Update 01 focused tests: 27 passed. +- Full test suite: 609 passed. +- Compileall: PASS. +- Smoke test with database skipped: PASS. +- Release gate with database skipped: PASS. +- Remote feature-branch SHA verification: PASS. +- Working tree after push: CLEAN. + +### Security and Architecture Boundaries + +- Raw GitHub installation access tokens are not persisted or returned. +- Private source contents are not logged. +- API routes do not call GitHub directly. +- Repository Analysis does not call GitHub directly. +- No database migration was added. +- No Cloudflare operation or provider enablement was added. +- `WORKER_PROVIDER_EXECUTION_MODE=disabled` remains unchanged. + +### Not Executed + +- Live GitHub API acquisition against a controlled installation. +- Live PostgreSQL verification. +- Live Redis worker execution. +- Live Cloudflare API execution. +- Real Cloudflare Pages deployment. +- Merge, production redeploy, or branch-protection change. + All notable YGIT MVP implementation releases and active engineering foundations are tracked here. ## Unreleased — Phase 0 Completion Record — 2026-07-24 diff --git a/GITHUB_APP_INTEGRATION.md b/GITHUB_APP_INTEGRATION.md index 8d0fada..03d1c48 100644 --- a/GITHUB_APP_INTEGRATION.md +++ b/GITHUB_APP_INTEGRATION.md @@ -1,7 +1,7 @@ # YGIT GitHub App Integration Architecture -Version: 1.1 -Status: Architecture Locked +Version: 1.2 +Status: Architecture Locked; Update 01 Implementation Aligned Owner: Repository Integration ## Purpose @@ -94,6 +94,41 @@ Cloudflare remains a separate OAuth-connected provider. GitHub App credentials must never be reused as Cloudflare credentials, and Cloudflare OAuth credentials must never be used for GitHub integration. +## Update 01 Repository Acquisition Contract + +Update 01 implements repository reads through the connected GitHub App installation: + +```text +Connected Account installation reference + ↓ +server-generated GitHub App JWT + ↓ +short-lived installation access token + ↓ +repository metadata + ↓ +default branch head commit + ↓ +exact commit tree + ↓ +complete recursive inventory + ↓ +bounded analysis-critical text files +``` + +The installation token is ephemeral. It is used only inside the Connected Accounts and GitHub Provider boundary for the current request. It must not be stored in `connected_accounts`, `repository_metadata`, analysis results, job payloads, logs, API responses, or documentation artifacts. + +The minimum repository-read permission target remains: + +```text +metadata:read +contents:read +``` + +A missing, disconnected, invalid, or expired installation credential fails closed. A truncated or oversized recursive tree is not accepted as complete analyzer evidence. + +Update 01 does not add GitHub write operations, webhook processing, GitHub OAuth, repository mutation, deployment execution, or Cloudflare credential reuse. + ## Webhook Capability State The current YGIT MVP does not expose an approved GitHub webhook receiver endpoint and does not process GitHub webhook deliveries. @@ -144,3 +179,22 @@ This contract does not: - change Cloudflare OAuth; - change AG-001; - create a YGIT App Engine. + +## Update 01 Evidence + +```text +Implementation branch: feature/github-repository-acquisition-analyzer-input-v1 +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Focused tests: 27 passed +Full suite: 609 passed +Ruff 0.15.20: PASS +Remote branch verification: PASS +Live GitHub API verification: NOT EXECUTED +``` + +## Revision History + +| Date | Version | Summary | +|---|---|---| +| 2026-07-25 | 1.2 | Aligned the locked GitHub App architecture with Update 01 ephemeral installation-token repository acquisition | +| Earlier | 1.1 | Locked GitHub App repository authorization, Keycloak user authentication, and default-disabled webhook capability | diff --git a/GITHUB_APP_PERMISSION_CAPTURE_SPEC.md b/GITHUB_APP_PERMISSION_CAPTURE_SPEC.md index c05631b..948f670 100644 --- a/GITHUB_APP_PERMISSION_CAPTURE_SPEC.md +++ b/GITHUB_APP_PERMISSION_CAPTURE_SPEC.md @@ -1,7 +1,7 @@ # YGIT GitHub App Installation Permission Capture Specification -Version: 1.0 -Status: Approved for Implementation +Version: 1.1 +Status: Implemented; Controlled Live Installation Verification Pending Owner: Connected Accounts / GitHub Provider ## Purpose @@ -62,3 +62,32 @@ This patch does not: - create a database migration. A later independent UI patch will display the captured permission posture. + +## Update 01 Integration + +Repository Engine requests an ephemeral GitHub installation credential through the Connected Accounts public API. Connected Accounts validates that the stored GitHub account record is connected and that its installation reference can be resolved before issuing the short-lived credential. + +The captured scope metadata is safe account metadata; it is not itself an access token. Update 01 depends on the least-privilege target: + +```text +metadata:read +contents:read +``` + +The acquisition path fails closed when the GitHub account is missing, disconnected, invalid, expired, or cannot produce an installation credential. + +## Verification State + +```text +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Local permission and acquisition contract tests: PASS +Live installation permission verification: NOT EXECUTED +GitHub App settings mutation: NOT PERFORMED +``` + +## Revision History + +| Date | Version | Summary | +|---|---|---| +| 2026-07-25 | 1.1 | Recorded Update 01 credential-acquisition dependency and live-verification boundary | +| Earlier | 1.0 | Approved deterministic installation-permission capture contract | diff --git a/PROJECT_ENGINE_DELIVERY_NOTE.md b/PROJECT_ENGINE_DELIVERY_NOTE.md index 6efb7eb..ec94764 100644 --- a/PROJECT_ENGINE_DELIVERY_NOTE.md +++ b/PROJECT_ENGINE_DELIVERY_NOTE.md @@ -22,3 +22,19 @@ Project Engine implementation added on top of YGIT Auth Engine v0.1.0. - Deployment History implementation Those remain separate implementation phases under Architecture Freeze v1.1. + +## Update 01 Integration Addendum + +Update 01 does not change Project Engine ownership. It documents and verifies the existing integration path in which project creation receives the Repository Engine record and initial Repository Analysis result through public boundaries. + +```text +Project creation +→ Repository Engine commit-pinned snapshot +→ Repository Analysis quick result +→ Project repository/analysis attachment +→ deploy readiness remains gated by stored analysis evidence +``` + +Recalculation-time reattachment of a newer analysis remains Update 02 work. No Project Engine model, migration, route, or ownership boundary was added by Update 01. + +Verification reference: `cb796401afe8be64eb311a45e566fc315a13a00c`; full suite 609 passed. diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md index 15079d5..e004196 100644 --- a/PROJECT_STATUS.md +++ b/PROJECT_STATUS.md @@ -1,8 +1,8 @@ # YGIT Project Status Version: 0.1.0 -Status: Active Engineering Snapshot -Updated: 2026-07-24 +Status: Update 01 Implementation Verified; Draft PR Documentation Gate +Updated: 2026-07-25 Product: YGIT Company: Vib Tools @@ -71,6 +71,15 @@ Repository acquisition status: **commit-pinned snapshot implemented**. ## Recently Completed Foundations +### Update 01 — GitHub Repository Acquisition + Analyzer Input + +- Connected Accounts resolves the stored GitHub App installation reference and creates a short-lived installation access token for approved Repository Engine calls. +- GitHub Provider reads repository metadata through the installation token, resolves the default branch to commit `cb796401afe8be64eb311a45e566fc315a13a00c`-era implementation behavior, resolves the exact Git tree, and acquires a complete recursive inventory. +- Repository snapshots are commit-pinned, versioned, bounded, and fail closed when GitHub reports a truncated or oversized tree. +- Selected analysis-critical UTF-8 files are included within bounded content limits; sensitive filenames and unsupported/binary content are excluded. +- Repository Analysis receives the complete snapshot through the Repository Engine public boundary and does not call GitHub directly. +- No migration, Cloudflare execution, provider-mode change, UI-V2 change, or production deployment was included. + - Worker database-aware dispatch contract. - Cloudflare credential acquisition boundary. - Connected Accounts avatar, status, connection date, scopes, last sync, and imported-repository reuse UI. @@ -108,47 +117,35 @@ GitHub integration is architecture-locked to a GitHub App. Vib ID / Keycloak rem ## Latest Verification Evidence ```text -Provider execution policy unit tests: 18 passed -Provider policy runtime integration: 9 passed -Deploy/redeploy handler binding regression: 5 passed -Dispatcher DB regression: 5 passed -Worker Runtime architecture: 4 passed -Deploy/redeploy architecture: 2 passed -AG-001 regression: 15 passed -AG-002 regression: 9 passed -Deployment History runtime: 8 passed -Deployment History idempotency: 4 passed -Live-readiness tooling: 18 passed -Runtime image packaging: 4 passed -GitHub App permission capture: 7 passed -Dashboard compact provider cards: 10 passed -Project Open UI: 9 passed -Project Deploy UI: 9 passed -Full suite: 579 passed, 1 warning +Update 01 branch: feature/github-repository-acquisition-analyzer-input-v1 +Update 01 implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Update 01 base commit: d2ad741c0ee48bf3eb3887a3357f6f450825c99a +Implementation file scope: 19 +Ruff 0.15.20: PASS +Update 01 focused tests: 27 passed +Full suite: 609 passed +Previous foundation full suite: 579 passed, 1 warning +Compileall: PASS Smoke --skip-db: PASS Release gate --skip-db: PASS +Remote feature-branch SHA match: PASS +Working tree after push: CLEAN Backend CI workflow: IMPLEMENTED -Backend CI / Validate: SUCCESS -Backend CI final-head pull-request run: 30096212556 -Backend CI final-head pull-request job: 89490793519 -Backend CI post-merge push run: 30106115262 -Backend CI post-merge push job: 89523839117 -Backend CI post-merge push verification: SUCCESS -Phase 0 merge commit: 6e44866de9ec3a3a745777afc12276f903259709 -Phase 0 completion: COMPLETE +Draft PR: PENDING DOCUMENTATION COMMIT +Provider execution mode: DISABLED Branch protection required-check enablement: NOT AUTHORIZED ``` -Database checks were skipped. External providers were not executed. +Database checks were skipped. Live GitHub, Cloudflare, PostgreSQL, Redis-worker, and production deployment behavior were not executed. ## Remaining Critical Path -1. Redeploy the verified current `main` branch and validate the Dashboard compact provider cards, Project Open flow, and backend-readiness-gated Deploy flow. -2. Reduce the GitHub App to the approved minimum permissions, reconnect the controlled installation, and verify captured permission scopes. -3. Implement GitHub App installation-token repository acquisition with a pinned commit SHA and normalized real file-tree snapshot. -4. Implement the approved deep-analysis execution and Project reattachment boundaries. -5. Confirm `deploy_ready=true` from real repository evidence and execute one controlled Cloudflare Pages deployment. -6. Resolve only defects demonstrated by live evidence. +1. Commit and push the Update 01 documentation closure. +2. Create the detailed Draft PR and require successful `Backend CI / Validate` on its final head. +3. Complete exact-scope review and merge only after explicit authorization. +4. Run controlled live GitHub App acquisition against an approved private repository after merge. +5. Implement Update 02 strong analyzer and Cloudflare-readiness contracts using the acquired text evidence. +6. Execute one controlled Cloudflare Pages deployment only after separate provider-execution authorization. 7. Keep AG-001 and AG-002 runtime integration deferred until separate post-MVP architecture approval. ## Documentation Authority @@ -166,6 +163,11 @@ Current-state documents: - `docs/ci/BACKEND_CI_SPECIFICATION.md` - `docs/ci/BACKEND_CI_IMPLEMENTATION_PLAN.md` - `docs/ci/BACKEND_CI_TESTING_AND_ROLLBACK_SPECIFICATION.md` +- `docs/updates/README.md` +- `docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md` +- `docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md` +- `docs/updates/UPDATE_01_PR_CHECKLIST.md` +- `docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md` - `.github/workflows/backend-ci.yml` Historical release artifacts retain their original versioned purpose. Where a historical artifact conflicts with this current snapshot, this document and the current source code take precedence for development status. @@ -174,6 +176,7 @@ Historical release artifacts retain their original versioned purpose. Where a hi | Date | Revision | Summary | |---|---|---| +| 2026-07-25 | 2.3 | Recorded Update 01 implementation commit `cb796401afe8be64eb311a45e566fc315a13a00c`, exact remote-branch verification, 609-test evidence, complete documentation set, and Draft PR gate | | 2026-07-24 | 2.2 | Recorded PR #1 controlled merge, merge commit `6e44866de9ec3a3a745777afc12276f903259709`, successful merged-main Backend CI, and Phase 0 completion | | 2026-07-24 | 2.1 | Recorded Backend CI implementation, successful Draft PR validation, security boundaries, and remaining post-merge Phase 0 gates | | 2026-07-23 | 2.0 | Locked the running baseline, reconciled current status/configuration evidence, and added the standalone AG-002 Repository Provider Gate foundation without runtime wiring | diff --git a/README.md b/README.md index 028ed99..2a31a3b 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,9 @@ Website Live ## Current Development Status -Status: **Ready for controlled live redeploy** +Status: **Update 01 implementation verified; Draft PR documentation gate ready** -The MVP architecture and core engines are substantially implemented. Concrete Cloudflare Pages orchestration, provider pipeline binding, DB-aware handlers, and trusted server-owned provider-policy handoff are present. The default server policy remains `disabled`, so provider execution stays off until an operator explicitly selects the supported `cloudflare` mode. +The MVP architecture and core engines are substantially implemented. Update 01 now provides installation-authenticated GitHub repository acquisition, exact commit pinning, a complete recursive tree snapshot, bounded analysis-critical text input, and Repository Analysis handoff. The default server policy remains `disabled`; no Cloudflare provider execution was enabled by Update 01. Engineering estimate: @@ -83,8 +83,8 @@ Object storage target: Cloudflare R2 | Auth Engine | Implemented with OIDC/session boundaries | | Connected Accounts Module | Implemented with GitHub/Cloudflare connection state, credential references, credential acquisition boundary, metadata UI, scopes, last-sync data, and disconnect/reconnect flows | | Project Engine | Implemented | -| Repository Engine | Metadata persistence implemented; AG-002 standalone provider gate foundation added but not runtime-wired; real installation-token tree acquisition remains incomplete | -| Repository Analysis Engine | Quick-analysis contract implemented; real repository tree input, deep execution, and Project reattachment remain incomplete | +| Repository Engine | GitHub App installation-authenticated metadata acquisition, exact commit pinning, complete recursive tree snapshot, bounded analysis-text collection, and analyzer-input preparation implemented; AG-002 remains standalone and not runtime-wired | +| Repository Analysis Engine | Quick Analysis receives a complete commit-pinned snapshot and bounded text evidence; stronger detection, deep worker completion, and recalculation reattachment remain Update 02 work | | Deploy Engine | Implemented for validation, queued deployment lifecycle, redeploy, cancel, and reads | | Deploy Pipeline | Contracts, build stage, Cloudflare operation plan, concrete Cloudflare Pages gateway, completion result branch, and isolated provider pipeline factory implemented | | Deployment History Engine | Pipeline completion/failure persistence, retry-safe intent consumption, logs, and provider-result storage integrated through the public engine boundary | @@ -95,7 +95,7 @@ Object storage target: Cloudflare R2 | Notification Engine | Implemented for in-app MVP scope | | Dashboard | Implemented, including Connected Accounts metadata and imported-repository reuse UI | | Admin Panel | Implemented as a protected operations console | -| GitHub Provider | Repository metadata and account integration foundations implemented | +| GitHub Provider | Installation-token metadata, default-branch head, commit/tree, recursive inventory, and bounded content acquisition implemented; controlled live API validation remains pending | | Cloudflare Provider | OAuth/account foundations and Cloudflare Pages project, artifact, asset upload, hash registration, and deployment primitives implemented | ## App Gate Foundations @@ -166,36 +166,28 @@ Still disabled or incomplete: ## Verification Snapshot -Latest verified local run for the current foundation: +Latest verified local and remote-branch run for Update 01: ```text -Running baseline lock: PASS at b9019b79d1af3fe73d1a74769792ebb6958c4f4c -Provider execution policy unit tests: 18 passed -Provider policy runtime integration tests: 9 passed -Handler binding regression: 5 passed -Dispatcher DB regression: 5 passed -Worker Runtime architecture tests: 4 passed -Deploy/redeploy architecture tests: 2 passed -AG-001 regression: 15 passed -AG-002 regression: 9 passed -Deployment History runtime tests: 8 passed -Deployment History idempotency tests: 4 passed -Live-readiness tooling tests: 18 passed -Runtime image packaging tests: 4 passed -Full suite: 579 passed, 1 warning +Implementation branch: feature/github-repository-acquisition-analyzer-input-v1 +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Branch base: d2ad741c0ee48bf3eb3887a3357f6f450825c99a +Changed implementation files: 19 +Ruff: 0.15.20 — PASS +Update 01 focused tests: 27 passed +Full suite: 609 passed +Compileall: PASS Smoke test with database skipped: PASS Release gate with database skipped: PASS -Backend CI workflow: IMPLEMENTED at .github/workflows/backend-ci.yml -Backend CI final-head pull-request validation: SUCCESS — run 30096212556 / job 89490793519 -Backend CI post-merge main validation: SUCCESS — run 30106115262 / job 89523839117 -Phase 0 merge commit: 6e44866de9ec3a3a745777afc12276f903259709 -Phase 0 completion record: COMPLETE -Branch-protection required-check enablement: NOT AUTHORIZED +Remote feature-branch SHA match: PASS +Working tree after push: CLEAN +Draft PR: PENDING DOCUMENTATION COMMIT Live PostgreSQL: NOT EXECUTED Live Redis worker loop: NOT EXECUTED -GitHub API integration: NOT EXECUTED +Live GitHub API acquisition: NOT EXECUTED Cloudflare API integration: NOT EXECUTED Real Cloudflare Pages deployment: NOT EXECUTED +Provider execution mode: DISABLED ``` The recurring `StarletteDeprecationWarning` is non-blocking and relates to the existing test-client dependency combination. @@ -239,13 +231,20 @@ Live checks must use the controlled runtime runbook and dedicated test accounts. ## Immediate Critical Path -1. Redeploy the verified current `main` branch and validate the Dashboard compact provider cards, Project Open flow, and backend-readiness-gated Deploy flow. -2. Reduce the GitHub App to the approved minimum permissions, reconnect the controlled installation, and verify captured permission scopes. -3. Implement GitHub App installation-token repository acquisition with a pinned commit SHA and normalized real file-tree snapshot. -4. Implement the approved deep-analysis execution and Project reattachment boundaries. -5. Confirm `deploy_ready=true` from real repository evidence and execute one controlled Cloudflare Pages deployment. -6. Resolve only defects demonstrated by live evidence. -7. Keep AG-001 and AG-002 runtime integration deferred until separate post-MVP architecture approval. +1. Commit this documentation closure, push the feature branch, and open the detailed Draft PR. +2. Require successful `Backend CI / Validate` on the final PR head and review the exact documentation and implementation scope. +3. Merge only after explicit approval; no production redeploy is part of this documentation patch. +4. After merge, perform controlled live GitHub App validation against an approved private repository. +5. Begin Update 02 for stronger analyzer rules, monorepo/root detection, command/output resolution, environment-reference detection, and Cloudflare readiness. +6. Execute one controlled Cloudflare Pages deployment only after readiness is derived from real repository evidence and provider execution is separately authorized. +7. Keep AG-001 and AG-002 runtime integration deferred until separate architecture approval. + +## Update 01 Documentation + +- [Completion Record](docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md) +- [Verification Evidence](docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md) +- [Draft PR Checklist](docs/updates/UPDATE_01_PR_CHECKLIST.md) +- [Architecture Specification](docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md) ## Documentation Scope diff --git a/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md b/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md index c51beba..c7d7f12 100644 --- a/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md +++ b/REPOSITORY_ANALYSIS_CURRENT_STATE_AUDIT.md @@ -2,6 +2,7 @@ Version: 1.1 Status: Implemented Current-State Record +Update 01 State: Implementation Verified; Draft PR Documentation Gate Date: 2026-07-25 Owner: Repository Analysis Engine Implementation Change: Update 1 — GitHub Repository Acquisition + Analyzer Input @@ -94,6 +95,26 @@ Deep Analysis still creates or enqueues a `repository_analysis_deep` job referen The recalculation path still runs Quick Analysis using the Repository ID without reattaching the new result through Project Engine. This remains separate approved work. +## Update 01 Closure Evidence + +```text +Baseline commit: d2ad741c0ee48bf3eb3887a3357f6f450825c99a +Implementation branch: feature/github-repository-acquisition-analyzer-input-v1 +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Implementation file count: 19 +Ruff 0.15.20: PASS +Focused tests: 27 passed +Full suite: 609 passed +Compileall: PASS +Smoke --skip-db: PASS +Release gate --skip-db: PASS +Remote branch SHA match: PASS +Working tree clean after push: PASS +Draft PR: pending documentation commit +``` + +No live GitHub API, PostgreSQL, Redis-worker, Cloudflare API, or production deployment execution is claimed by this evidence. + ## Remaining Implementation Sequence 1. Strengthen detection using the acquired text evidence. @@ -134,5 +155,6 @@ Passing tests prove the deterministic provider, engine, snapshot, analyzer-input | Date | Version | Summary | |---|---|---| +| 2026-07-25 | 1.2 | Added authoritative branch/commit, complete verification evidence, PR gate, and explicit live-validation boundary | | 2026-07-25 | 1.1 | Implemented installation-authenticated metadata, commit pinning, complete tree snapshot, bounded text acquisition, and analyzer-input handoff | | 2026-07-22 | 1.0 | Recorded the metadata-only gap and remaining analysis boundaries | diff --git a/REPOSITORY_ANALYSIS_ENGINE_DELIVERY_NOTE.md b/REPOSITORY_ANALYSIS_ENGINE_DELIVERY_NOTE.md index 3dd3ade..fb32233 100644 --- a/REPOSITORY_ANALYSIS_ENGINE_DELIVERY_NOTE.md +++ b/REPOSITORY_ANALYSIS_ENGINE_DELIVERY_NOTE.md @@ -32,3 +32,23 @@ Result Store ## Notes Deep Analysis is contract-enabled and returns a queued job reference. Durable job persistence and worker execution remain owned by the Worker / Job System phase. + +## Update 01 Input Addendum + +Repository Analysis now receives a complete Repository Engine snapshot rather than metadata-only placeholder input. + +The input includes: + +- exact repository commit SHA; +- exact Git tree SHA; +- snapshot schema version; +- `snapshot_complete=true`; +- complete file-path inventory within the approved tree limit; +- bounded contents of analysis-critical text files; +- skipped-file and acquisition-limit metadata. + +Quick Analysis continues to use the existing v0.1.0 detection rules. Update 01 does not expand framework, monorepo, install-command, build-command, output-directory, environment-reference, or Cloudflare-readiness detection. + +Incomplete snapshots are rejected before they can become deploy-readiness evidence. + +Verification reference: `cb796401afe8be64eb311a45e566fc315a13a00c`; 27 focused tests and 609 full-suite tests passed. diff --git a/REPOSITORY_ENGINE_DELIVERY_NOTE.md b/REPOSITORY_ENGINE_DELIVERY_NOTE.md index 8a82db3..c8ddff7 100644 --- a/REPOSITORY_ENGINE_DELIVERY_NOTE.md +++ b/REPOSITORY_ENGINE_DELIVERY_NOTE.md @@ -63,3 +63,53 @@ Repository Analysis Engine Deep analysis worker Cloudflare deployment flow ``` + +## Update 01 Delivery Addendum — GitHub Repository Acquisition + +### Added + +- Connected Accounts public-boundary acquisition of an ephemeral GitHub installation credential. +- Installation-authenticated repository metadata reads. +- Default-branch head resolution to an exact commit SHA. +- Exact commit-tree resolution. +- Complete recursive Git-tree inventory with fail-closed truncation handling. +- Bounded UTF-8 contents for analysis-critical configuration files. +- Versioned commit-pinned snapshot persistence. +- Complete Repository Analysis input preparation. + +### Failure Contract + +```text +repository inaccessible +→ REPOSITORY_NOT_FOUND or access failure + +default branch missing +→ REPOSITORY_DEFAULT_BRANCH_MISSING + +tree truncated, oversized, or incomplete +→ REPOSITORY_SNAPSHOT_INCOMPLETE + +installation credential unavailable or invalid +→ connected-account/provider failure +``` + +### Verification + +```text +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Update 01 focused tests: 27 passed +Full suite: 609 passed +Ruff 0.15.20: PASS +Smoke --skip-db: PASS +Release gate --skip-db: PASS +Remote branch SHA: VERIFIED +``` + +### Boundaries + +- No raw GitHub token persistence. +- No database migration. +- No Cloudflare call. +- No provider execution enablement. +- No AG-002 runtime wiring. +- Live GitHub API validation remains pending. diff --git a/VERSION.json b/VERSION.json index 36665a4..54caded 100644 --- a/VERSION.json +++ b/VERSION.json @@ -34,12 +34,12 @@ "migration_head": "0012_notification_engine" }, "development_snapshot": { - "date": "2026-07-24", - "status": "ready_for_controlled_live_redeploy", + "date": "2026-07-25", + "status": "update_01_implementation_verified_pr_documentation_gate", "mvp_completion_estimate": "approximately_97_percent", "controlled_live_path_estimate": "approximately_98_percent", "authoritative_status_document": "PROJECT_STATUS.md", - "full_test_suite": "579_passed_1_warning", + "full_test_suite": "609_passed", "release_gate_skip_db": "pass", "live_provider_execution_verified": false, "live_readiness_tooling": { @@ -122,6 +122,27 @@ "deployment_performed": false, "coolify_redeploy_performed": false, "branch_protection_enabled": false + }, + "update_01": { + "name": "github_repository_acquisition_analyzer_input", + "status": "implementation_verified_remote_branch_pr_documentation_gate", + "feature_branch": "feature/github-repository-acquisition-analyzer-input-v1", + "base_commit": "d2ad741c0ee48bf3eb3887a3357f6f450825c99a", + "implementation_commit": "cb796401afe8be64eb311a45e566fc315a13a00c", + "implementation_file_count": 19, + "ruff": "0.15.20_pass", + "focused_tests": "27_passed", + "full_tests": "609_passed", + "compileall": "pass", + "smoke_skip_db": "pass", + "release_gate_skip_db": "pass", + "remote_branch_verified": true, + "working_tree_clean_after_push": true, + "draft_pull_request_created": false, + "live_github_api_verified": false, + "live_cloudflare_api_verified": false, + "database_migration": false, + "provider_execution_enabled": false } }, "domain": { @@ -368,14 +389,25 @@ "updates": { "github_repository_acquisition_analyzer_input": { "version": "1.0.0", - "status": "implemented_for_verification", + "status": "implementation_verified_remote_branch_pr_documentation_gate", "baseline_commit": "d2ad741c0ee48bf3eb3887a3357f6f450825c99a", "github_app_installation_credential": "ephemeral", "commit_pinned_snapshot": true, "recursive_tree_required_complete": true, "analysis_critical_text_files_bounded": true, "database_migration": false, - "cloudflare_provider_execution_changed": false + "cloudflare_provider_execution_changed": false, + "feature_branch": "feature/github-repository-acquisition-analyzer-input-v1", + "implementation_commit": "cb796401afe8be64eb311a45e566fc315a13a00c", + "implementation_file_count": 19, + "focused_tests": "27_passed", + "full_tests": "609_passed", + "ruff": "0.15.20_pass", + "smoke_skip_db": "pass", + "release_gate_skip_db": "pass", + "remote_branch_verified": true, + "draft_pull_request_created": false, + "live_github_api_verified": false } } } diff --git a/backend/engines/auth_engine/connected_accounts_module/README.md b/backend/engines/auth_engine/connected_accounts_module/README.md index b26b7e7..eb26d35 100644 --- a/backend/engines/auth_engine/connected_accounts_module/README.md +++ b/backend/engines/auth_engine/connected_accounts_module/README.md @@ -55,3 +55,25 @@ DELETE /api/v1/connected-accounts/{provider} - Provider modules remain external API wrappers only. - Installation access tokens are generated ephemerally for approved engine calls and are never persisted. + +## Update 01 GitHub Installation Credential Contract + +```text +Repository Engine public service +→ Connected Accounts public service +→ validate connected GitHub record +→ resolve stored installation reference +→ create GitHub App JWT +→ request short-lived installation access token +→ return secret-wrapped runtime credential +``` + +Rules: + +- The stored value remains an installation reference, not a raw access token. +- The installation access token is generated only when an approved engine call requires it. +- The token is never persisted, logged, included in repository snapshots, or returned through API routes. +- Missing, disconnected, invalid, or expired GitHub connections fail closed. +- GitHub Provider receives the credential only for the bounded repository-acquisition request. + +Update 01 verification reference: `cb796401afe8be64eb311a45e566fc315a13a00c`. Live installation execution remains pending. diff --git a/backend/engines/repository_analysis_engine/README.md b/backend/engines/repository_analysis_engine/README.md index d39ad1f..01cd3a5 100644 --- a/backend/engines/repository_analysis_engine/README.md +++ b/backend/engines/repository_analysis_engine/README.md @@ -55,3 +55,21 @@ backend.engines.repository_analysis_engine.models ## Analyzer Input Contract Quick and deep analysis receive a commit-pinned, complete Repository Engine snapshot containing file paths and bounded configuration-file contents. The analysis engine does not call GitHub directly and rejects incomplete snapshots through the Repository Engine public boundary. + +## Update 01 Evidence Boundary + +The snapshot proves which commit and tree were analyzed. It does not prove live GitHub availability, Cloudflare compatibility, or production deployability. + +Update 01 provides the evidence substrate. Update 02 remains responsible for: + +- framework confidence and evidence; +- monorepo and application-root resolution; +- package-manager and lockfile correlation; +- install and build command resolution from repository scripts; +- output-directory resolution; +- environment-variable reference detection without values; +- explicit Cloudflare readiness states and blocking reasons; +- deep-analysis completion; +- recalculation reattachment through Project Engine. + +Cloudflare provider execution remains disabled. diff --git a/backend/engines/repository_engine/README.md b/backend/engines/repository_engine/README.md index e545158..48c3fc6 100644 --- a/backend/engines/repository_engine/README.md +++ b/backend/engines/repository_engine/README.md @@ -38,3 +38,46 @@ repository_metadata ## Boundary Rules Repository Engine may call the Connected Accounts public API for an ephemeral GitHub installation credential and may call GitHub Provider. It must not persist raw credentials, call Cloudflare Provider, call Deploy Pipeline, call Deploy Engine mutation services, or import Repository Analysis internals. + +## Update 01 Snapshot Contract + +A complete GitHub snapshot includes: + +```text +schema_version +default_branch +commit_sha +tree_sha +snapshot_complete +files / paths +bounded text contents +acquisition limits +skipped-file reasons +``` + +The persisted `latest_commit_sha` must match the snapshot commit SHA before the snapshot can be prepared for analysis. + +The engine rejects: + +- missing default branches; +- inaccessible repositories; +- missing installation credentials; +- truncated recursive trees; +- oversized or otherwise incomplete snapshots; +- inconsistent persisted and snapshot commit identifiers. + +## Update 01 Data Flow + +```text +Repository URL +→ Connected Accounts ephemeral credential +→ GitHub Provider metadata +→ branch head commit +→ commit tree +→ recursive inventory +→ selected bounded text +→ repository_metadata +→ RepositoryAnalysisInput +``` + +No raw credential crosses into persistence or Repository Analysis. diff --git a/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md b/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md index ac6a504..0976b58 100644 --- a/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md +++ b/docs/architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md @@ -1,75 +1,335 @@ # GitHub Repository Acquisition and Analyzer Input Specification -**Version:** 1.0.0
-**Status:** Implemented for Verification
+**Version:** 1.1.0
+**Status:** Implementation Verified; Draft PR Documentation Gate
**Product:** YGIT
-**Update:** Update 1 — GitHub Repository Acquisition + Analyzer Input
-**Baseline Commit:** `d2ad741c0ee48bf3eb3887a3357f6f450825c99a` +**Update:** Update 01 — GitHub Repository Acquisition + Analyzer Input
+**Baseline Commit:** `d2ad741c0ee48bf3eb3887a3357f6f450825c99a`
+**Implementation Commit:** `cb796401afe8be64eb311a45e566fc315a13a00c`
+**Feature Branch:** `feature/github-repository-acquisition-analyzer-input-v1` -## Purpose +## 1. Purpose -This update establishes the authenticated, commit-pinned GitHub repository acquisition path required before strengthening framework and Cloudflare readiness analysis. +Update 01 establishes the authenticated, commit-pinned GitHub repository acquisition path required by Repository Analysis. It replaces the metadata-only input gap with a complete recursive path inventory and bounded analysis-critical text evidence. -## Runtime Flow +This update is an evidence-acquisition update. It does not implement the stronger analyzer rules or Cloudflare readiness decisions assigned to Update 02. + +## 2. Goals + +- Use the connected GitHub App installation for repository reads. +- Resolve the repository default branch to an exact commit SHA. +- Resolve the exact Git tree associated with that commit. +- Acquire a complete recursive file inventory within approved bounds. +- Acquire bounded UTF-8 contents for selected analysis-critical files. +- Persist a normalized, versioned, commit-pinned snapshot. +- Prepare Repository Analysis input through the Repository Engine public boundary. +- Fail closed when repository evidence is incomplete or credentials are unavailable. +- Preserve raw-token secrecy and engine/provider boundaries. + +## 3. Non-Goals + +- GitHub write operations. +- GitHub webhook processing. +- GitHub OAuth user login or repository authorization. +- Strong framework or monorepo detection. +- Deep-analysis worker completion. +- Recalculation-time Project reattachment. +- Cloudflare readiness expansion. +- Cloudflare provider execution. +- Database migration. +- UI-V2 changes. +- AG-002 runtime wiring. +- Production deployment. + +## 4. Ownership and Boundaries + +| Responsibility | Owner | +|---|---| +| Stored GitHub connection and installation reference | Connected Accounts Module | +| Ephemeral installation access token acquisition | Connected Accounts Module | +| GitHub HTTP requests and payload normalization | GitHub Provider | +| Repository identity, metadata, snapshot persistence, and analyzer-input preparation | Repository Engine | +| Framework/build/readiness analysis | Repository Analysis Engine | +| Project association and lifecycle | Project Engine | +| Cloudflare deployment | Deploy Engine / Deploy Pipeline / Cloudflare Provider | + +Forbidden shortcuts: + +- API routes must not call GitHub Provider directly. +- Repository Analysis must not call GitHub Provider directly. +- GitHub Provider must not contain YGIT project or readiness business logic. +- Repository Engine must not persist raw installation credentials. +- Update 01 must not enable Cloudflare provider execution. + +## 5. End-to-End Runtime Flow ```text -Project Engine -→ Repository Engine public API -→ Connected Accounts public API -→ ephemeral GitHub App installation access token -→ GitHub Provider -→ repository metadata -→ default branch head commit -→ exact Git tree -→ bounded analysis-critical text files -→ Repository Engine persistence -→ Repository Analysis Engine input +Project or Repository request + ↓ +Repository Engine public API + ↓ +Connected Accounts public API + ↓ +validate connected GitHub installation reference + ↓ +create GitHub App JWT + ↓ +request short-lived installation access token + ↓ +GitHub Provider repository metadata + ↓ +resolve default branch + ↓ +resolve branch head commit SHA + ↓ +resolve commit tree SHA + ↓ +request complete recursive Git tree + ↓ +select analysis-critical text blobs + ↓ +fetch bounded UTF-8 content + ↓ +normalize versioned snapshot + ↓ +persist repository metadata and snapshot + ↓ +prepare RepositoryAnalysisInput + ↓ +Repository Analysis quick flow ``` -## Security Contract +## 6. Credential Lifecycle -- The GitHub App installation reference remains stored as a safe reference. -- Installation access tokens are generated server-side only when needed. -- Raw GitHub tokens are never stored in `repository_metadata`. -- Raw GitHub tokens are never returned by API routes. -- Repository snapshots contain only repository paths, Git object identifiers, bounded text content, and safe metadata. -- Provider execution for Cloudflare remains disabled. +The database stores a safe installation reference. It does not store the installation access token used for acquisition. -## Snapshot Contract +```text +stored installation reference +→ server-generated app JWT +→ short-lived installation token +→ one bounded repository-acquisition operation +→ token discarded +``` -A complete snapshot contains: +The token must not appear in: -- schema version; -- default branch; -- exact commit SHA; -- exact root tree SHA; -- non-truncated recursive file inventory; -- bounded UTF-8 contents for analysis-critical configuration files; -- acquisition limits and skipped-file reasons. +- database fields; +- repository snapshots; +- analysis results; +- job payloads; +- structured logs; +- exception messages; +- API responses; +- test snapshots; +- documentation. + +## 7. GitHub Read Sequence + +The provider performs read-only operations sufficient to establish immutable repository evidence: + +1. Read repository metadata. +2. Read the default branch reference. +3. Resolve the branch head commit. +4. Resolve the commit's root tree. +5. Request the recursive tree. +6. Reject truncated or otherwise incomplete tree responses. +7. Select approved analysis-critical text paths. +8. Read bounded blob contents. +9. Return normalized safe metadata to Repository Engine. + +No repository mutation endpoint is used. + +## 8. Snapshot Contract + +A valid snapshot contains: + +```text +schema_version +default_branch +commit_sha +tree_sha +snapshot_complete=true +files / paths +bounded text contents +acquisition_limits +skipped_files +``` + +Required invariants: + +- `latest_commit_sha` equals snapshot `commit_sha`. +- The root `tree_sha` belongs to the resolved commit. +- The path inventory is complete within the approved repository limit. +- `snapshot_complete` is true. +- Every included text value is bounded and UTF-8-decodable. +- Skipped content records a safe reason without exposing secrets. +- No credential or token field is present. + +## 9. Analysis-Critical Text Selection + +Update 01 collects only files useful to later deterministic analysis, subject to configured bounds. Typical categories include: + +- package manifests; +- lockfiles; +- framework configuration; +- build configuration; +- static-site configuration; +- Cloudflare configuration; +- workspace/monorepo configuration; +- dependency and runtime metadata. + +Binary files, oversized files, sensitive filenames, unsupported encodings, and non-selected source files are not copied into bounded analysis text input. + +The complete path inventory remains available even when a file's contents are not selected. + +## 10. Fail-Closed Error Model + +| Condition | Required behavior | +|---|---| +| GitHub connection missing or disconnected | Stop through Connected Accounts error boundary | +| Installation reference invalid or expired | Stop; do not fall back to anonymous access for private evidence | +| Repository unavailable to installation | Return normalized repository/provider error | +| Default branch missing | Return `REPOSITORY_DEFAULT_BRANCH_MISSING` | +| Recursive tree truncated or oversized | Return `REPOSITORY_SNAPSHOT_INCOMPLETE` | +| Snapshot commit inconsistent with persisted commit | Reject analyzer input | +| Provider temporarily unavailable | Return normalized provider-unavailable error | +| Selected blob invalid or exceeds limits | Record safe skip reason or fail when required by completeness contract | + +Partial evidence must not be treated as deploy-ready evidence. -Truncated GitHub tree responses fail closed with `REPOSITORY_SNAPSHOT_INCOMPLETE`. Partial repository analysis is not accepted. +## 11. Persistence and Data Flow -## Analyzer Input Contract +No migration is required. + +Existing fields are used: + +- `repository_metadata.latest_commit_sha`; +- `repository_metadata.file_tree_snapshot`; +- safe provider metadata. + +Repository Engine owns persistence. GitHub Provider returns normalized data but does not write the database. + +## 12. Analyzer Input Contract Repository Analysis receives: -- repository identity; +- repository ID and identity; +- provider and normalized repository URL; +- default branch; +- visibility; - exact commit SHA; - exact tree SHA; -- snapshot version; -- `snapshot_complete=true`; -- complete file-path inventory; -- bounded analysis-critical file contents. +- snapshot schema version; +- completeness flag; +- complete path inventory; +- bounded analysis-critical text contents. + +Quick Analysis remains compatible with its current path extraction logic. Update 02 will consume the text evidence for stronger decisions. + +## 13. Project Integration + +Project creation may attach the repository and initial analysis result through existing public boundaries. + +Deploy readiness remains fail closed: + +```text +stored analysis deploy_ready=true +→ deployment may proceed to later gates + +unknown, incomplete, unsupported, or blocked evidence +→ deployment remains blocked +``` + +Recalculation-time reattachment of the newest analysis is not completed by Update 01. + +## 14. API Impact + +No new public API route is required. Existing repository metadata routes use the stronger internal acquisition path. + +Raw credentials and private file contents are not added to public response contracts. + +## 15. Database, Configuration, and Deployment Impact + +| Area | Impact | +|---|---| +| Database migration | None | +| New table | None | +| Environment mutation | None | +| Provider execution mode | Remains `disabled` | +| Cloudflare API execution | None | +| Production deployment | None | +| UI-V2 | None | + +## 16. Security Review + +- Installation tokens are ephemeral and secret-wrapped. +- Token references and tokens have separate types and lifecycles. +- Private repository contents are not logged. +- Snapshot content is bounded. +- Sensitive filenames are excluded from text acquisition. +- GitHub errors are normalized before crossing provider boundaries. +- Cloudflare credentials are not involved. +- GitHub OAuth variables remain forbidden. +- Webhook capability remains default-disabled. + +## 17. Verification Evidence + +```text +Implementation branch: feature/github-repository-acquisition-analyzer-input-v1 +Implementation commit: cb796401afe8be64eb311a45e566fc315a13a00c +Base commit: d2ad741c0ee48bf3eb3887a3357f6f450825c99a +Implementation file count: 19 +Ruff version: 0.15.20 +Ruff result: PASS +Focused tests: 27 passed +Full test suite: 609 passed +Compileall: PASS +Smoke --skip-db: PASS +Release gate --skip-db: PASS +Remote feature-branch SHA match: PASS +Working tree after push: CLEAN +``` + +The evidence proves local deterministic contracts and remote branch integrity. It does not prove live GitHub or Cloudflare behavior. + +## 18. Rollout Gate + +1. Complete documentation commit. +2. Push the final feature-branch head. +3. Create the detailed Draft PR. +4. Require successful Backend CI on the final head. +5. Review exact scope and security boundaries. +6. Merge only after explicit authorization. +7. Perform controlled live GitHub validation after merge. +8. Keep Cloudflare provider execution disabled until its separate authorization gate. + +## 19. Rollback + +Before merge, rollback is branch-only: + +- close the Draft PR; +- preserve evidence; +- do not merge the feature branch. + +After merge, rollback requires a reviewed revert of the merge commit. No database rollback is required because Update 01 adds no migration. -Update 1 does not expand framework detection rules. Strong framework/build/output/readiness detection belongs to Update 2. +## 20. Update 02 Handoff -## Database Impact +Update 02 will use the acquired evidence for: -No migration is required. Existing `repository_metadata.latest_commit_sha`, `file_tree_snapshot`, and `metadata` fields store the normalized acquisition result. +- framework and language confidence; +- monorepo/application-root detection; +- package-manager and lockfile correlation; +- install/build command resolution; +- output-directory resolution; +- environment-reference detection; +- explicit Cloudflare readiness and blocking reasons; +- deep-analysis completion; +- Project reattachment after recalculation. -## Provider Impact +## Revision History -- GitHub App read access is used. -- No GitHub write operation is added. -- No Cloudflare API execution is added. -- `WORKER_PROVIDER_EXECUTION_MODE=disabled` remains unchanged. +| Date | Version | Summary | +|---|---|---| +| 2026-07-25 | 1.1.0 | Added complete implementation, security, failure, verification, rollout, rollback, and Update 02 handoff contracts | +| 2026-07-25 | 1.0.0 | Initial Update 01 implementation specification | diff --git a/docs/updates/README.md b/docs/updates/README.md new file mode 100644 index 0000000..55aab3b --- /dev/null +++ b/docs/updates/README.md @@ -0,0 +1,14 @@ +# YGIT Versioned Update Records + +This directory contains versioned implementation-completion and release-control records. + +## Update 01 + +**GitHub Repository Acquisition + Analyzer Input** + +- [Completion Record](UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md) +- [Verification Evidence](UPDATE_01_VERIFICATION_EVIDENCE.md) +- [Draft PR Checklist](UPDATE_01_PR_CHECKLIST.md) +- [Architecture Specification](../architecture/GITHUB_REPOSITORY_ACQUISITION_ANALYZER_INPUT_SPECIFICATION.md) + +Update 01 is implementation-verified and remote-branch-verified. Live provider validation, merge, production redeploy, and Update 02 remain separate controlled gates. diff --git a/docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md b/docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md new file mode 100644 index 0000000..ceef83a --- /dev/null +++ b/docs/updates/UPDATE_01_GITHUB_REPOSITORY_ACQUISITION_COMPLETION.md @@ -0,0 +1,81 @@ +# Update 01 Completion Record + +**Update:** GitHub Repository Acquisition + Analyzer Input
+**Status:** Implementation Verified; Draft PR Documentation Gate
+**Date:** 2026-07-25
+**Baseline:** `d2ad741c0ee48bf3eb3887a3357f6f450825c99a`
+**Implementation Branch:** `feature/github-repository-acquisition-analyzer-input-v1`
+**Implementation Commit:** `cb796401afe8be64eb311a45e566fc315a13a00c` + +## Outcome + +Update 01 replaces metadata-only repository acquisition with an installation-authenticated, commit-pinned, complete snapshot path suitable for Repository Analysis input. + +## Delivered Capabilities + +- Connected Accounts acquisition of a short-lived GitHub installation credential. +- Installation-authenticated repository metadata. +- Exact default-branch head commit. +- Exact commit tree. +- Complete recursive path inventory within approved limits. +- Bounded analysis-critical UTF-8 contents. +- Versioned normalized snapshot. +- Repository Analysis input through the Repository Engine public boundary. +- Fail-closed incomplete-snapshot behavior. +- Token non-persistence and secret-safe boundaries. + +## Authoritative Flow + +```text +Connected GitHub installation +→ ephemeral installation access token +→ repository metadata +→ default branch +→ exact commit +→ exact tree +→ complete recursive inventory +→ bounded selected text +→ Repository Engine persistence +→ Repository Analysis input +``` + +## Scope + +Implementation commit `cb796401afe8be64eb311a45e566fc315a13a00c` contains 19 files relative to `d2ad741c0ee48bf3eb3887a3357f6f450825c99a`. + +The documentation-closure patch updates only Markdown and `VERSION.json`; it does not change implementation logic. + +## Safety Boundaries + +- No raw GitHub token persistence. +- No GitHub write operation. +- No migration. +- No Cloudflare API operation. +- No provider-execution enablement. +- No UI-V2 change. +- No production deployment. +- No AG-002 runtime wiring. + +## Remaining Gates + +1. Documentation commit and final branch push. +2. Detailed Draft PR creation. +3. Backend CI on the final PR head. +4. Exact-scope and security review. +5. Explicit merge authorization. +6. Controlled live GitHub validation. +7. Update 02 analyzer/readiness implementation. +8. Separately authorized controlled Cloudflare deployment. + +## Completion Decision + +```text +Implementation: COMPLETE +Local deterministic verification: PASS +Remote implementation branch: VERIFIED +Documentation closure: INCLUDED IN THIS PATCH +Draft PR: CREATED ONLY BY THE FINAL PR SCRIPT +Merge: NOT AUTHORIZED +Production redeploy: NOT AUTHORIZED +Provider execution: DISABLED +``` diff --git a/docs/updates/UPDATE_01_PR_CHECKLIST.md b/docs/updates/UPDATE_01_PR_CHECKLIST.md new file mode 100644 index 0000000..0f5831d --- /dev/null +++ b/docs/updates/UPDATE_01_PR_CHECKLIST.md @@ -0,0 +1,75 @@ +# Update 01 Draft PR Checklist + +**Base:** `main`
+**Head:** `feature/github-repository-acquisition-analyzer-input-v1`
+**Implementation Commit:** `cb796401afe8be64eb311a45e566fc315a13a00c` + +## Before PR Creation + +- [ ] Documentation patch applied and verified. +- [ ] Documentation commit pushed. +- [ ] Local working tree clean. +- [ ] Remote branch SHA matches local head. +- [ ] `origin/main` remains `d2ad741c0ee48bf3eb3887a3357f6f450825c99a`. +- [ ] Temporary-index whitespace gate passes. +- [ ] `VERSION.json` parses. +- [ ] Full suite, smoke, and release gate pass. +- [ ] No source-code file is changed by the documentation patch. + +## PR Metadata + +**Title** + +```text +feat: acquire GitHub repositories for analysis +``` + +**Type** + +```text +Draft Pull Request +``` + +**Base / Head** + +```text +main +← +feature/github-repository-acquisition-analyzer-input-v1 +``` + +## Required PR Sections + +- Summary +- Motivation and previous gap +- Architecture and ownership +- Runtime data flow +- Credential lifecycle +- Snapshot and analyzer-input contracts +- Error and fail-closed behavior +- Security review +- Database/configuration/deployment impact +- Exact implementation scope +- Verification evidence +- Live validation not executed +- Rollout and rollback +- Update 02 handoff +- Reviewer checklist + +## CI and Review Gate + +- [ ] `Backend CI / Validate` succeeds on the final PR head. +- [ ] PR head SHA equals the remote feature-branch SHA. +- [ ] Changed-file scope is reviewed. +- [ ] No migration is present. +- [ ] No Cloudflare provider execution is enabled. +- [ ] No raw credential or secret is present. +- [ ] No provider call bypasses engine/public boundaries. +- [ ] Truncated snapshots fail closed. +- [ ] Live validation limitations are explicit. + +## Ready and Merge Gate + +The PR must remain Draft until the project owner explicitly authorizes Ready status. + +Merge requires separate explicit authorization after CI and review. This patch does not mark the PR ready, merge it, redeploy production, or enable provider execution. diff --git a/docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md b/docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md new file mode 100644 index 0000000..d0e0c45 --- /dev/null +++ b/docs/updates/UPDATE_01_VERIFICATION_EVIDENCE.md @@ -0,0 +1,58 @@ +# Update 01 Verification Evidence + +**Date:** 2026-07-25
+**Implementation Commit:** `cb796401afe8be64eb311a45e566fc315a13a00c`
+**Branch:** `feature/github-repository-acquisition-analyzer-input-v1` + +## Verified Gates + +| Gate | Result | +|---|---| +| Exact implementation scope | 19 files | +| Ruff | `0.15.20` — PASS | +| Update 01 focused tests | 27 passed | +| Full test suite | 609 passed | +| Compileall | PASS | +| Smoke test with database skipped | PASS | +| Release gate with database skipped | PASS | +| Remote feature-branch SHA | Matches local commit | +| Working tree after push | Clean | +| Database migration | None | +| Provider execution enablement | None | + +## Commands Used + +```powershell +python -m compileall -q backend scripts +ruff check +python -m pytest -q backend/tests/test_github_repository_acquisition_analyzer_input.py backend/tests/test_repository_analysis_current_state_audit.py +python -m pytest -q +python scripts/smoke_test.py --skip-db +python scripts/release_gate.py --skip-db +``` + +## Security Evidence + +- No raw installation token is returned by APIs. +- No raw installation token is persisted in repository metadata. +- Repository Analysis does not import or call GitHub Provider. +- API routes do not import or call GitHub Provider directly. +- Truncated recursive trees fail closed. +- Commit and snapshot identifiers must agree. +- Cloudflare provider mode remains disabled. + +## Not Executed + +The following require separate controlled environments and are not represented by local test success: + +- live GitHub App installation-token acquisition; +- live private-repository recursive-tree acquisition; +- live PostgreSQL persistence verification; +- live Redis worker loop; +- live Cloudflare API execution; +- real Cloudflare Pages deployment; +- production redeploy. + +## Evidence Interpretation + +Local and CI tests prove deterministic contracts, architecture boundaries, error normalization, snapshot completeness rules, and secret-safe data flow. They do not replace live provider validation.