diff --git a/docs/architecture.md b/docs/architecture.md index 606400a..baa6bf3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -33,11 +33,11 @@ CloudEmu follows a three-layer architecture that separates portable API concerns ### Layer 1: Portable API -The top layer lives in service-specific packages (`storage/`, `compute/`, `database/`, etc.). Each portable API type wraps a driver with cross-cutting concerns. For example, `storage.Bucket` wraps `driver.Bucket` and adds call recording, metrics collection, rate limiting, error injection, and simulated latency to every operation. This layer is provider-agnostic -- the same `storage.Bucket` works with S3, Blob Storage, or GCS. +The top layer lives in service-specific packages (`services/storage/`, `services/compute/`, `services/database/`, etc.). Each portable API type wraps a driver with cross-cutting concerns. For example, `storage.Bucket` wraps `driver.Bucket` and adds call recording, metrics collection, rate limiting, error injection, and simulated latency to every operation. This layer is provider-agnostic -- the same `storage.Bucket` works with S3, Blob Storage, or GCS. ### Layer 2: Driver Interfaces -Each service defines a minimal Go interface in `/driver/driver.go`. These interfaces specify the operations that every provider must implement. For example, `storage/driver/driver.go` defines the `Bucket` interface with methods like `CreateBucket`, `PutObject`, `GetObject`, etc. Driver interfaces use plain Go types (no cloud SDK dependencies). +Each service defines a minimal Go interface in `/driver/driver.go`. These interfaces specify the operations that every provider must implement. For example, `services/storage/driver/driver.go` defines the `Bucket` interface with methods like `CreateBucket`, `PutObject`, `GetObject`, etc. Driver interfaces use plain Go types (no cloud SDK dependencies). ### Layer 3: Provider Implementations @@ -47,7 +47,7 @@ The bottom layer contains the actual mock implementations for each cloud provide Sitting above Layer 2 (driver interfaces) are **cross-service engines** that consume driver interfaces directly without going through the portable API. They're peers of each other, not layers in the three-layer stack. Two exist today: -- `topology/` -- reads from compute, networking, and DNS drivers to simulate real network connectivity (`CanConnect`, `TraceRoute`, `Resolve`, security-group and NACL evaluation). See [topology.md](topology.md). +- `features/topology/` -- reads from compute, networking, and DNS drivers to simulate real network connectivity (`CanConnect`, `TraceRoute`, `Resolve`, security-group and NACL evaluation). See [topology.md](topology.md). - `server/` -- exposes driver interfaces over HTTP in each cloud's native SDK wire format, so real `aws-sdk-go-v2`, `azure-sdk-for-go`, and `cloud.google.com/go` clients work against CloudEmu by only changing the endpoint. Covers Storage, Compute, Database, Networking, Monitoring, Serverless, and Message Queue across all 3 providers (AWS S3/EC2/DynamoDB/Lambda/SQS/CloudWatch + sibling Azure ARM and GCP REST handlers). Uses a pluggable `Handler` registry so new services drop in as self-contained packages without touching the core. See [sdk-server.md](sdk-server.md). Both engines depend only on Layer 2 interfaces -- never on concrete provider types -- so they work uniformly across AWS, Azure, and GCP backends. @@ -126,53 +126,25 @@ errors/ internal/ memstore/ # Generic Store[V] idgen/ # ID generators (ARNs, Azure IDs, GCP IDs) -statemachine/ - machine.go # Generic FSM - transitions.go # Transition map -pagination/ # Generic Paginate[T] -recorder/ # Call recording for assertions -metrics/ # In-memory metrics collection -ratelimit/ # Token bucket rate limiter -inject/ # Error injection (policies + injector) -cost/ - cost.go # Cost tracker with default rates -storage/ - storage.go # Portable storage API - driver/driver.go # Bucket interface -compute/ - driver/driver.go # Compute interface -database/ - driver/driver.go # Database interface -serverless/ - driver/driver.go # Serverless interface -networking/ - driver/driver.go # Networking interface -monitoring/ - driver/driver.go # Monitoring interface -iam/ - driver/driver.go # IAM interface -dns/ - driver/driver.go # DNS interface -loadbalancer/ - driver/driver.go # LoadBalancer interface -messagequeue/ - driver/driver.go # MessageQueue interface -cache/ - driver/driver.go # Cache interface -secrets/ - driver/driver.go # Secrets interface -logging/ - driver/driver.go # Logging interface -notification/ - driver/driver.go # Notification interface -containerregistry/ - driver/driver.go # ContainerRegistry interface -eventbus/ - driver/driver.go # EventBus interface -relationaldb/ - driver/driver.go # Relational DB interface (RDS, Aurora, Neptune, - # DocumentDB, Redshift, Azure SQL, Postgres/MySQL - # Flexible Server, Cloud SQL) + statemachine/ # Generic FSM + pagination/ # Generic Paginate[T] +services/ # emulated cloud services (portable API + driver interface) + storage/ + storage.go # Portable storage API + driver/driver.go # Bucket interface + compute/ database/ relationaldb/ serverless/ networking/ monitoring/ + iam/ dns/ loadbalancer/ messagequeue/ cache/ secrets/ logging/ + notification/ eventbus/ containerregistry/ kubernetes/ resourcediscovery/ + bedrock/ sagemaker/ vertexai/ databricks/ azureai/ azuresearch/ + parameterstore/ tablestorage/ cost/ + # each: .go (portable API) + driver/ (interface) +features/ # cross-cutting capabilities you wrap drivers with + chaos/ # fault / latency / throttle injection + recorder/ # call recording for assertions + metrics/ # in-memory metrics collection + ratelimit/ # token-bucket rate limiter + inject/ # error injection (policies + injector) + topology/ # network reachability (CanConnect / TraceRoute / Resolve) providers/ aws/ aws.go # AWS factory (wires all services) diff --git a/docs/features.md b/docs/features.md index c270ccd..2ef78de 100644 --- a/docs/features.md +++ b/docs/features.md @@ -374,7 +374,7 @@ clock.Set(time.Date(2025, 1, 2, 0, 0, 0, 0, time.UTC)) ## 10. Cross-Service Resource Discovery -CloudEmu ships a cross-service inventory engine (`resourcediscovery/`) that walks every service driver a provider holds and returns a single normalized view of what exists. It sits next to the `topology/` engine as a peer of the portable API — it owns no state, constructs from driver interfaces, and is purely query-driven. +CloudEmu ships a cross-service inventory engine (`services/resourcediscovery/`) that walks every service driver a provider holds and returns a single normalized view of what exists. It sits next to the `features/topology/` engine as a peer of the portable API — it owns no state, constructs from driver interfaces, and is purely query-driven. The engine is the foundation for three SDK-compat handlers that speak the real cloud inventory APIs: **AWS Resource Explorer 2 + Resource Groups Tagging API**, **Azure Resource Graph**, and **GCP Cloud Asset Inventory**. A tag set through any one of those paths is immediately visible through the others, and through the engine's own `SearchByTag`. diff --git a/docs/services.md b/docs/services.md index 89f92a9..67fc7d3 100644 --- a/docs/services.md +++ b/docs/services.md @@ -23,7 +23,7 @@ This document lists every service and operation available in CloudEmu across all | 15 | Container Registry | `ecr` | `acr` | `artifactregistry` | | 16 | Event Bus | `eventbridge` | `eventgrid` | `eventarc` | | 17 | Relational Database | `rds` (+ Aurora/Neptune/DocumentDB engines), `redshift` | `azuresql`, `postgresflex`, `mysqlflex` | `cloudsql` | -| 18 | Kubernetes | `eks` + shared `kubernetes/` | `aks` + shared `kubernetes/` | `gke` + shared `kubernetes/` | +| 18 | Kubernetes | `eks` + shared `services/kubernetes/` | `aks` + shared `services/kubernetes/` | `gke` + shared `services/kubernetes/` | | 19 | Resource Discovery | `resourceexplorer2` + `resourcegroupstaggingapi` | `resourcegraph` | `cloudasset` | | 20 | Generative AI | `bedrock` (+ `bedrock-runtime`) | — | — | | 21 | Databricks | — | `databricks` | — | @@ -34,7 +34,7 @@ This document lists every service and operation available in CloudEmu across all ## 1. Storage -**Driver interface:** `storage/driver/driver.go` +**Driver interface:** `services/storage/driver/driver.go` **AWS:** S3 | **Azure:** Blob Storage | **GCP:** GCS ### Bucket Operations @@ -132,7 +132,7 @@ This document lists every service and operation available in CloudEmu across all ## 2. Compute -**Driver interface:** `compute/driver/driver.go` +**Driver interface:** `services/compute/driver/driver.go` **AWS:** EC2 | **Azure:** Virtual Machines | **GCP:** GCE ### Instance Operations @@ -223,7 +223,7 @@ This document lists every service and operation available in CloudEmu across all ## 3. Database -**Driver interface:** `database/driver/driver.go` +**Driver interface:** `services/database/driver/driver.go` **AWS:** DynamoDB | **Azure:** Cosmos DB | **GCP:** Firestore ### Table Operations @@ -288,7 +288,7 @@ This document lists every service and operation available in CloudEmu across all ## 4. Serverless -**Driver interface:** `serverless/driver/driver.go` +**Driver interface:** `services/serverless/driver/driver.go` **AWS:** Lambda | **Azure:** Functions | **GCP:** Cloud Functions ### Function Operations @@ -354,7 +354,7 @@ This document lists every service and operation available in CloudEmu across all ## 5. Networking -**Driver interface:** `networking/driver/driver.go` +**Driver interface:** `services/networking/driver/driver.go` **AWS:** VPC | **Azure:** VNet | **GCP:** GCP VPC ### VPC Operations @@ -474,7 +474,7 @@ This document lists every service and operation available in CloudEmu across all ## 6. Monitoring -**Driver interface:** `monitoring/driver/driver.go` +**Driver interface:** `services/monitoring/driver/driver.go` **AWS:** CloudWatch | **Azure:** Azure Monitor | **GCP:** Cloud Monitoring ### Metric Operations @@ -515,7 +515,7 @@ This document lists every service and operation available in CloudEmu across all ## 7. IAM -**Driver interface:** `iam/driver/driver.go` +**Driver interface:** `services/iam/driver/driver.go` **AWS:** IAM | **Azure:** Azure IAM | **GCP:** GCP IAM ### Users @@ -599,7 +599,7 @@ This document lists every service and operation available in CloudEmu across all ## 8. DNS -**Driver interface:** `dns/driver/driver.go` +**Driver interface:** `services/dns/driver/driver.go` **AWS:** Route 53 | **Azure:** Azure DNS | **GCP:** Cloud DNS ### Zone Operations @@ -638,7 +638,7 @@ This document lists every service and operation available in CloudEmu across all ## 9. Load Balancer -**Driver interface:** `loadbalancer/driver/driver.go` +**Driver interface:** `services/loadbalancer/driver/driver.go` **AWS:** ELB | **Azure:** Azure LB | **GCP:** GCP LB ### Load Balancer Operations @@ -696,7 +696,7 @@ This document lists every service and operation available in CloudEmu across all ## 10. Message Queue -**Driver interface:** `messagequeue/driver/driver.go` +**Driver interface:** `services/messagequeue/driver/driver.go` **AWS:** SQS | **Azure:** Service Bus | **GCP:** Pub/Sub ### Queue Operations @@ -749,7 +749,7 @@ This document lists every service and operation available in CloudEmu across all ## 11. Cache -**Driver interface:** `cache/driver/driver.go` +**Driver interface:** `services/cache/driver/driver.go` **AWS:** ElastiCache | **Azure:** Azure Cache | **GCP:** Memorystore ### Cache Instance Operations @@ -794,7 +794,7 @@ This document lists every service and operation available in CloudEmu across all ## 12. Secrets -**Driver interface:** `secrets/driver/driver.go` +**Driver interface:** `services/secrets/driver/driver.go` **AWS:** Secrets Manager | **Azure:** Key Vault | **GCP:** Secret Manager ### Secret Operations @@ -820,7 +820,7 @@ This document lists every service and operation available in CloudEmu across all ## 13. Logging -**Driver interface:** `logging/driver/driver.go` +**Driver interface:** `services/logging/driver/driver.go` **AWS:** CloudWatch Logs | **Azure:** Log Analytics | **GCP:** Cloud Logging ### Log Group Operations @@ -862,7 +862,7 @@ This document lists every service and operation available in CloudEmu across all ## 14. Notification -**Driver interface:** `notification/driver/driver.go` +**Driver interface:** `services/notification/driver/driver.go` **AWS:** SNS | **Azure:** Notification Hubs | **GCP:** FCM ### Topic Operations @@ -894,7 +894,7 @@ This document lists every service and operation available in CloudEmu across all ## 15. Container Registry -**Driver interface:** `containerregistry/driver/driver.go` +**Driver interface:** `services/containerregistry/driver/driver.go` **AWS:** ECR | **Azure:** ACR | **GCP:** Artifact Registry ### Repository Management @@ -937,7 +937,7 @@ This document lists every service and operation available in CloudEmu across all ## 16. Event Bus -**Driver interface:** `eventbus/driver/driver.go` +**Driver interface:** `services/eventbus/driver/driver.go` **AWS:** EventBridge | **Azure:** Event Grid | **GCP:** Eventarc ### Bus Management @@ -986,7 +986,7 @@ This document lists every service and operation available in CloudEmu across all ## 17. Relational Database -**Driver interface:** `relationaldb/driver/driver.go` +**Driver interface:** `services/relationaldb/driver/driver.go` **AWS:** `rds` (covers Aurora, Neptune, and DocumentDB engines), `redshift` | **Azure:** `azuresql`, `postgresflex`, `mysqlflex` | **GCP:** `cloudsql` A single portable interface backs every RDBMS handler. Engine selection (MySQL / PostgreSQL / Aurora / Neptune / DocumentDB / Redshift / Cloud SQL / Azure SQL / …) is a field on the input config, not a separate driver. @@ -1039,7 +1039,7 @@ A single portable interface backs every RDBMS handler. Engine selection (MySQL / ## 18. Kubernetes **Control plane:** AWS `eks`, Azure `aks`, GCP `gke` — cluster, node-pool, and addon / Fargate-profile / maintenance-config lifecycle, driven by the real cloud SDKs. -**Data plane:** shared `kubernetes/` package — an in-memory Kubernetes API server registered by every cluster across all three providers. Kubeconfigs returned by the control plane point at `/k8s/` so `client-go` and `kubectl` operate end-to-end. +**Data plane:** shared `services/kubernetes/` package — an in-memory Kubernetes API server registered by every cluster across all three providers. Kubeconfigs returned by the control plane point at `/k8s/` so `client-go` and `kubectl` operate end-to-end. Each provider exposes its native control-plane API. The data plane has no portable driver — clients connect via the kubeconfig the control plane hands out, then talk standard Kubernetes REST. @@ -1075,7 +1075,7 @@ Operations: **18** Operations: **26** -### Data plane (`kubernetes/`) +### Data plane (`services/kubernetes/`) Shared in-memory K8s API server registered by every cluster from any provider. URL: `/k8s//...`. Anonymous auth (kubeconfigs use `insecure-skip-tls-verify: true`). @@ -1100,11 +1100,11 @@ Shared in-memory K8s API server registered by every cluster from any provider. U ## 19. Resource Discovery -**Engine:** `resourcediscovery/` — a cross-service inventory engine that walks the Compute, Networking, Storage, Database, and Serverless drivers of any provider and returns a normalized `Resource` view (provider, service, type, ID, ARN/URN, region, tags, created-at). Auto-wired by every provider factory and exposed as `Provider.ResourceDiscovery`. +**Engine:** `services/resourcediscovery/` — a cross-service inventory engine that walks the Compute, Networking, Storage, Database, and Serverless drivers of any provider and returns a normalized `Resource` view (provider, service, type, ID, ARN/URN, region, tags, created-at). Auto-wired by every provider factory and exposed as `Provider.ResourceDiscovery`. **SDK-compat handlers:** AWS Resource Explorer Two + Resource Groups Tagging API, Azure Resource Graph, and GCP Cloud Asset Inventory. All three sit on top of the same engine, so a tag written through any one path is visible through the others. -### Engine (`resourcediscovery/`) +### Engine (`services/resourcediscovery/`) | Operation | Signature | |-----------|-----------| @@ -1155,7 +1155,7 @@ Operations: **Engine 8** + **AWS Resource Explorer 1** + **AWS Resource Groups T ## 20. Generative AI -**Driver interface:** `bedrock/driver/driver.go` +**Driver interface:** `services/bedrock/driver/driver.go` **AWS:** `bedrock` (+ `bedrock-runtime`) | **Azure:** — | **GCP:** — AWS-only. Backs the real `aws-sdk-go-v2/service/bedrock` and `.../bedrockruntime` clients against the in-memory backend. @@ -1223,7 +1223,7 @@ AWS-only. Backs the real `aws-sdk-go-v2/service/bedrock` and `.../bedrockruntime ## 21. Databricks -**Driver interfaces:** `databricks/driver/driver.go` (control plane), `databricks/driver/dataplane.go` (data plane) +**Driver interfaces:** `services/databricks/driver/driver.go` (control plane), `services/databricks/driver/dataplane.go` (data plane) **AWS:** — | **Azure:** `databricks` | **GCP:** — Azure-only. The control plane backs the real `armdatabricks` SDK; the data plane backs the real `databricks-sdk-go` WorkspaceClient. The SDK-compat-only workspace families (secrets, tokens, git credentials, repos, DBFS, workspace files, SQL warehouses, pipelines, serving endpoints, SCIM identity, Unity Catalog) have no portable Go API — see [sdk-server.md](sdk-server.md). @@ -1328,7 +1328,7 @@ Azure-only. The control plane backs the real `armdatabricks` SDK; the data plane ### AWS — SageMaker AI -**Driver interface:** `sagemaker/driver/driver.go` (control plane + `Runtime`) +**Driver interface:** `services/sagemaker/driver/driver.go` (control plane + `Runtime`) The control plane speaks awsJson1_1 (`X-Amz-Target: SageMaker.*`); the runtime speaks restJson1 (`POST /endpoints/{name}/invocations`). Asynchronous jobs complete synchronously @@ -1354,7 +1354,7 @@ clients. **Total: 121 operations.** ### GCP — Vertex AI -**Driver interface:** `vertexai/driver/` — `aiplatform.googleapis.com` +**Driver interface:** `services/vertexai/driver/` — `aiplatform.googleapis.com` REST rooted at `/v1/projects/{p}/locations/{l}/...` with the Model Garden `generateContent` surface at `/v1/publishers/...`. Control-plane mutations return done @@ -1385,7 +1385,7 @@ rates integrate Vertex with the cross-cutting layers like every other service. ### Azure — Azure AI -**Driver interface:** `azureai/driver/` — spans both ARM providers plus the data planes. +**Driver interface:** `services/azureai/driver/` — spans both ARM providers plus the data planes. **Azure:** Azure AI Foundry / AI Studio / Azure OpenAI (`Microsoft.CognitiveServices`) and Azure Machine Learning (`Microsoft.MachineLearningServices`). @@ -1423,7 +1423,7 @@ integrate Azure AI with the cross-cutting layers like every other service. ## 23. AI Search -**Driver interface:** `azuresearch/driver/` — `Microsoft.Search/searchServices` (ARM control +**Driver interface:** `services/azuresearch/driver/` — `Microsoft.Search/searchServices` (ARM control plane) plus the `{service}.search.windows.net` data plane. **Azure:** Azure AI Search (the RAG / retrieval backbone). **AWS / GCP:** _not applicable_.