Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 22 additions & 50 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ CloudEmu follows a three-layer architecture that separates portable API concerns

### Layer 1: Portable API

The top layer lives in service-specific packages (`storage/`, `compute/`, `database/`, etc.). Each portable API type wraps a driver with cross-cutting concerns. For example, `storage.Bucket` wraps `driver.Bucket` and adds call recording, metrics collection, rate limiting, error injection, and simulated latency to every operation. This layer is provider-agnostic -- the same `storage.Bucket` works with S3, Blob Storage, or GCS.
The top layer lives in service-specific packages (`services/storage/`, `services/compute/`, `services/database/`, etc.). Each portable API type wraps a driver with cross-cutting concerns. For example, `storage.Bucket` wraps `driver.Bucket` and adds call recording, metrics collection, rate limiting, error injection, and simulated latency to every operation. This layer is provider-agnostic -- the same `storage.Bucket` works with S3, Blob Storage, or GCS.

### Layer 2: Driver Interfaces

Each service defines a minimal Go interface in `<service>/driver/driver.go`. These interfaces specify the operations that every provider must implement. For example, `storage/driver/driver.go` defines the `Bucket` interface with methods like `CreateBucket`, `PutObject`, `GetObject`, etc. Driver interfaces use plain Go types (no cloud SDK dependencies).
Each service defines a minimal Go interface in `<service>/driver/driver.go`. These interfaces specify the operations that every provider must implement. For example, `services/storage/driver/driver.go` defines the `Bucket` interface with methods like `CreateBucket`, `PutObject`, `GetObject`, etc. Driver interfaces use plain Go types (no cloud SDK dependencies).

### Layer 3: Provider Implementations

Expand All @@ -47,7 +47,7 @@ The bottom layer contains the actual mock implementations for each cloud provide

Sitting above Layer 2 (driver interfaces) are **cross-service engines** that consume driver interfaces directly without going through the portable API. They're peers of each other, not layers in the three-layer stack. Two exist today:

- `topology/` -- reads from compute, networking, and DNS drivers to simulate real network connectivity (`CanConnect`, `TraceRoute`, `Resolve`, security-group and NACL evaluation). See [topology.md](topology.md).
- `features/topology/` -- reads from compute, networking, and DNS drivers to simulate real network connectivity (`CanConnect`, `TraceRoute`, `Resolve`, security-group and NACL evaluation). See [topology.md](topology.md).
- `server/` -- exposes driver interfaces over HTTP in each cloud's native SDK wire format, so real `aws-sdk-go-v2`, `azure-sdk-for-go`, and `cloud.google.com/go` clients work against CloudEmu by only changing the endpoint. Covers Storage, Compute, Database, Networking, Monitoring, Serverless, and Message Queue across all 3 providers (AWS S3/EC2/DynamoDB/Lambda/SQS/CloudWatch + sibling Azure ARM and GCP REST handlers). Uses a pluggable `Handler` registry so new services drop in as self-contained packages without touching the core. See [sdk-server.md](sdk-server.md).

Both engines depend only on Layer 2 interfaces -- never on concrete provider types -- so they work uniformly across AWS, Azure, and GCP backends.
Expand Down Expand Up @@ -126,53 +126,25 @@ errors/
internal/
memstore/ # Generic Store[V]
idgen/ # ID generators (ARNs, Azure IDs, GCP IDs)
statemachine/
machine.go # Generic FSM
transitions.go # Transition map
pagination/ # Generic Paginate[T]
recorder/ # Call recording for assertions
metrics/ # In-memory metrics collection
ratelimit/ # Token bucket rate limiter
inject/ # Error injection (policies + injector)
cost/
cost.go # Cost tracker with default rates
storage/
storage.go # Portable storage API
driver/driver.go # Bucket interface
compute/
driver/driver.go # Compute interface
database/
driver/driver.go # Database interface
serverless/
driver/driver.go # Serverless interface
networking/
driver/driver.go # Networking interface
monitoring/
driver/driver.go # Monitoring interface
iam/
driver/driver.go # IAM interface
dns/
driver/driver.go # DNS interface
loadbalancer/
driver/driver.go # LoadBalancer interface
messagequeue/
driver/driver.go # MessageQueue interface
cache/
driver/driver.go # Cache interface
secrets/
driver/driver.go # Secrets interface
logging/
driver/driver.go # Logging interface
notification/
driver/driver.go # Notification interface
containerregistry/
driver/driver.go # ContainerRegistry interface
eventbus/
driver/driver.go # EventBus interface
relationaldb/
driver/driver.go # Relational DB interface (RDS, Aurora, Neptune,
# DocumentDB, Redshift, Azure SQL, Postgres/MySQL
# Flexible Server, Cloud SQL)
statemachine/ # Generic FSM
pagination/ # Generic Paginate[T]
services/ # emulated cloud services (portable API + driver interface)
storage/
storage.go # Portable storage API
driver/driver.go # Bucket interface
compute/ database/ relationaldb/ serverless/ networking/ monitoring/
iam/ dns/ loadbalancer/ messagequeue/ cache/ secrets/ logging/
notification/ eventbus/ containerregistry/ kubernetes/ resourcediscovery/
bedrock/ sagemaker/ vertexai/ databricks/ azureai/ azuresearch/
parameterstore/ tablestorage/ cost/
# each: <name>.go (portable API) + driver/ (interface)
features/ # cross-cutting capabilities you wrap drivers with
chaos/ # fault / latency / throttle injection
recorder/ # call recording for assertions
metrics/ # in-memory metrics collection
ratelimit/ # token-bucket rate limiter
inject/ # error injection (policies + injector)
topology/ # network reachability (CanConnect / TraceRoute / Resolve)
providers/
aws/
aws.go # AWS factory (wires all services)
Expand Down
2 changes: 1 addition & 1 deletion docs/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@ clock.Set(time.Date(2025, 1, 2, 0, 0, 0, 0, time.UTC))

## 10. Cross-Service Resource Discovery

CloudEmu ships a cross-service inventory engine (`resourcediscovery/`) that walks every service driver a provider holds and returns a single normalized view of what exists. It sits next to the `topology/` engine as a peer of the portable API — it owns no state, constructs from driver interfaces, and is purely query-driven.
CloudEmu ships a cross-service inventory engine (`services/resourcediscovery/`) that walks every service driver a provider holds and returns a single normalized view of what exists. It sits next to the `features/topology/` engine as a peer of the portable API — it owns no state, constructs from driver interfaces, and is purely query-driven.

The engine is the foundation for three SDK-compat handlers that speak the real cloud inventory APIs: **AWS Resource Explorer 2 + Resource Groups Tagging API**, **Azure Resource Graph**, and **GCP Cloud Asset Inventory**. A tag set through any one of those paths is immediately visible through the others, and through the engine's own `SearchByTag`.

Expand Down
56 changes: 28 additions & 28 deletions docs/services.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This document lists every service and operation available in CloudEmu across all
| 15 | Container Registry | `ecr` | `acr` | `artifactregistry` |
| 16 | Event Bus | `eventbridge` | `eventgrid` | `eventarc` |
| 17 | Relational Database | `rds` (+ Aurora/Neptune/DocumentDB engines), `redshift` | `azuresql`, `postgresflex`, `mysqlflex` | `cloudsql` |
| 18 | Kubernetes | `eks` + shared `kubernetes/` | `aks` + shared `kubernetes/` | `gke` + shared `kubernetes/` |
| 18 | Kubernetes | `eks` + shared `services/kubernetes/` | `aks` + shared `services/kubernetes/` | `gke` + shared `services/kubernetes/` |
| 19 | Resource Discovery | `resourceexplorer2` + `resourcegroupstaggingapi` | `resourcegraph` | `cloudasset` |
| 20 | Generative AI | `bedrock` (+ `bedrock-runtime`) | — | — |
| 21 | Databricks | — | `databricks` | — |
Expand All @@ -34,7 +34,7 @@ This document lists every service and operation available in CloudEmu across all

## 1. Storage

**Driver interface:** `storage/driver/driver.go`
**Driver interface:** `services/storage/driver/driver.go`
**AWS:** S3 | **Azure:** Blob Storage | **GCP:** GCS

### Bucket Operations
Expand Down Expand Up @@ -132,7 +132,7 @@ This document lists every service and operation available in CloudEmu across all

## 2. Compute

**Driver interface:** `compute/driver/driver.go`
**Driver interface:** `services/compute/driver/driver.go`
**AWS:** EC2 | **Azure:** Virtual Machines | **GCP:** GCE

### Instance Operations
Expand Down Expand Up @@ -223,7 +223,7 @@ This document lists every service and operation available in CloudEmu across all

## 3. Database

**Driver interface:** `database/driver/driver.go`
**Driver interface:** `services/database/driver/driver.go`
**AWS:** DynamoDB | **Azure:** Cosmos DB | **GCP:** Firestore

### Table Operations
Expand Down Expand Up @@ -288,7 +288,7 @@ This document lists every service and operation available in CloudEmu across all

## 4. Serverless

**Driver interface:** `serverless/driver/driver.go`
**Driver interface:** `services/serverless/driver/driver.go`
**AWS:** Lambda | **Azure:** Functions | **GCP:** Cloud Functions

### Function Operations
Expand Down Expand Up @@ -354,7 +354,7 @@ This document lists every service and operation available in CloudEmu across all

## 5. Networking

**Driver interface:** `networking/driver/driver.go`
**Driver interface:** `services/networking/driver/driver.go`
**AWS:** VPC | **Azure:** VNet | **GCP:** GCP VPC

### VPC Operations
Expand Down Expand Up @@ -474,7 +474,7 @@ This document lists every service and operation available in CloudEmu across all

## 6. Monitoring

**Driver interface:** `monitoring/driver/driver.go`
**Driver interface:** `services/monitoring/driver/driver.go`
**AWS:** CloudWatch | **Azure:** Azure Monitor | **GCP:** Cloud Monitoring

### Metric Operations
Expand Down Expand Up @@ -515,7 +515,7 @@ This document lists every service and operation available in CloudEmu across all

## 7. IAM

**Driver interface:** `iam/driver/driver.go`
**Driver interface:** `services/iam/driver/driver.go`
**AWS:** IAM | **Azure:** Azure IAM | **GCP:** GCP IAM

### Users
Expand Down Expand Up @@ -599,7 +599,7 @@ This document lists every service and operation available in CloudEmu across all

## 8. DNS

**Driver interface:** `dns/driver/driver.go`
**Driver interface:** `services/dns/driver/driver.go`
**AWS:** Route 53 | **Azure:** Azure DNS | **GCP:** Cloud DNS

### Zone Operations
Expand Down Expand Up @@ -638,7 +638,7 @@ This document lists every service and operation available in CloudEmu across all

## 9. Load Balancer

**Driver interface:** `loadbalancer/driver/driver.go`
**Driver interface:** `services/loadbalancer/driver/driver.go`
**AWS:** ELB | **Azure:** Azure LB | **GCP:** GCP LB

### Load Balancer Operations
Expand Down Expand Up @@ -696,7 +696,7 @@ This document lists every service and operation available in CloudEmu across all

## 10. Message Queue

**Driver interface:** `messagequeue/driver/driver.go`
**Driver interface:** `services/messagequeue/driver/driver.go`
**AWS:** SQS | **Azure:** Service Bus | **GCP:** Pub/Sub

### Queue Operations
Expand Down Expand Up @@ -749,7 +749,7 @@ This document lists every service and operation available in CloudEmu across all

## 11. Cache

**Driver interface:** `cache/driver/driver.go`
**Driver interface:** `services/cache/driver/driver.go`
**AWS:** ElastiCache | **Azure:** Azure Cache | **GCP:** Memorystore

### Cache Instance Operations
Expand Down Expand Up @@ -794,7 +794,7 @@ This document lists every service and operation available in CloudEmu across all

## 12. Secrets

**Driver interface:** `secrets/driver/driver.go`
**Driver interface:** `services/secrets/driver/driver.go`
**AWS:** Secrets Manager | **Azure:** Key Vault | **GCP:** Secret Manager

### Secret Operations
Expand All @@ -820,7 +820,7 @@ This document lists every service and operation available in CloudEmu across all

## 13. Logging

**Driver interface:** `logging/driver/driver.go`
**Driver interface:** `services/logging/driver/driver.go`
**AWS:** CloudWatch Logs | **Azure:** Log Analytics | **GCP:** Cloud Logging

### Log Group Operations
Expand Down Expand Up @@ -862,7 +862,7 @@ This document lists every service and operation available in CloudEmu across all

## 14. Notification

**Driver interface:** `notification/driver/driver.go`
**Driver interface:** `services/notification/driver/driver.go`
**AWS:** SNS | **Azure:** Notification Hubs | **GCP:** FCM

### Topic Operations
Expand Down Expand Up @@ -894,7 +894,7 @@ This document lists every service and operation available in CloudEmu across all

## 15. Container Registry

**Driver interface:** `containerregistry/driver/driver.go`
**Driver interface:** `services/containerregistry/driver/driver.go`
**AWS:** ECR | **Azure:** ACR | **GCP:** Artifact Registry

### Repository Management
Expand Down Expand Up @@ -937,7 +937,7 @@ This document lists every service and operation available in CloudEmu across all

## 16. Event Bus

**Driver interface:** `eventbus/driver/driver.go`
**Driver interface:** `services/eventbus/driver/driver.go`
**AWS:** EventBridge | **Azure:** Event Grid | **GCP:** Eventarc

### Bus Management
Expand Down Expand Up @@ -986,7 +986,7 @@ This document lists every service and operation available in CloudEmu across all

## 17. Relational Database

**Driver interface:** `relationaldb/driver/driver.go`
**Driver interface:** `services/relationaldb/driver/driver.go`
**AWS:** `rds` (covers Aurora, Neptune, and DocumentDB engines), `redshift` | **Azure:** `azuresql`, `postgresflex`, `mysqlflex` | **GCP:** `cloudsql`

A single portable interface backs every RDBMS handler. Engine selection (MySQL / PostgreSQL / Aurora / Neptune / DocumentDB / Redshift / Cloud SQL / Azure SQL / …) is a field on the input config, not a separate driver.
Expand Down Expand Up @@ -1039,7 +1039,7 @@ A single portable interface backs every RDBMS handler. Engine selection (MySQL /
## 18. Kubernetes

**Control plane:** AWS `eks`, Azure `aks`, GCP `gke` — cluster, node-pool, and addon / Fargate-profile / maintenance-config lifecycle, driven by the real cloud SDKs.
**Data plane:** shared `kubernetes/` package — an in-memory Kubernetes API server registered by every cluster across all three providers. Kubeconfigs returned by the control plane point at `<base>/k8s/<cluster-uid>` so `client-go` and `kubectl` operate end-to-end.
**Data plane:** shared `services/kubernetes/` package — an in-memory Kubernetes API server registered by every cluster across all three providers. Kubeconfigs returned by the control plane point at `<base>/k8s/<cluster-uid>` so `client-go` and `kubectl` operate end-to-end.

Each provider exposes its native control-plane API. The data plane has no portable driver — clients connect via the kubeconfig the control plane hands out, then talk standard Kubernetes REST.

Expand Down Expand Up @@ -1075,7 +1075,7 @@ Operations: **18**

Operations: **26**

### Data plane (`kubernetes/`)
### Data plane (`services/kubernetes/`)

Shared in-memory K8s API server registered by every cluster from any provider. URL: `<base>/k8s/<cluster-uid>/...`. Anonymous auth (kubeconfigs use `insecure-skip-tls-verify: true`).

Expand All @@ -1100,11 +1100,11 @@ Shared in-memory K8s API server registered by every cluster from any provider. U

## 19. Resource Discovery

**Engine:** `resourcediscovery/` — a cross-service inventory engine that walks the Compute, Networking, Storage, Database, and Serverless drivers of any provider and returns a normalized `Resource` view (provider, service, type, ID, ARN/URN, region, tags, created-at). Auto-wired by every provider factory and exposed as `Provider.ResourceDiscovery`.
**Engine:** `services/resourcediscovery/` — a cross-service inventory engine that walks the Compute, Networking, Storage, Database, and Serverless drivers of any provider and returns a normalized `Resource` view (provider, service, type, ID, ARN/URN, region, tags, created-at). Auto-wired by every provider factory and exposed as `Provider.ResourceDiscovery`.

**SDK-compat handlers:** AWS Resource Explorer Two + Resource Groups Tagging API, Azure Resource Graph, and GCP Cloud Asset Inventory. All three sit on top of the same engine, so a tag written through any one path is visible through the others.

### Engine (`resourcediscovery/`)
### Engine (`services/resourcediscovery/`)

| Operation | Signature |
|-----------|-----------|
Expand Down Expand Up @@ -1155,7 +1155,7 @@ Operations: **Engine 8** + **AWS Resource Explorer 1** + **AWS Resource Groups T

## 20. Generative AI

**Driver interface:** `bedrock/driver/driver.go`
**Driver interface:** `services/bedrock/driver/driver.go`
**AWS:** `bedrock` (+ `bedrock-runtime`) | **Azure:** — | **GCP:** —

AWS-only. Backs the real `aws-sdk-go-v2/service/bedrock` and `.../bedrockruntime` clients against the in-memory backend.
Expand Down Expand Up @@ -1223,7 +1223,7 @@ AWS-only. Backs the real `aws-sdk-go-v2/service/bedrock` and `.../bedrockruntime

## 21. Databricks

**Driver interfaces:** `databricks/driver/driver.go` (control plane), `databricks/driver/dataplane.go` (data plane)
**Driver interfaces:** `services/databricks/driver/driver.go` (control plane), `services/databricks/driver/dataplane.go` (data plane)
**AWS:** — | **Azure:** `databricks` | **GCP:** —

Azure-only. The control plane backs the real `armdatabricks` SDK; the data plane backs the real `databricks-sdk-go` WorkspaceClient. The SDK-compat-only workspace families (secrets, tokens, git credentials, repos, DBFS, workspace files, SQL warehouses, pipelines, serving endpoints, SCIM identity, Unity Catalog) have no portable Go API — see [sdk-server.md](sdk-server.md).
Expand Down Expand Up @@ -1328,7 +1328,7 @@ Azure-only. The control plane backs the real `armdatabricks` SDK; the data plane

### AWS — SageMaker AI

**Driver interface:** `sagemaker/driver/driver.go` (control plane + `Runtime`)
**Driver interface:** `services/sagemaker/driver/driver.go` (control plane + `Runtime`)

The control plane speaks awsJson1_1 (`X-Amz-Target: SageMaker.*`); the runtime speaks
restJson1 (`POST /endpoints/{name}/invocations`). Asynchronous jobs complete synchronously
Expand All @@ -1354,7 +1354,7 @@ clients. **Total: 121 operations.**

### GCP — Vertex AI

**Driver interface:** `vertexai/driver/` — `aiplatform.googleapis.com`
**Driver interface:** `services/vertexai/driver/` — `aiplatform.googleapis.com`

REST rooted at `/v1/projects/{p}/locations/{l}/...` with the Model Garden `generateContent`
surface at `/v1/publishers/...`. Control-plane mutations return done
Expand Down Expand Up @@ -1385,7 +1385,7 @@ rates integrate Vertex with the cross-cutting layers like every other service.

### Azure — Azure AI

**Driver interface:** `azureai/driver/` — spans both ARM providers plus the data planes.
**Driver interface:** `services/azureai/driver/` — spans both ARM providers plus the data planes.
**Azure:** Azure AI Foundry / AI Studio / Azure OpenAI (`Microsoft.CognitiveServices`) and
Azure Machine Learning (`Microsoft.MachineLearningServices`).

Expand Down Expand Up @@ -1423,7 +1423,7 @@ integrate Azure AI with the cross-cutting layers like every other service.

## 23. AI Search

**Driver interface:** `azuresearch/driver/` — `Microsoft.Search/searchServices` (ARM control
**Driver interface:** `services/azuresearch/driver/` — `Microsoft.Search/searchServices` (ARM control
plane) plus the `{service}.search.windows.net` data plane.
**Azure:** Azure AI Search (the RAG / retrieval backbone). **AWS / GCP:** _not applicable_.

Expand Down
Loading