v2.0.0 — Role-based restructure, /v2 module, and five new SDK-compat services#267
Merged
Conversation
…ata Plane) (#232) * Add Azure AI Driver And In-Memory Provider Add the azureai driver interfaces and in-memory provider Mock spanning both ARM providers that make up Azure AI — Microsoft.CognitiveServices (AI Foundry / AI Studio / the AI Services resource / Azure OpenAI) and Microsoft.MachineLearningServices (Azure ML) — plus the data planes: Azure OpenAI inference (chat/completions, embeddings, completions), the AI Foundry Agents/Assistants API, and AML online-endpoint scoring. CognitiveServices covers accounts (CRUD, keys, models, skus, usages), deployments, projects, raiPolicies, commitmentPlans, and private-endpoint connections. MachineLearningServices covers workspaces, computes (with a start/stop/restart state machine), online/batch endpoints + deployments, jobs, versioned assets (models/data/environments/components/featuresets), datastores, connections, schedules, and registries. Slices/maps are deep-copied on store and return and mutators use copy-then-Set. Auto-metrics push to Azure Monitor via SetMonitoring; wired into the Azure provider. * Add Azure AI SDK-Compat ARM And Data-Plane Servers Serve Microsoft.CognitiveServices and Microsoft.MachineLearningServices over the ARM JSON wire protocol, plus the Azure OpenAI inference / Assistants and AML scoring data planes. Real armcognitiveservices / armmachinelearning clients (and any REST caller) configured with a custom endpoint hit these handlers the same way they hit management.azure.com. CognitiveServices uses the shared azurearm.ParsePath; Azure ML nests deeper (onlineEndpoints/{e}/deployments/{d}, models/{n}/versions/{v}, computes/{c}/start) so its handler parses the trailing segments itself. The data plane is host/path-routed on /openai/ and /score, with the account scope derived from the request Host subdomain. Roundtrip tests start a real httptest server and cover every family over HTTP. Wired into the Azure server Drivers bundle. * Add Azure AI Portable Layer-1 Wrapper, Chaos And Cost Integration Add the portable azureai.AzureAI Layer-1 wrapper implementing the full driver.AzureAI surface through a single do() pipeline (recorder, metrics, rate limiting, error injection, latency), mirroring sagemaker/vertexai. Add a chaos wrapper (chaos.WrapAzureAI) that injects failures on account/workspace and deployment/job creation and the inference/scoring runtimes, plus default Azure AI cost rates. Brings Azure AI to full parity with how the other AI services participate in the cross-cutting layers. * Document Azure AI In Services Reference * Add Azure AI Search Driver And In-Memory Provider Add the azuresearch driver interfaces and in-memory provider Mock for Azure AI Search (Microsoft.Search) — the ARM control plane (searchServices lifecycle, admin/query keys, shared private links, private-endpoint connections) and the {service}.search.windows.net data plane (indexes, documents with upload/merge/delete + search/suggest/autocomplete/count/get, indexers with run/reset/status, data sources, skillsets, synonym maps, aliases, service statistics). Slices/maps are deep-copied and mutators copy-then-Set; document search is a deterministic substring match bounded by maxSearchTop. Auto-metrics push to Azure Monitor; wired into the Azure provider. * Add Azure AI Search SDK-Compat ARM And Data-Plane Servers Serve Microsoft.Search/searchServices over ARM and the search data plane over its host/path-routed surface (/indexes, /indexes/{i}/docs/*, /indexers, /datasources, /skillsets, /synonymmaps, /aliases, /servicestats). The data-plane service scope is derived from the {service}.search.windows.net Host subdomain. Roundtrip tests start a real httptest server and cover service lifecycle + keys + private links and the full index/document/indexer/datasource/skillset/synonym/ alias/stats surface. Wired into the Azure server Drivers bundle. * Add Azure AI Search Layer-1 Wrapper, Chaos And Cost Integration Add the portable azuresearch.AzureSearch Layer-1 wrapper implementing the full driver.AzureSearch surface through a do() pipeline (recorder, metrics, rate limiting, error injection, latency), a chaos wrapper (chaos.WrapAzureSearch) that injects failures on service/index creation, document indexing and the query runtime, and default Azure AI Search cost rates. * Document Azure AI Search In Services Reference * Address Review Comments On Azure AI And Search Emulation Resolve PR #232 review feedback across the CognitiveServices, MachineLearningServices, and Azure AI Search provider and server layers: - Search: mergeOrUpload now merges onto the existing document instead of replacing it; reject documents missing the key field; honor $filter (single OData comparison), $orderby, $top, $skip and $select; return 207 Multi-Status on partial-failure index batches. - Persist regenerated admin/query and Cognitive Services account keys so rotation is observable via a subsequent list. - Sort ListMessages/ListAssistants by creation sequence for deterministic ordering; pick the latest asset version numerically, not lexically. - Derive the AML online-endpoint name from the inference host so /score no longer collapses every endpoint to "default". - Emit a string error.code (not the numeric status) in data-plane error envelopes, and enforce the correct HTTP method on key/account actions. Add regression tests to the existing roundtrip test files. * Add Real-SDK Roundtrip Tests And Fix Data-Plane Input Handling Address the re-review follow-ups on PR #232: - Embeddings: parseEmbeddingInput now accepts token-array inputs ([]int and [][]int) in addition to string / []string, rendering each to a synthetic string so every input yields one embedding row. - Search data plane: normalize the fused OData key form the SDKs emit, so /indexes('name') and /docs('key') route identically to the slash forms. Add real-client roundtrip tests driving the official Azure SDKs against the in-memory server, proving wire compatibility per surface: - armcognitiveservices: account lifecycle + key rotation - armmachinelearning: workspace + versioned assets (numeric latest version) - azopenai: chat completions + embeddings - armsearch: service lifecycle + admin/query key rotation Azure ships no Go data-plane search SDK, so the search data plane's wire fidelity (paren-form routing) is covered by explicit fused-URL tests.
No CI existed. Adds .github/workflows/ci.yml running go build / vet / test on push to development|master and on every PR (Go 1.25, module cache). A golangci-lint v2 job runs advisory-only for now (the repo's strict config was never enforced in CI); flip continue-on-error once the tree is lint-clean.
golangci-lint-action@v6 installs golangci-lint v1, incompatible with the repo's v2 config. Install v2 via the official script instead. Still advisory (continue-on-error) until the tree is lint-clean.
Add server/aws/sts implementing the AWS STS query protocol (GetCallerIdentity, AssumeRole, GetSessionToken) so the real aws-sdk-go-v2/service/sts client works against cloudemu. Many SDK code paths call sts:GetCallerIdentity/AssumeRole on init; this removes that friction. STS needs no driver: identity is derived from the existing Drivers.AccountID / Region. It is gated on a new Drivers.STS bool and registered among the query-protocol handlers before EC2. Its Matches parses the form and claims only its disjoint action set, so it does not shadow RDS/Redshift/IAM/ELBv2/ElastiCache/SNS/EC2. Tests drive the real STS SDK against httptest, plus a dispatch test proving STS does not shadow EC2 RunInstances. Closes #241
Public repo, so Actions minutes are free — but keep it lean: trigger on pull_request (feature pushes covered by their PR) plus push to master; drop the separate golangci-lint download job (go vet stays); keep caching and cancel-in-progress.
Extend the SDK-compat S3 server handler to speak the multipart upload, object tagging, and bucket versioning REST/XML operations against the existing storage driver (handler-only; no new driver methods). Operations: - Multipart: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads - Object tagging: PutObjectTagging, GetObjectTagging, DeleteObjectTagging - Bucket versioning: PutBucketVersioning, GetBucketVersioning - ListObjectVersions (current objects reported as the null version) Sub-resource routing keys on query params (?uploads, ?tagging, ?versioning, ?versions, ?uploadId) checked before the existing method-based object/bucket dispatch, leaving PUT/GET/DELETE untouched. Adds sdk_roundtrip_test.go driving the real aws-sdk-go-v2 S3 client: multipart upload of a multi-part object then GetObject returns the concatenated body; object tagging put/get/delete round-trip; bucket versioning Enabled round-trip. Refs #118
CI (ci.yml): Build, Vet, Test, Tidy as blocking parallel jobs; Format and Lint advisory (pre-existing test-file gofmt + never-enforced golangci config). Security (security.yml): govulncheck, gosec, CodeQL, and Dependency Review as separate parallel jobs (advisory scanners to start), on PR / push-to-master / weekly. Every job restores the Go build+module cache to stay fast.
Add two Azure Storage data-plane REST handlers so real azqueue and aztables clients work against cloudemu: - server/azure/queue: Azure Queue Storage REST+XML handler backed by the existing messagequeue driver (create/delete/list queue, enqueue, dequeue, delete message). Wired as azure.Drivers.QueueStorage, reusing a dedicated servicebus.Mock instance as the messagequeue backend. - server/azure/table: Azure Table Storage OData/JSON handler backed by a new lightweight tablestorage/driver plus an in-memory providers/azure/tablestorage provider (create/delete/list table; insert/ get/update(merge|replace)/delete/query entities). Both register before the permissive Blob fallback. Detection signals are disjoint from Blob/Cosmos/Databricks and each other except the one inherently-ambiguous root GET /?comp=list shape (list-queues vs list-containers), documented in the handler. Includes real-SDK round-trip tests per service plus a no-shadowing dispatch test registering Blob+Queue+Table together.
Format, Lint, govulncheck, and gosec now run and surface findings as workflow annotations without failing the check, so the PR isn't red on pre-existing findings while the real gates (Build/Vet/Test/Tidy) stay blocking. CodeQL reports to the Security tab; Dependency Review blocks new high-severity CVEs.
Address review of #241: gofmt-align xml.go, and trim trailing slashes in roleNameFromArn so a stray 'MyRole/' doesn't produce an empty last segment (and fall back to a default if the name resolves empty).
- Fix pre-existing 'using resp before checking for errors' (httpresponse vet check) in kubernetes/handlers_http_test.go: check the Do() error before deferring resp.Body.Close() (4 sites). - Combine Build and Vet into one job so 'go vet' reuses the compile cache 'go build' just populated, instead of a standalone 'go vet ./...' that recompiles the whole module (~4-5 min). Test stays a parallel job.
setup-go's built-in cache keys on an exact go.sum hash, so every branch that adds a dependency gets a cold cache and recompiles the whole cloud-SDK tree (~4-5 min). Switch to actions/cache with a restore-keys fallback (go-<os>-*) so a changed go.sum still restores the previous build cache and only the changed deps recompile — repeat/PR runs drop to ~1-2 min.
Address review of #118: - assemblePartsInOrder sorts parts by PartNumber before concatenating, so an out-of-order (or unsorted-SDK) CompleteMultipartUpload no longer corrupts the object (data-integrity bug). - Multipart ops (Upload/Complete/Abort) map a driver not-found to NoSuchUpload instead of the object-level NoSuchKey, so the SDK decodes the right typed error. - CompleteMultipartUpload with an empty part list now returns MalformedXML instead of silently writing a 0-byte object.
The prior test codified the pre-fix bug (reversed parts yielding part2||part1); update it to assert the correct AAAABBBB ordering now that parts are sorted.
…stname collisions Address review of #243: - Table PUT/MERGE with no If-Match against a missing row now inserts (insert-or-replace/merge) instead of 404, matching aztables UpsertEntity. - Correct the Queue Matches docstring: the /{queue}/messages and /?comp=list shapes are NOT strictly disjoint from Blob (a blob named "messages", or list-containers) — Azure separates them only by hostname, so they're a documented, inherent limitation of serving Queue+Blob on one endpoint.
ci/cd: parallel CI jobs + security scanning (build, vet, test, tidy, lint, govulncheck, gosec, CodeQL, dependency-review)
feat(s3): advanced ops — multipart, object tagging, bucket versioning
feat(sts): AWS STS SDK-compat handler
feat(azure): Storage Queues + Table Storage SDK-compat
Emulate AWS Systems Manager Parameter Store so the real aws-sdk-go-v2/service/ssm client works against cloudemu. - parameterstore/driver: portable ParameterStore driver interface - parameterstore: portable wrapper (recorder/metrics/ratelimit/inject/latency) - providers/aws/ssm: in-memory memstore-backed provider (versioning, labels, hierarchical paths, history) - server/aws/ssm: AWS JSON 1.1 handler dispatched on X-Amz-Target AmazonSSM.* - wired SSM onto providers/aws.Provider and server/aws Drivers/Register Operations: PutParameter (Overwrite + versioning), GetParameter, GetParameters, GetParametersByPath (Recursive), DeleteParameter, DeleteParameters, DescribeParameters, GetParameterHistory, LabelParameterVersion. Version/label selectors via ':' suffix. SecureString is stored as-is (no KMS); WithDecryption is a no-op. Tests: sdk_roundtrip_test.go drives the real SDK client against httptest, asserting typed responses and a typed ParameterNotFound.
Address review of #239: - LabelParameterVersion now rejects labels that start with a digit or with aws/ssm (case-insensitive), returning them in InvalidLabels instead of attaching them — which also stops a numeric label (e.g. "5") from being attached and then made unreachable by version-number selector resolution. - PutParameter's Overwrite-race retry now uses the caller's context instead of context.Background().
feat(ssm): AWS SSM Parameter Store (driver + provider + SDK-compat handler)
…iders/server layout
…ghbors Resolves the 4 CodeQL go/uncontrolled-allocation-size high-severity alerts (pre-existing; first surfaced now that CodeQL runs in CI): - EC2/GCE/Azure-VM RunInstances validated count>0 but had no upper bound, so an oversized MaxCount could drive an unbounded slice allocation (OOM). Cap at 1000 per call, mirroring real provider limits. - Vertex AI FindNeighbors was already clamped to maxNeighbors (1000); drop the sized make() capacity so the (already-safe) allocation isn't flagged.
refactor(v2)!: restructure repo into services/ + features/, bump module to /v2
The v2 restructure moved service packages under services/ and cross-cutting ones under features/. Update the architecture directory tree and the prose path references (e.g. storage/driver -> services/storage/driver, topology/ -> features/topology/) in architecture.md, services.md, and features.md.
architecture.md prose (storage/, compute/, database/) and services.md kubernetes/ references now point at services/<name>/.
docs(v2): update in-repo docs to the services/features layout
NitinKumar004
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A major release. cloudemu is now organized by role so the repository stays legible as it grows — service packages live under
services/, the cross-cutting wrappers underfeatures/, and shared plumbing underinternal/. Because that moves public import paths, the module is nowgithub.com/stackshy/cloudemu/v2. On top of the restructure this release adds five new SDK-compat surfaces — AWS STS, AWS SSM Parameter Store, S3 multipart / tagging / versioning, Azure Queue & Table Storage, and Azure AI — and ships a full CI/CD + security pipeline./v2. Update the dependency and your imports:services/<name>(e.g.storage→services/storage,compute→services/compute,kubernetes→services/kubernetes,resourcediscovery→services/resourcediscovery,cost→services/cost).features/<name>:chaos,recorder,metrics,ratelimit,inject,topology.internal/:statemachine→internal/statemachine,pagination→internal/pagination.providers/{aws,azure,gcp}andserver/{aws,azure,gcp}are unchanged — only their import prefixes gain/v2.cloudemu.NewAWS(),NewAzure(),NewGCP()) and every driver, wire protocol, and error type are identical. The upgrade is a find-and-replace of the import prefix.✨ Features
AWS STS — SDK-compat (query protocol)
The real
aws-sdk-go-v2STS client now works against cloudemu:GetCallerIdentity,AssumeRole, andGetSessionTokenreturn well-formed credentials/identity envelopes so code that resolves an identity or assumes a role before calling other services runs end-to-end.AWS SSM Parameter Store — SDK-compat (#239)
AWS Systems Manager Parameter Store drives a new
parameterstoreservice:PutParameter(with overwrite + version bump),GetParameter/GetParameters/GetParametersByPath(recursive),DeleteParameter(s), and parameter labels — includingString,StringList, andSecureStringtypes and typedParameterNotFound/ParameterAlreadyExistserrors.S3 — multipart upload, tagging, and versioning
The S3 handler gains the advanced object surface the SDK exercises for large and mutable objects: multipart (
CreateMultipartUpload→UploadPart→CompleteMultipartUpload/AbortMultipartUpload, parts assembled inPartNumberorder), object tagging (PutObjectTagging/GetObjectTagging/DeleteObjectTagging), and versioning (PutBucketVersioningplus version-awareGetObject/DeleteObject).Azure Queue Storage + Table Storage — SDK-compat (#243)
Two Azure data-plane services land: Queue Storage (queue create/delete, put / get / peek / update / delete message with visibility-timeout and dequeue-count semantics) and Table Storage (table CRUD; insert / upsert / merge / replace / delete entity and OData-style query). Both speak the real
azure-sdk-for-gowire format.Azure AI — CognitiveServices + Machine Learning (#232)
Azure AI emulation across ARM and data plane: CognitiveServices accounts (ARM lifecycle) plus an assistants data plane, and Azure Machine Learning workspaces and their child resources. Unmodified
azure-sdk-for-goclients drive both control-plane provisioning and data-plane calls.🔧 Enhancements
CI/CD + security pipeline
A complete GitHub Actions setup lands with this release:
ci.yml—build-vet(build then vet, sharing one cache),test,tidy, and advisoryformat/lint(golangci-lint v2) jobs, running in parallel with Go build + module caching (actions/cache@v4withrestore-keys) so repeat runs finish in well under a minute. Triggers on PRs and pushes tomaster, withconcurrencycancel-in-progress.security.yml—govulncheck,gosec, CodeQL, anddependency-review(PR-only, fail-on high) as a separate parallel workflow.Security hardening (CodeQL)
RunInstancescount across AWS EC2, Azure Virtual Machines, and GCP GCE — a request over the per-call maximum (1000) now returnsInvalidArgumentinstead of attempting an uncontrolled allocation.FindNeighbors.Correctness fixes
PartNumberbefore assembly, maps unknown upload IDs toNoSuchUpload, and rejects an emptyCompleteMultipartUpload.contextthrough thePutParameterretry path.UpsertEntity; Queue/Blob/Table shared-hostname collisions are documented.Documentation
Every in-repo doc (
architecture.md,services.md,features.md,getting-started.md) and the package overview were rewritten for theservices//features//internal/layout, and all import paths and badges updated to the/v2module.Technical Details
Repository restructure &
/v2modulechore(v2)!: bump module path to github.com/stackshy/cloudemu/v2—go.modmodule path gains the required/v2suffix; every internal import rewritten in lockstep across providers, servers, examples, and helpers.services/(28 services), cross-cutting wrappers underfeatures/(chaos, recorder, metrics, ratelimit, inject, topology), and generic helpers underinternal/(statemachine, pagination).providers/andserver/trees kept their internal shape; only import prefixes changed. Build,go vet, and the full test suite are green on the new layout.AWS STS & SSM Parameter Store
server/aws/sts, query protocol) — GetCallerIdentity, AssumeRole, GetSessionToken; typed error envelopes; registered with a disjoint action set.server/aws/ssm, query protocol →providers/aws/ssm) — PutParameter / GetParameter / GetParameters / GetParametersByPath (recursive) / DeleteParameter(s); label + version handling;String/StringList/SecureString; typedParameterNotFound/ParameterAlreadyExists.fix(ssm)adds label validation and context threading in the Put retry path.S3 — multipart, tagging, versioning
feat(s3): add multipart, tagging, and versioning wire operations— CreateMultipartUpload / UploadPart / CompleteMultipartUpload / AbortMultipartUpload; PutObjectTagging / GetObjectTagging / DeleteObjectTagging; PutBucketVersioning with version-aware read/delete.fix(s3): sort multipart parts, map NoSuchUpload, reject empty Complete+test(s3)— out-of-order parts now assemble byPartNumber(regression test assertsAAAA…BBBBordering), unknown upload IDs map toNoSuchUpload, and an empty Complete is rejected.Azure Queue & Table Storage
server/azure/queue) — queue create/delete; put / get / peek / update / delete message with visibility-timeout and dequeue-count semantics.server/azure/table) — table CRUD; insert / upsert / merge / replace / delete entity; OData-style query.fix(azure-table)addsUpsertEntitysupport and documents Queue/Blob/Table shared-hostname collisions.Azure AI (CognitiveServices + Machine Learning)
server/azure/azureai) — ARMaccountslifecycle plus anassistantsdata plane.machinelearning*.go) over ARM, with data-plane round-trip coverage.CI/CD & security workflows
ci.yml—build-vetshares a single Go build+module cache (actions/cache@v4,restore-keys: go-${{ runner.os }}-);test,tidy, advisoryformat(gofmt, report-only) andlint(golangci-lint v2 via the official install script, report-only) run in parallel. Triggers:pull_request+pushtomaster;concurrencywith cancel-in-progress.security.yml—govulncheck(report-only),gosec(report-only), CodeQL (init/autobuild/analyze@v3,security-events: write), anddependency-review(fail-on-severity: high, PR-only), run as an independent parallel workflow.fix(security)—const maxRunInstances = 1000guard in EC2/VM/GCE RunInstances; drop sized alloc in Vertex AIFindNeighbors. Resolves the CodeQLgo/uncontrolled-allocation-sizefindings.