feat(aws): add CloudTrail to ECS v8.0.0 transforms (core + comprehensive) - #33
Open
behobu wants to merge 3 commits into
Open
feat(aws): add CloudTrail to ECS v8.0.0 transforms (core + comprehensive)#33behobu wants to merge 3 commits into
behobu wants to merge 3 commits into
Conversation
Retarget the AWS CloudTrail normalization templates (core + comprehensive) to ECS v8.0.0, for the many deployments still on the 8.0 line. Same jq logic as the v8.11.0 pair — every fieldset used (event, cloud, user, source, user_agent, related, error, tls, and the aws.cloudtrail.* vendor namespace) is present in ECS 8.0.0, verified against Elastic's 8.0 field reference — so this is a clean version bump with no field surgery. Validated locally with jq against a 5-record matrix (S3 data event, IAM assumed-role, console-login failure, access-denied, service event) covering the success/failure, authentication/iam/configuration, and access/creation/deletion/change/start/info branches. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Mirrors the same fix on the v8.11.0 pair so the two version trees stay identical apart from their version strings. The $category and $type branches piped $r.eventName straight into test(), which hard-errors on a record where eventName is absent or null. $r.eventSource two lines away was already guarded with // "". The realistic trigger is feeding the aws-sqs-s3-cloudtrail connector's default by_size output in without exploding .Records[] first — that envelope has no top-level eventName, so the transform died on a cryptic jq error instead of degrading. Output on records that do carry eventName is byte-identical before and after. Verified across the same 17-case matrix; ecs.version stays 8.0.0 throughout. Also corrects the header comment about aws-sqs-s3-cloudtrail: its Chunking Mode decides the shape. per_record already emits one bare event; only by_size (the default) needs the .Records[] explode.
Live CloudTrail records from the jfrog demo org carried userIdentity.onBehalfOf, which the comprehensive transform dropped despite its description promising full CloudTrail field coverage. AWS sets it when the caller is an IAM Identity Center user (userIdentity.type "IdentityCenterUser"), and it is the only thing in the record that identifies which Identity Center user the call was made on behalf of — so losing it loses attribution for SSO-initiated activity. Mapped into the vendor namespace as aws.cloudtrail.user_identity.on_behalf_of with the two documented subfields, snake_cased to match the surrounding style: userId -> user_id identityStoreArn -> identity_store_arn Field names taken from the AWS userIdentity reference. Records without onBehalfOf are unaffected: $ui.onBehalfOf is null, both subfields resolve to null, and prune drops the empty object rather than emitting an empty on_behalf_of. Verified against a record carrying the full object, one carrying only userId, one with no onBehalfOf at all, and an empty record, plus the full 17-case matrix. Core is deliberately unchanged — it has no vendor namespace.
behobu
marked this pull request as ready for review
July 30, 2026 18:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the same two AWS CloudTrail → ECS transforms retargeted to ECS v8.0.0, for deployments still on the 8.0 line:
ecs/v8.0.0/aws/aws-cloudtrail-core.yamlecs/v8.0.0/aws/aws-cloudtrail-comprehensive.yamlRelationship to the v8.11.0 PR
Same jq logic as the v8.11.0 pair (#32) — a clean version bump, no field surgery. This branch is based on
mainand is independent of #32, so the two can review/merge separately. The two trees are byte-identical apart from their version strings, verified by diffing with version lines filtered out.Every ECS field used exists in 8.0.0. Verified programmatically against Elastic's generated field list for
v8.0.0— all 27 emitted ECS fields are present, and theevent.category/event.typevalues used (authentication,iam,configuration/start,access,creation,deletion,change,info) are all in 8.0.0's allowed-value sets. No field needs version-specific treatment, which is why the logic can be identical.Validation
37 real CloudTrail records run through both files, from two sources. No failures on either file.
ecs.versionis8.0.0throughout — confirmed no cross-contamination from the 8.11.0 tree.1. Live pipeline records (30)
Pulled off the input node of the live
cloudtrailinput in the jfrog demo org —use_synthetic_data: false, real S3 bucket, and an organization trail (4 distinctrecipientAccountIds, 6 regions including us-east-1).eventNameacross 12 distincteventSource.Recordswrapper), confirming the connector contract.aws.cloudtrail.*vendor paths.Real-world shapes verified to map correctly:
userIdentity.typeAssumedRole(8+) andAWSService(1 →userblock pruned),tlsDetails(3 →tls.version/cipher/client.server_name),resources[](2 →ARN→arn,accountId→account_id), non-IPsourceIPAddress(3 →is_ipguard holds,source.ipomitted),sessionContext(8).2. Real failure records (7)
The live pipeline sample contained no failed calls, so these were sourced separately from CloudTrail Event History (
lookup-events) rather than through the connector — genuineInternalFailurerecords onCreateServiceLinkedChannel, all carrying anerrorMessage.event.outcome: "failure"on all.error.code7/7 anderror.message7/7 populated in comprehensive.creationevent type.Combined live branch coverage: outcomes success + failure; categories configuration, iam, authentication; types access, change, creation.
3. Synthetic matrix
17 cases for shapes real traffic doesn't produce: empty object, missing/null
eventName,by_sizeenvelope, IPv6 and service-namesourceIPAddress, ConsoleLogin success/failure/no-response, assumed-role name fallback,readOnly,resources, and each category/type branch. 0 failures.Fixes found during validation
eventNameguard (e67123f).$categoryand$typepiped$r.eventNamestraight intotest(), which hard-errors wheneventNameis absent or null. Realistic trigger: feedingaws-sqs-s3-cloudtrail's defaultby_sizeoutput in without exploding.Records[]first. Now guarded with// ""; output on records carryingeventNameis byte-identical before and after.userIdentity.onBehalfOfmapping (63b6d68). Live records carried this field and the comprehensive transform dropped it. AWS sets it for IAM Identity Center callers — the only attribution for SSO-initiated activity. Now mapped toaws.cloudtrail.user_identity.on_behalf_of(user_id,identity_store_arn), field names per AWS'suserIdentityreference. Absent → pruned, no empty object.Header comment corrected:
aws-sqs-s3-cloudtrail's Chunking Mode decides the shape —per_recordneeds nothing extra, onlyby_size(the default) needs the.Records[]explode.Known coverage limits
ConsoleLoginis not testable in our AWS org. Monad authenticates via IAM Identity Center, which doesn't emitConsoleLoginin the target account — confirmed zero such events across 4 accounts over a 7-day window. That branch is covered synthetically only; it remains correct for customers using IAM console sign-in.deletionevent type is likewise synthetic-only — no delete calls appeared in the sampled window.Note on version choice
ECS is currently at v9.4.0, and this repo otherwise only carries a
v8.11.0tree — so this PR adds an older version directory. Worth confirming there's a consumer actually pinned to the 8.0 line before merging; otherwise a v9.x variant would likely serve more users.Refs: CS-135