feat(azure_blob source): add Azure Blob Storage source

[Renizmy]

Summary

Adds an azure_blob source: the S3/SQS pattern from aws_s3, ported to Azure.

An Event Grid subscription publishes Microsoft.Storage.BlobCreated notifications to an Azure Storage Queue, which Vector polls. Each notification is resolved to a blob, downloaded, optionally decompressed, decoded with any codec, and its queue message deleted once the events are durably accepted by the pipeline (end-to-end acknowledgements).

• Event Grid and CloudEvents 1.0 notification schemas, base64 or raw JSON, auto-detected.
• compression: auto infers gzip/zstd from Content-Encoding, Content-Type, or blob suffix, then verifies that guess against the stream's magic bytes. On a mismatch it reads the blob raw and logs which signal disagreed. An explicitly configured codec is never second-guessed.
• Auth reuses azure_common from the azure_blob sink unchanged, so a config that works for the sink works here: connection string (account key or SAS), or account_name / blob_endpoint plus any Azure token credential.
• Blob and queue endpoints derive from the connection string or account name, with blob_endpoint / queue_endpoint overrides for sovereign clouds, private endpoints, and Azurite.
• At-least-once. A failed blob leaves its message queued to redeliver after the visibility timeout; dequeue_count is logged so poison messages are visible.

New metrics: azure_blob_event_ignored_total, azure_blob_processing_{succeeded,failed}_duration_seconds, azure_queue_message_{delete,processing,receive}_succeeded_total, azure_queue_message_received_messages_total.

Relationship to the aws_s3 source

Where aws_s3 had a precedent I copied it: file layout, type visibility, config field names, decoding pipeline, metric naming, #[allow]s.

aws_s3	azure_blob
sources/aws_s3/mod.rs + sqs.rs	sources/azure_blob/mod.rs + queue.rs
S3 bucket notification → SQS	Blob created → Event Grid → Storage Queue
Strategy::Sqs (default, sole variant)	Strategy::StorageQueue (default, sole variant)
sqs::Config (pub(super))	queue::Config (pub(super))
Ingestor / IngestorProcess / State	identical names and visibility
ProcessingError (pub), IngestorNewError (pub(super))	identical
Compression + determine_compression	similar, object_key_ → blob_name_
s3_object_processing_{succeeded,failed}_duration_seconds	azure_blob_processing_{…}
sqs_message_{delete,processing,receive}_succeeded_total	azure_queue_message_{…}

Six of the seven queue options keep their aws_s3 names and defaults: poll_secs, visibility_timeout_secs, max_number_of_messages, client_concurrency, delete_message, delete_failed_message. queue_url becomes queue_name because the Azure SDK derives the URL from the service endpoint. deferred, timeout, and tls_options are not ported.

Where Azure forced a divergence:

• Two notification schemas. S3 notifies SQS directly; Azure interposes Event Grid, which emits its own schema or CloudEvents 1.0 and base64-encodes the body. Hence the untagged QueueEvent enum and decode_message_text.
• No dead-letter queue. SQS has redrive; Azure Storage Queues have none, so a permanently failing message redelivers forever. This is why delete_failed_message matters more here, and why a compression mismatch downgrades to reading raw rather than failing the blob.
• No server-side long polling. aws_s3's poll_secs is the SQS long-poll wait time; Azure's GetMessages returns immediately, so the same name and default (15s) instead caps a client-side backoff between empty polls. Same knob, different mechanism.
• Shared Key signing. The generated queue client leaves Content-Length to the transport, which runs after the signing policy, so a ContentLengthPolicy is pushed ahead of it. No aws_s3 analogue; the AWS SDK signs its own requests.

Note for reviewers: this is the first Rust I have written. Since it is a port, the divergences above are where I had no precedent to lean on.

References

Vector configuration

Connection string:

sources:
  azure_logs:
    type: azure_blob
    connection_string: "${AZURE_STORAGE_CONNECTION_STRING}"
    queue:
      queue_name: vector-blob-events

sinks:
  out:
    type: console
    inputs: [azure_logs]
    encoding:
      codec: json

Managed Identity, tuned queue options, gzip blobs decoded as JSON:

sources:
  azure_logs:
    type: azure_blob
    account_name: mylogstorage
    auth:
      azure_credential_kind: managed_identity
    queue:
      queue_name: vector-blob-events
      poll_secs: 15
      visibility_timeout_secs: 300
      max_number_of_messages: 10
      client_concurrency: 4
    compression: gzip
    decoding:
      codec: json

Azurite, which is what the integration tests run against. Endpoints are explicit because a local address cannot be derived from the account name:

sources:
  azure_logs:
    type: azure_blob
    connection_string: "UseDevelopmentStorage=true;DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=<key>;BlobEndpoint=http://localhost:10000/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;"
    queue:
      queue_name: vector-blob-events

How did you test this PR?

Real Azure environment. Ran the source against a live storage account with an Event Grid subscription delivering Microsoft.Storage.BlobCreated notifications to a Storage Queue, ingesting both uncompressed blobs and gzip (.gz) blobs under the default compression: auto.

Unit tests (50): endpoint resolution (public cloud, development storage, DevelopmentStorageProxyUri port rewriting including IPv6 literals, explicit overrides, SAS appending); notification parsing for both schemas and the array-wrapped form; base64 vs raw bodies; subject/URL blob resolution and percent-decoding precedence; compression detection and magic-byte verification; the azure_core::Error source-chain renderer; config validation.

cargo nextest run --no-default-features \
  --features sources-azure_blob,sinks-azure_blob \
  -E 'test(azure_blob)'

Integration tests (14) against Azurite's blob and queue services. Azurite does not run Event Grid, so the tests enqueue synthetic notifications themselves.

cargo vdev int start azure
cargo vdev int test azure

Covers all three notification formats (Event Grid base64, Event Grid raw JSON, CloudEvents base64); JSON and bytes decoding; the Vector log namespace; blob names with spaces; a blob over the SDK's 4 MiB download partition size, which takes the partitioned path and requires the azure_core/tokio feature; gzip, multipart gzip, multipart zstd; multiline aggregation; rejected batches with and without delete_failed_message; non-BlobCreated events ignored and deleted. Each test asserts the resulting queue depth. The 13 that go through the shared test_event helper also run under assert_source_compliance(&SOURCE_TAGS, ...); the ignored-event-type test does not, since it deliberately produces no events and so emits none of the telemetry that check requires.

Checks:

make fmt
make check-clippy            # clean, full workspace
make check-generated-docs    # clean
make check-changelog-fragments
cargo vdev check events      # 0 errors
cargo check --no-default-features --features sources-azure_blob --bin vector

The source uses azure_common from src/sinks/, so src/sinks/mod.rs gates that module on sources-azure_blob too. Declaring sinks-azure_blob as a Cargo dependency instead would register the azure_blob sink in source-only builds.

Is this a breaking change?

• Yes
• No

New source behind a new sources-azure_blob feature. Existing files are touched only for registration: module declarations (src/sources/mod.rs, src/internal_events/mod.rs), feature definitions (Cargo.toml), metric-name enums (lib/vector-common/src/internal_event/metric_name.rs), and docs. The only change affecting existing behavior is one added #[cfg] feature on the already-shared azure_common module; azure_common itself is unmodified.

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

Fragment added at changelog.d/azure_blob_source.feature.md.

Cargo.lock changed (azure_storage_queue, plus the azure_core/tokio feature). LICENSE-3rdparty.csv gains one azure_storage_queue entry; its transitive dependencies (async-trait, azure_core, serde, time) were already listed.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[Renizmy]

I have read the CLA Document and I hereby sign the CLA

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bf4fa5c72d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 305ccd6b3a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97be5b4f10

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".