You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/event-hubs/azure-event-hubs-apache-kafka-overview.md
+18-17Lines changed: 18 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,10 @@
1
1
---
2
-
title: Azure Event Hubs for Apache Kafka
2
+
title: Apache Kafka Protocol Support in Azure Event Hubs
3
3
description: Learn what Apache Kafka in Azure Event Hubs is and how to use it to stream data from Apache Kafka applications without setting up a Kafka cluster on your own.
4
+
#customer intent: As a developer using Apache Kafka, I want to understand how Azure Event Hubs supports the Kafka protocol so that I can integrate my applications without modifying code.
4
5
ms.topic: overview
5
6
ms.subservice: kafka
6
-
ms.date: 12/12/2024
7
+
ms.date: 02/05/2026
7
8
# Customer intent: How does Azure Event Hubs support the Apache Kafka protocol?
8
9
---
9
10
@@ -39,7 +40,7 @@ Conceptually, Apache Kafka and Event Hubs are very similar. They're both partiti
39
40
Kafka Streams is a client library for stream analytics that is part of the Apache Kafka open-source project, but is separate from the Apache Kafka event broker.
40
41
41
42
> [!NOTE]
42
-
> Kafka Streams is currently in Public preview in Premium, and Dedicated tier.
43
+
> Kafka Streams is currently in public preview in Premium and Dedicated tier.
43
44
44
45
Azure Event Hubs supports the Kafka Streams client library, with details and concepts available [here](apache-kafka-streams.md).
45
46
@@ -60,28 +61,28 @@ Standalone and without ksqlDB, Kafka Streams has fewer capabilities than many al
60
61
### Kafka Transactions
61
62
62
63
> [!NOTE]
63
-
> Kafka Transactions is currently in Public preview in Premium, and Dedicated tier.
64
+
> Kafka Transactions is currently in public preview in Premium and Dedicated tier.
64
65
>
65
66
66
-
Azure Event Hubs supports Kafka transactions. More details regarding the support and concepts is available [here](apache-kafka-transactions.md)
67
+
Azure Event Hubs supports Kafka transactions. For more details about the support and concepts, see [Apache Kafka transactions](apache-kafka-transactions.md).
67
68
68
69
### Compression
69
70
70
71
> [!NOTE]
71
-
> The Kafka compression for Event Hubs is only supported in Premium and Dedicated tiers currently.
72
+
> Kafka compression for Event Hubs is currently supported only in Premium and Dedicated tiers.
72
73
>
73
74
74
75
The client-side [compression](https://cwiki.apache.org/confluence/display/KAFKA/Compression) feature in Apache Kafka clients conserves compute resources and bandwidth by compressing a batch of multiple messages into a single message on the producer side and decompressing the batch on the consumer side. The Apache Kafka broker treats the batch as a special message.
75
76
76
-
Kafka producer application developers can enable message compression by setting the compression.type property. Azure Event Hubs currently supports `gzip` compression.
77
+
Kafka producer application developers can enable message compression by setting the `compression.type` property. Azure Event Hubs currently supports `gzip` compression.
77
78
78
79
```properties
79
80
80
81
Compression.type = none | gzip
81
82
82
83
```
83
84
84
-
While the feature is only supported for Apache Kafka traffic producer and consumer traffic, AMQP consumer can consume compressed Kafka traffic as decompressed messages.
85
+
While the feature is only supported for Apache Kafka producer traffic and consumer traffic, AMQP consumer can consume compressed Kafka traffic as decompressed messages.
85
86
86
87
87
88
## Key differences between Apache Kafka and Azure Event Hubs
@@ -94,24 +95,24 @@ Scale in Event Hubs is controlled by how many [throughput units (TUs)](event-hub
94
95
95
96
## Is Apache Kafka the right solution for your workload?
96
97
97
-
Coming from building applications using Apache Kafka, it's also useful to understand that Azure Event Hubs is part of a fleet of services, which also includes [Azure Service Bus](../service-bus-messaging/service-bus-messaging-overview.md), and [Azure Event Grid](../event-grid/overview.md).
98
+
If you come from a background of building applications that use Apache Kafka, it's also useful to understand that Azure Event Hubs is part of a fleet of services that also includes [Azure Service Bus](../service-bus-messaging/service-bus-messaging-overview.md) and [Azure Event Grid](../event-grid/overview.md).
98
99
99
-
While some providers of commercial distributions of Apache Kafka might suggest that Apache Kafka is a one-stop-shop for all your messaging platform needs, the reality is that Apache Kafka doesn't implement, for instance, the [competing-consumer](/azure/architecture/patterns/competing-consumers) queue pattern, doesn't have support for [publish-subscribe](/azure/architecture/patterns/publisher-subscriber) at a level that allows subscribers access to the incoming messages based on server-evaluated rules other than plain offsets, and it has no facilities to track the lifecycle of a job initiated by a message or sidelining faulty messages into a dead-letter queue, all of which are foundational for many enterprise messaging scenarios.
100
+
While some providers of commercial distributions of Apache Kafka might suggest that Apache Kafka is a one-stopshop for all your messaging platform needs, the reality is that Apache Kafka doesn't implement, for instance, the [competing-consumer](/azure/architecture/patterns/competing-consumers) queue pattern. It doesn't support [publish-subscribe](/azure/architecture/patterns/publisher-subscriber) at a level that allows subscribers access to the incoming messages based on server-evaluated rules other than plain offsets. It has no facilities to track the lifecycle of a job initiated by a message or sidelining faulty messages into a dead-letter queue. All of these features are foundational for many enterprise messaging scenarios.
100
101
101
-
To understand the differences between patterns and which pattern is best covered by which service, see the [Asynchronous messaging options in Azure](/azure/architecture/guide/technology-choices/messaging) guidance. As an Apache Kafka user, you can find that communication paths you have so far realized with Kafka, can be realized with far less basic complexity and yet more powerful capabilities using either Event Grid or Service Bus.
102
+
To understand the differences between patterns and which pattern is best covered by which service, see the [Asynchronous messaging options in Azure](/azure/architecture/guide/technology-choices/messaging) guidance. As an Apache Kafka user, you can find that communication paths you realized with Kafka can be realized with far less basic complexity and yet more powerful capabilities by using either Event Grid or Service Bus.
102
103
103
104
If you need specific features of Apache Kafka that aren't available through the Event Hubs for Apache Kafka interface or if your implementation pattern exceeds the [Event Hubs quotas](event-hubs-quotas.md), you can also run a [native Apache Kafka cluster in Azure HDInsight](../hdinsight/kafka/apache-kafka-introduction.md).
104
105
105
106
## Security and authentication
106
-
Every time you publish or consume events from an Event Hubs for Kafka, your client is trying to access the Event Hubs resources. You want to ensure that the resources are accessed using an authorized entity. When using Apache Kafka protocol with your clients, you can set your configuration for authentication and encryption using the SASL mechanisms. When using Event Hubs for Kafka requires the TLS-encryption (as all data in transit with Event Hubs is TLS encrypted), it can be done specifying the SASL_SSL option in your configuration file.
107
+
Every time you publish or consume events from an Event Hubs for Kafka, your client tries to access the Event Hubs resources. You want to ensure that the resources are accessed using an authorized entity. When using Apache Kafka protocol with your clients, set your configuration for authentication and encryption using the SASL mechanisms. When using Event Hubs for Kafka, TLS-encryption is required (as all data in transit with Event Hubs is TLS encrypted). You can specify the SASL_SSL option in your configuration file.
107
108
108
-
Azure Event Hubs provides multiple options to authorize access to your secure resources.
109
+
Azure Event Hubs provides multiple options to authorize access to your secure resources:
109
110
110
111
- OAuth 2.0
111
112
- Shared access signature (SAS)
112
113
113
114
### OAuth 2.0
114
-
Event Hubs integrates with Microsoft Entra ID, which provides an **OAuth 2.0** compliant centralized authorization server. With Microsoft Entra ID, you can use Azure role-based access control (Azure RBAC) to grant fine grained permissions to your client identities. You can use this feature with your Kafka clients by specifying **SASL_SSL** for the protocol and **OAUTHBEARER** for the mechanism. For details about Azure roles and levels for scoping access, see [Authorize access with Microsoft Entra ID](authorize-access-azure-active-directory.md).
115
+
Event Hubs integrates with Microsoft Entra ID, which provides an **OAuth 2.0** compliant centralized authorization server. By using Microsoft Entra ID, you can use Azure role-based access control (Azure RBAC) to grant fine grained permissions to your client identities. Use this feature with your Kafka clients by specifying **SASL_SSL** for the protocol and **OAUTHBEARER** for the mechanism. For details about Azure roles and levels for scoping access, see [Authorize access with Microsoft Entra ID](authorize-access-azure-active-directory.md).
> These configuration properties are for the Java programming language. For **samples** that show how to use OAuth with Event Hubs for Kafka using different programming languages, see [samples on GitHub](https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/oauth).
126
127
127
128
### Shared Access Signature (SAS)
128
-
Event Hubs also provides the **Shared Access Signatures (SAS)** for delegated access to Event Hubs for Kafka resources. Authorizing access using OAuth 2.0 token-based mechanism provides superior security and ease of use over SAS. The built-in roles can also eliminate the need for ACL-based authorization, which has to be maintained and managed by the user. You can use this feature with your Kafka clients by specifying **SASL_SSL** for the protocol and **PLAIN** for the mechanism.
129
+
Event Hubs also provides **Shared Access Signatures (SAS)** for delegated access to Event Hubs for Kafka resources. Authorizing access using the OAuth 2.0 token-based mechanism provides superior security and ease of use over SAS. The built-in roles can also eliminate the need for ACL-based authorization, which you have to maintain and manage. You can use this feature with your Kafka clients by specifying **SASL_SSL** for the protocol and **PLAIN** for the mechanism.
@@ -156,10 +157,10 @@ Additionally, Event Hubs features such as [Capture](event-hubs-capture-overview.
156
157
157
158
Azure Event Hubs for Apache Kafka supports both idempotent producers and idempotent consumers.
158
159
159
-
One of the core tenets of Azure Event Hubs is the concept of **at-least once** delivery. This approach ensures that events are always delivered. It also means that events can be received more than once, even repeatedly, by consumers such as a function. For this reason, it's important that the consumer supports the [idempotent consumer](https://microservices.io/patterns/communication-style/idempotent-consumer.html) pattern.
160
+
One of the core tenets of Azure Event Hubs is the concept of **at-least once** delivery. This approach ensures that events are always delivered. It also means that consumers can receive events more than once, even repeatedly. For this reason, it's important that the consumer supports the [idempotent consumer](https://microservices.io/patterns/communication-style/idempotent-consumer.html) pattern.
160
161
161
162
## Related content
162
-
This article provided an introduction to Event Hubs for Kafka. To learn more, see [Apache Kafka developer guide for Azure Event Hubs](apache-kafka-developer-guide.md).
163
+
This article introduces Event Hubs for Kafka. To learn more, see [Apache Kafka developer guide for Azure Event Hubs](apache-kafka-developer-guide.md).
163
164
164
165
For a **tutorial** with step-by-step instructions to create an event hub and access it using SAS or OAuth, see [Quickstart: Data streaming with Event Hubs using the Kafka protocol](event-hubs-quickstart-kafka-enabled-event-hubs.md).
Copy file name to clipboardExpand all lines: articles/event-hubs/event-hubs-capture-enable-through-portal.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ To create an event hub within the namespace, follow these steps:
33
33
34
34
1. On the **Overview** page for your namespace, select **+ Event hub** on the command bar.
35
35
36
-
:::image type="content" source="./media/event-hubs-quickstart-portal/create-event-hub4.png" lightbox="./media/event-hubs-quickstart-portal/create-event-hub4.png" alt-text="Screenshot of the selection of Add event hub button on the command bar.":::
36
+
:::image type="content" source="./media/event-hubs-quickstart-portal/create-event-hub-button.png" lightbox="./media/event-hubs-quickstart-portal/create-event-hub-button.png" alt-text="Screenshot of the selection of Add event hub button on the command bar.":::
37
37
2. On the **Create event hub** page, type a name for your event hub, then select **Next: Capture** at the bottom of the page.
38
38
39
39
:::image type="content" source="./media/event-hubs-capture-enable-through-portal/create-event-hub-basics-page.png" alt-text="Screenshot of the Create event hub page.":::
0 commit comments