Skip to content

Commit f0a53ce

Browse files
JillGrant615JillGrant615
authored
Merge pull request #296500 from apurbasroy/hadoop-job-powershell-entra-auth
Hadoop job powershell entra auth
2 parents a9475d6 + cea80c0 commit f0a53ce

22 files changed

Lines changed: 3307 additions & 0 deletions

‎articles/hdinsight/TOC.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,30 @@ items:
347347
href: ./troubleshoot-sqoop.md
348348
- name: Understand and resolve WebHCat errors
349349
href: ./hdinsight-hadoop-templeton-webhcat-debug-errors.md
350+
- name: Cluster operations with Entra ID authentication
351+
items:
352+
- name: Create and manage Entra ID enabled Azure HDInsight clusters
353+
href: ./hdinsight-with-entra-authentication/create-clusters-with-entra.md
354+
- name: Manage Entra ID enabled clusters with ARM templates
355+
href: ./hdinsight-with-entra-authentication/manage-entra-id-enabled-azure-hdinsight-clusters-with-arm-templates.md
356+
- name: Manage Entra ID enabled clusters using REST API
357+
href: ./hdinsight-with-entra-authentication/manage-entra-id-enabled-cluster-with-rest-api.md
358+
- name: Manage Apache Hadoop clusters in Entra Auth enabled clusters by using the .NET SDK
359+
href: ./hdinsight-with-entra-authentication/manage-hadoop-cluster-dot-net-sdk.md
360+
- name: Run Hadoop jobs on Entra ID enabled clusters using PowerShell
361+
href: ./hdinsight-with-entra-authentication/run-map-reduce-jobs-entra-id-enabled-using-powershell.md
362+
- name: Run Apache Hive queries using PowerShell on Entra ID enabled clusters
363+
href: ./hdinsight-with-entra-authentication/run-apache-hive-queries-using-powershell-on-entra-enabled-hdinsight-cluster.md
364+
- name: Run Spark jobs in Entra ID enabled clusters using Rest API
365+
href: ./hdinsight-with-entra-authentication/run-spark-jobs-using-rest-api.md
366+
- name: Run Apache Hive queries in Entra ID enabled clusters using REST API
367+
href: ./hdinsight-with-entra-authentication/run-apache-hive-queries-using-rest-api.md
368+
- name: Run MapReduce jobs In Entra ID enabled clusters using REST API
369+
href: ./hdinsight-with-entra-authentication/run-map-reduce-rest-jobs.md
370+
- name: Run MapReduce jobs Entra ID enabled clusters using .NET SDK
371+
href: ./hdinsight-with-entra-authentication/run-map-reduce-jobs-dot-net-sdk.md
372+
- name: Run Hive Queries In Entra ID enabled clusters using .NET SDK
373+
href: ./hdinsight-with-entra-authentication/run-hive-queries-using-dot-net-sdk.md
350374
- name: Reference
351375
items:
352376
- name: Azure CLI

‎articles/hdinsight/hdinsight-with-entra-authentication/create-clusters-with-entra.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
title: Create and Manage Azure HDInsight clusters enabled with Entra ID Authentication
3+
description: Learn how to create Azure HDInsight clusters with Entra ID Authentication
4+
ms.service: azure-hdinsight
5+
ms.topic: how-to
6+
author: apurbasroy
7+
ms.author: apsinhar
8+
ms.reviewer: nijelsf
9+
ms.date: 10/02/2025
10+
---
11+
12+
# Create and manage Azure HDInsight cluster with Entra ID authentication
13+
14+
This article provides comprehensive information on creating and managing Azure HDInsight clusters with Microsoft Entra ID authentication.
15+
Users can securely authenticate and manage access to HDInsight clusters, ensuring enterprise-grade security and centralized identity governance by using Entra ID.
16+
17+
With this capability, organizations can enforce role-based access, streamline user onboarding and offboarding, and enhance compliance by using existing Entra ID policies.
18+
It simplifies cluster security management while providing a seamless login experience for data engineers, analysts, and administrators.
19+
20+
## Prerequisites
21+
22+
Before you begin, ensure the following requirements are met:
23+
24+
- Azure Subscription
25+
26+
- An active Azure subscription with sufficient permissions to create HDInsight clusters.
27+
28+
- Microsoft Entra ID Tenant
29+
30+
- Access to an Entra ID tenant linked to your Azure subscription.
31+
32+
- Permissions to create and assign Entra ID groups and roles.
33+
34+
- Resource Group
35+
36+
- A resource group in Azure where the HDInsight cluster can be deployed.
37+
38+
- HDInsight Cluster Requirements
39+
40+
- HDInsight cluster type (for example, Hadoop, Spark, HBase, or Kafka) selected for deployment.
41+
42+
- Correct region chosen that supports Entra ID integration.
43+
44+
## Overview
45+
46+
Users have to perform the following for setting up Entra ID authentication when creating a HDInsight cluster:
47+
48+
- Select the desired authentication method: Entra ID
49+
50+
- Add one (or more) admin Entra ID user during cluster creation(adding atleast one admin is mandatory).
51+
52+
:::image type="content" source="./media/create-clusters-with-entra/creation-cluster.png" alt-text="Screenshot of HDInsight cluster creation landing page." border="true" lightbox="./media/create-clusters-with-entra/creation-cluster.png":::
53+
54+
55+
56+
## User Profiles in Ambari
57+
58+
Entra ID enabled users are assigned one of two profiles:
59+
60+
- Cluster Admin: Admin permission.
61+
62+
- Cluster User: View only permission.
63+
64+
65+
>[!Note]
66+
>During cluster creation if the admin chooses Entra ID for authentication, then all the users in the cluster must be authenticated using Entra ID.
67+
>If the admin chooses basic authentication during cluster creation then all the users in the cluster must be authenticated using basic authentication.
68+
>During cluster creation if the user selects Entra ID authentication, during the entire lifecycle of the particular cluster authentication can be only done using Entra ID.
69+
>If the admin chooses basic authentication, during the entire lifecycle of the particular cluster authentication can be only done using Basic Authentication.
70+
> **User can use only one mode on authentication for a particular cluster.**
71+
72+
73+
:::image type="content" source="./media/create-clusters-with-entra/select-entra-button.png" alt-text="Screenshot of HDInsight landing page showing the selection of entra ID option in HDInsight landing page." border="true" lightbox="./media/create-clusters-with-entra/select-entra-button.png":::
74+
75+
:::image type="content" source="./media/create-clusters-with-entra/select-entra-user.png" alt-text="Screenshot of user selecting the users entra ID when selecting a cluster admin." border="true" lightbox="./media/create-clusters-with-entra/select-entra-user.png":::
76+
77+
## Login Options
78+
79+
Users can log in via Multifactor Authentication (MFA) once they input their Entra ID.
80+
81+
## Adding users with API
82+
83+
Admin can add multiple users at once via an API, ideal for managing large clusters.
84+
85+
This operation allows users to change the cluster gateway HTTP credentials.
86+
87+
88+
| **Method** | **Request URI** |
89+
|------------|-----------------|
90+
| POST | `https://management.azure.com/subscriptions/{subscription Id}/resourceGroups/{resourceGroup Name}/providers/Microsoft.HDInsight/clusters/{cluster name}/updateGatewaySettings?api-version={api-version}` |
91+
| Entra Cluster API Version| greater than or equal to `2025-01-15-preview`|
92+
93+
94+
```json
95+
{
96+
"restAuthEntraUsers": [
97+
{
98+
"objectId": "0d7c4bd6-d042-45ec-9ae5-1ed7871c38e0",
99+
"displayName": "Hemant Gupta",
100+
"upn": "[email protected]"
101+
}
102+
]
103+
}
104+
```
105+
106+
### Response
107+
HTTP 202 (Accepted) on successful completion of the operation.
108+
109+
## Authentication Process
110+
The authentication process varies based on the chosen method during cluster creation:
111+
112+
If Entra ID is selected:
113+
114+
- The cluster creator provides an ID for the default cluster administrator user in Ambari.
115+
116+
- The default admin can add Ambari users after cluster creation. Users may have either **Cluster Administrator** or **Cluster User** permissions, set via the Ambari UI or REST API.
117+
The cluster admin also has to add Object ID, and the display name and click on "**Save**".
118+
119+
:::image type="content" source="./media/create-clusters-with-entra/add-users.png" alt-text="Screenshot of Ambari page showing the users in the Ambari portal." border="true" lightbox="./media/create-clusters-with-entra/add-users.png":::
120+
121+
:::image type="content" source="./media/create-clusters-with-entra/user-roles.png" alt-text="Screenshot of Ambari add users page where cluster admin selects roles of newly added users." border="true" lightbox="./media/create-clusters-with-entra/user-roles.png":::
122+
123+
- A multifactor authentication prompt appears when the user logs in with their Entra ID.
124+
125+
## Basic Authentication
126+
127+
Users can use the legacy basic authentication way of authenticating users as well
128+
129+
If Basic Authentication is selected:
130+
131+
- User provides a User ID and password for the default admin user.
132+
133+
- New users can be created with various roles, similar to current functionality.
134+
135+
- Users are prompted to enter their User ID and password upon login.
136+
137+
## Steps to add object ID in Ambari UI
138+
139+
1. Log in to the Ambari portal.
140+
141+
:::image type="content" source="./media/create-clusters-with-entra/login-page.png" alt-text="Screenshot of Ambari landing page." border="true" lightbox="./media/create-clusters-with-entra/login-page.png":::
142+
143+
1. Navigate to "**Manage Ambari**" option.
144+
145+
:::image type="content" source="./media/create-clusters-with-entra/click-manage.png" alt-text="Screenshot of Ambari landing page where cluster admin has to click on manage Ambari button." border="true" lightbox="./media/create-clusters-with-entra/click-manage.png":::
146+
147+
1. Click on user tab to see all present users in Ambari UI.
148+
149+
:::image type="content" source="./media/create-clusters-with-entra/open-user-tab.png" alt-text="Screenshot of Ambari page where cluster admin clicks on User tab." border="true" lightbox="./media/create-clusters-with-entra/open-user-tab.png":::
150+
151+
1. Click on "Add User" tab to add more users in the cluster.
152+
153+
:::image type="content" source="./media/create-clusters-with-entra/add-users.png" alt-text="Screenshot of Ambari page showing the users in the Ambari portal." border="true" lightbox="./media/create-clusters-with-entra/add-users.png":::
154+
155+
1. Input Object ID, display name, and select the user access (Cluster Administrator or Cluster User). Select "**Save**".
156+
157+
:::image type="content" source="./media/create-clusters-with-entra/add-object-id.png" alt-text="Screenshot of Ambari page showing thee add users tab where cluster admin has to input new user information." border="true" lightbox="./media/create-clusters-with-entra/add-object-id.png":::
158+
159+

0 commit comments

Comments
 (0)