Skip to content

hdinsight-administer-use-powershell.md

Latest commit

 

History

History
140 lines (94 loc) · 5.66 KB

hdinsight-administer-use-powershell.md

File metadata and controls

140 lines (94 loc) · 5.66 KB
title Manage Apache Hadoop clusters with PowerShell - Azure HDInsight
description Learn how to perform administrative tasks for the Apache Hadoop clusters in Azure HDInsight by using Azure PowerShell.
ms.service azure-hdinsight
ms.topic how-to
ms.custom hdinsightactive, devx-track-azurepowershell
author hareshg
ms.author hgowrisankar
ms.reviewer nijelsf
ms.date 10/17/2024

Manage Apache Hadoop clusters in Azure HDInsight by using PowerShell

[!INCLUDE selector]

You can use Azure PowerShell to control and automate the deployment and management of your workloads in Azure. In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Az PowerShell module. For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

[!INCLUDE updated-for-az]

The Az PowerShell module installed.

Create clusters

To create clusters, see Create Linux-based clusters in HDInsight by using Azure PowerShell.

List clusters

To list all clusters in the current subscription, use the following command:

Get-AzHDInsightCluster

Show clusters

To show details of a specific cluster in the current subscription, use the following command:

Get-AzHDInsightCluster -ClusterName <Cluster Name>

Delete clusters

To delete a cluster, use the following command:

Remove-AzHDInsightCluster -ClusterName <Cluster Name>

You can also delete a cluster by removing the resource group that contains the cluster. Deleting a resource group deletes all the resources in the group, including the default storage account.

Remove-AzResourceGroup -Name <Resource Group Name>

Scale clusters

You can use the cluster scaling feature to change the number of worker nodes that are used by a cluster that's running in HDInsight without having to re-create the cluster. To change the Hadoop cluster size by using PowerShell, run the following command from a client machine:

Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>

For more information about scaling clusters, see Scale HDInsight clusters.

Update HTTP user credentials

The Set-AzHDInsightGatewayCredential parameter sets the gateway HTTP credentials of an HDInsight cluster.

$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"

Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential

Find the default storage account

The following PowerShell script demonstrates how to get the default storage account name and the related information:

#Connect-AzAccount
$clusterName = "<HDInsight Cluster Name>"

$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStorageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]

echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStorageType"

if ($defaultStorageType -eq "blob")
{
    $defaultBlobContainerName = $cluster.DefaultStorageContainer
    $defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
    $defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey

    echo "Default Blob container name: $defaultBlobContainerName"
    echo "Default Storage account key: $defaultStorageAccountKey"
}

Find the resource group

In the Azure Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. To find the resource group, use the following command:

$clusterName = "<HDInsight Cluster Name>"

$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup

Submit jobs

To use the following products to submit jobs, follow the instructions in the references:

Upload data to Azure Blob Storage

To upload data to Azure Blob Storage, see Upload data to HDInsight.

Related content