Azure Databricks Hands-on (Tutorials)

To run these exercises, follow each instructions below in this readme.

1. Storage Settings
2. Basics of PySpark, Spark Dataframe, and Spark Machine Learning
3. Data Transformation and Spark ML Pipeline
4. Hyper-parameter Tuning
5. MLeap (requires ML runtime)
6. Spark PyTorch Distributor (requires ML runtime)
7. Structured Streaming (Basic)
8. Structured Streaming with Azure Event Hubs or Kafka
9. Delta Lake
10. MLflow (requires ML runtime)
11. Orchestration with Azure Data Services
12. Lakeflow Spark Declarative Pipelines
13. Databricks SQL

These exercises focus only on essential learning of data, integration, and primitive ML features - not including generative AI features, such as, building AI agents, AI functions, fine-tuning foundation models, etc. (First commit: Oct 2018)

Getting Started

• Create Azure Databricks resource in Microsoft Azure.
  (Select premium plan, not standard. Select hybrid type, not serverless.)
• After the resource is created, launch Databricks workspace UI by clicking "Launch Workspace".
• Create a compute (cluster) in Databricks UI. (Select "Compute" menu and proceed to create.)
  In the wizard, enable "machine learning" to create ML runtime.
• Clone this repository on your desktop. (Or download HandsOn.dbc.)
  git clone https://github.com/tsmatsuz/azure-databricks-exercise
• Import HandsOn.dbc into your Databricks workspace as follows.
  • Select "Workspace" in Databricks UI.
  • Go to your user folder, and then select "Import".
  • Pick up HandsOn.dbc and submit import.
• Open the imported notebooks and attach above compute (cluster) in every notebooks. (Select compute (cluster) on the top of notebook.)
• Run "Exercise 01 : Storage Settings (Prepare)", before running other notebooks.
  Make sure you have an account with permissions to manage catalogs in Azure Databricks. (e.g., Global Administrators in tenant will have this permission.)

Note : This requires classic compute culster, and you cannot use Azure trial (free) subscription, because of the limited quota. When you're in Azure free subscription, please promote to pay-as-you-go. (The credit in free subscription will be reserved, even when you transit to pay-as-you-go.)
Throughout exercises in this repository, we mostly use All-purpose compute as cluster for experimentation purpose, but please create a Job and use Job compute as cluster for production. (All-purpose compute is more expensive than Job compute.)

Tsuyoshi Matsuzaki @ Microsoft Asia