To mount Azure Data Lake Storage (ADLS) Gen2 in Azure Databricks, follow these steps:
-
Create a service principal and grant it access to your ADLS Gen2 account.
-
In your Databricks notebook, use the following code:
python
configs = {
"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": "<application-id>",
"fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="<scope-name>", key="<service-credential-key-name>"),
"fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<directory-id>/oauth2/token"
}source = “abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/”,
mount_point = “/mnt/<mount-name>”,
extra_configs = configs
)
Replace the placeholders with your specific values4.
While not recommended for production, you can use access keys for testing:
python
spark.conf.set(
"fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
dbutils.secrets.get(scope="<secret-scope>", key="<secret-key-name>")
)source = “abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/”,
mount_point = “/mnt/<mount-name>”
)
After mounting, verify using:
python
dbutils.fs.mounts()
-
Use service principal authentication for better security.
-
Store sensitive information like client secrets in Azure Key Vault.
-
Consider using Unity Catalog for managing data access instead of mounts4.
Remember, all users in the Azure Databricks workspace will have access to the mounted ADLS Gen2 account, so ensure proper access controls are in place4.