Use secrets in Databricks

Below scripts base on python notebook

dbutils.secrets.help()

Provides utilities for leveraging secrets within notebooks. Databricks documentation for more info.

get(scope: String, key: String): String -> Gets the string representation of a secret value with scope and key getBytes(scope: String, key: String): byte[] -> Gets the bytes representation of a secret value with scope and key list(scope: String): Seq -> Lists secret metadata for secrets within a scope listScopes: Seq -> Lists secret scopes

dbutils.secrets.listScopes()

Out[2]: [SecretScope(name='ADBDataLakeKeyVault'), SecretScope(name='DataLakeKeyVault')]

dbutils.secrets.list(scope="ADBDataLakeKeyVault")

Out[7]: [SecretMetadata(key='ADSL-AccountKey'),

SecretMetadata(key='ADSL-AccountName'),

SecretMetadata(key='ADSL-ContainerName-DWData'),

SecretMetadata(key='ADSL-ContainerName-ODSData'),

SecretMetadata(key='ADSL-ContainerName-RawData')]

dbutils.secrets.list(scope="DataLakeKeyVault")

Out[8]: [SecretMetadata(key='ADSL-AccountKey'),

SecretMetadata(key='ADSL-AccountName'),

SecretMetadata(key='ADSL-ContainerName-DWData'),

SecretMetadata(key='ADSL-ContainerName-ODSData'),

SecretMetadata(key='ADSL-ContainerName-RawData')]

dbutils.secrets.get(scope="ADBDataLakeKeyVault", key="ADSL-AccountKey")

Out[11]: '[REDACTED]'

spark.conf.set(
  "fs.azure.account.key." + dbutils.secrets.get(scope="DataLakeKeyVault",key="ADSL-AccountName") + ".dfs.core.windows.net",
  dbutils.secrets.get(scope="DataLakeKeyVault",key="ADSL-AccountKey"))

filePath = "abfss://" + dbutils.secrets.get(scope="DataLakeKeyVault",key="ADSL-ContainerName-RawData") + "@" + dbutils.secrets.get(scope="DataLakeKeyVault",key="ADSL-AccountName") + ".dfs.core.windows.net/"

dbutils.fs.ls(filePath)

Out[14]: [FileInfo(path='abfss://[REDACTED]@[REDACTED].dfs.core.windows.net/Customer.csv', name='Customer.csv', size=196514)]

dfCustomer = spark.read.format("csv") \
              .options(header='true', inferSchema='true') \
              .load(filePath + "Customer.csv")

dfCustomer.createOrReplaceTempView("customer")

%sql
select * from customer

Reference

Secrets CLI