Secrets & environment variables

Setting environment variables

Environment variables can easily be set by inserting this piece of configuration in a config template or in config overrides:

{
"driver": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
},
"executor": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
}
}

Setting environment variables using k8s secrets

If you have defined Kubernetes secrets, you can pass them to your Spark applications as an environment variable. Merge the following piece of configuration into a config template or into config overrides:

{
"driver": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
}
}
},
"executor": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
}
}
}
}

This will create an environment variable with the key ENV_VAR_KEY and the value being the content of the secret secret-name at the field secret-field.

The secret value is not visible in the application configuration, and it will be redacted in the Spark UI. However, at this time, all secrets (created in the namespace spark-apps) are accessible to all Data Mechanics users.

Retrieving environment variables in your code

This is how you can retrieve environment variables in your Spark application code:

import os
env_vars = os.environ # Dictionary of key-value pairs
value = os.environ['ENV_VAR_KEY'] # ENV_VAR_VALUE

To list all configurations you can set in Data Mechanics, check out the API reference.