Secrets & environment variables

Setting environment variables

Environment variables can easily be set by inserting this piece of configuration:

{
"driver": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
},
"executor": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
}
}

Setting environment variables using k8s secrets

If you have defined Kubernetes secrets, you can pass them to your Spark applications as an environment variable using the following piece of configuration:

{
"driver": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
},
}
},
"executor": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
},
}
}
}

This will create an environment variable with the key ENV_VAR_KEY and the value being the content of the secret secret-name at the field secret-field.

The secret value is not visible in the application configuration, and it will be redacted in the Spark UI. However, at this time, all secrets (created in the namespace spark-apps) are accessible to all Data Mechanics users.

Retrieving environment variables in your code

This is how you can retrieve environment variables in your Spark application code:

import os
env_vars = os.environ # Dictionary of key-value pairs
value = os.environ['ENV_VAR_KEY'] # ENV_VAR_VALUE