Secrets & environment variables
Setting environment variables
Environment variables can easily be set by inserting this piece of configuration:
{
"driver": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
},
"executor": {
"envVars": {
"ENV_VAR_KEY": "ENV_VAR_VALUE"
}
}
}
Setting environment variables using k8s secrets
If you have defined Kubernetes secrets, you can pass them to your Spark applications as an environment variable using the following piece of configuration:
{
"driver": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
},
}
},
"executor": {
"envSecretKeyRefs": {
"ENV_VAR_KEY": {
"name": "secret-name",
"key": "secret-field"
},
}
}
}
This will create an environment variable with the key ENV_VAR_KEY
and the value being the content of the secret secret-name
at the field secret-field
.
The secret value is not visible in the application configuration, and it will be redacted in the Spark UI. However, at this time, all secrets (created in the namespace
spark-apps
) are accessible to all Data Mechanics users.
Retrieving environment variables in your code
This is how you can retrieve environment variables in your Spark application code:
- Python
- Java & Scala
import os
env_vars = os.environ # Dictionary of key-value pairs
value = os.environ['ENV_VAR_KEY'] # ENV_VAR_VALUE