Setting environment variables
Setting environment variables using k8s secrets
If you have defined Kubernetes secrets, you can pass them to your Spark applications as an environment variable. Merge the following piece of configuration into a config template or into config overrides:
This will create an environment variable with the key
ENV_VAR_KEY and the value being the content of the secret
secret-name at the field
The secret value is not visible in the application configuration, and it will be redacted in the Spark UI. However, at this time, all secrets (created in the namespace
spark-apps) are accessible to all Data Mechanics users.
Retrieving environment variables in your code
This is how you can retrieve environment variables in your Spark application code:
- Java & Scala
To list all configurations you can set in Data Mechanics, check out the API reference.