Data Mechanics regularly updates its publicly available Docker images for Spark. This page contains the release notes of those images, indicating what changed in each version.
Check out this page to learn more about the Docker images offered by Data Mechanics.
Release dm16 (latest)
- Add new images with Spark 3.0.3 and Spark 3.1.2
- Upgrade Snowflake connector to 2.9.2
- Use the newly released Delta version 1.1.0 with Spark 3.2.0
- Add new images with Spark 3.2.0 and Hadoop 3.3.1
- Upgrade OS packages to apply latest security patches
- Use a JDK base image instead of a JRE base image to include tooks like
- Upgrade Snowflake connector to 2.9.1
- Pin pip version for Python 2 images. Latest pip versions are not compatible with Python 2 anymore.
- Use delta 1.0 on Spark ≥ 3.1
pyarrowsupport is added to all images. The version of
- Delta is upgraded to version 0.8.0 for all images with Spark version ≥ 3.0.0. The Delta version for Spark 2.4.x images is 0.6.1 (unchanged). Please note that Delta is still incompatible with Spark 3.1.1 at the time of this release.
dm12 is the first generation of images made available to the public.
- Snowflake version: 2.8.4
- AWS connector: determined by the Hadoop version
- Azure connector: determined by the Hadoop version
- GCS connector: 2.1.5
- Guava version: 29.0
- Delta version: 0.7.0 for Spark 3.0.0 and above, 0.6.1 for Spark 2.4.x