Data Mechanics regularly updates its publicly available Docker images for Spark. This page contains the release notes of those images, indicating what changed in each version.
Check out this page to learn more about the Docker images offered by Data Mechanics.
Release dm14 (latest)
- Use delta 1.0 on Spark ≥ 3.1
pyarrowsupport is added to all images. The version of
- Delta is upgraded to version 0.8.0 for all images with Spark version ≥ 3.0.0. The Delta version for Spark 2.4.x images is 0.6.1 (unchanged). Please note that Delta is still incompatible with Spark 3.1.1 at the time of this release.
dm12 is the first generation of images made available to the public.
- Snowflake version: 2.8.4
- AWS connector: determined by the Hadoop version
- Azure connector: determined by the Hadoop version
- GCS connector: 2.1.5
- Guava version: 29.0
- Delta version: 0.7.0 for Spark 3.0.0 and above, 0.6.1 for Spark 2.4.x