Apache SystemDS

The @SIGMODConf Programming Contest goes into another round. We (Bo Tang, Tilmann Rabl, and myself) just published the timeline and task overview: sigmod-contest-2025.github.io/index.html Thanks to @carlo_curino and @Microsoft for the continued support.

Glad to share, our paper MEMPHIS is accepted at EDBT 2025! 🎉 MEMPHIS extends the LIMA framework and proposes a holistic approach for fine-grained reuse of intermediates and memory management across multiple backends. Paper: openproceedings.org/2025/conf/edbt…

I'm grateful for being part of the great (and now accepted) collaborative research center FONDA II.

I went through the most popular AI repos on GitHub, categorized them, and studied their growth trajectories. Here are some of the learnings: 1. There are 845 generative AI repos with at least 500 stars on GitHub. They are built with contributions from over 20,000 developers, making almost a million commits. 2. I divided the AI stack into four layers: application, application development, model development, and infrastructure. The application and application development layers have seen the most growth in 2023. The infrastructure layer remains more or less the same. Some categories that have seen the most growth include AI interface, inference optimization, and prompt engineering. 3. The landscape exploded in late 2022 but seems to have calmed down since September 2023. 4. While big companies still dominate the landscape, there’s a rise in massively popular software hosted by individuals. Several have speculated that there will soon be billion-dollar one-person companies. 5. The Chinese’s open source ecosystem is rapidly growing. 6 out of 20 GitHub accounts with the most popular AI repos originate in China, with two from Tsinghua University and two from Shanghai AI Lab.

@TheASF Major additions in this release include lineage-based reuse of Spark actions, parallel compressed feature transformation, bug fixes, and performance improvements. systemds.apache.org/release-notes/…

Apache SystemDS 3.1.0 is now available at: dlcdn.apache.org/systemds/3.1.0/ pypi.org/project/system… #SystemDS is an #opensource #machinelearning system for the end-to-end data science lifecycle. #opensource #ML #datascience

The SystemDS team is pleased to announce the release of version 3.0.0. This is the first release with Java 11 and Spark 3. Major additions include a feature-complete federated backend, Top-K cleaning, various new builtins, and performance improvements. systemds.apache.org/release-notes/…

Our preliminary experiments in @ApacheSystemDS with JDK 17's Float/DoubleVectors (for mapping to packed SIMD on x64 and aarch64) already look very promising: github.com/apache/systemd… Thanks @KevinInnerebner for the initial exploration.

proud of our team (@ArnabPhani, Sebastian Baunsgaard) that we made reproducibility submissions for all of our SIGMOD'21 papers (I did it last minute, thanks to the CIDR camera-ready deadline extension). github.com/damslab/reprod…

The SystemDS team is pleased to announce the release of Apache SystemDS version 2.2.0. Major additions include full support for compression, multi-threaded feature transformations, improved Python API, and performance improvements. Release Notes: systemds.apache.org/release-notes/…

The SystemDS team is pleased to announce the release of Apache SystemDS version 2.1.0. Major additions include a robust Federated backend w/ a Parameter Server for federated fraining, many new builtins, and performance improvements. Release Notes: systemds.apache.org/release-notes/…

4/4: All results are integrated and usable in @ApacheSystemDS. Btw, @ArnabPhani just today cut the next release candidate rc3 of Apache SystemDS 2.1, which after voting has passed will become available soon.

we're truly grateful for the recognition of our SliceLine work in the new data science & engineering track. mboehm7.github.io/resources/sigm… Svetlana did awesome work and it's a nice example how a course project in OSS leads to a master thesis and back into @ApacheSystemDS.

Happy to announce Apache SystemDS 2.0.0, which we released mid October as the first release after merging SystemDS back into Apache SystemML: systemds.apache.org/download