Data Eng Weekly

To all my dear #BigData colleagues! Feel invited to submit an abstract to #BigDataTechWarsaw 2020, so that we can meet in Warsaw, talk about data and have a beer 🍺 This is the conference that I co-organise :) The CfP is open until Sep 30th. medium.com/getindata-blog…

A couple of good links, both from @DataEngWeekly: 1. Review of idempotence complete with elevator analogy: lispcast.com/what-is-idempo… 2. How to learn to build distributed systems: brooker.co.za/blog/2019/04/0…

What changed in the Big data landscape from 2013 to 2019 blog.marouni.fr/bidata-trends-… - interesting analysis from @abbassmarouni based on articles in @DataEngWeekly

My @apachekafka @kafkastreams challenges github repo has been announced in the last @DataEngWeekly issue: dataengweekly.com/Data-Eng-Weekl… Thank you guys!

@dacort Sorry about that! Did it end up in your spam folder by any chance?

Didn't get my @DataEngWeekly in my inbox yesterday for some reason so tweeting out the latest issue: dataengweekly.com/Data-Eng-Weekl… I especially like the Spark study notes.

List of new features in Amazon Redshift so far in 2018 - queries on nested data recently announced! blog.openbridge.com/whats-new-with…

Aaand another chapter is done! 🎉 The Early Release of "Stream Processing with @ApacheFlink" was updated with a new chapter about connectors and end-to-end consistency. Only two chapters ("Setup & Configuration", "Operations") are left. I should start looking for another hobby 🤔

Data Eng Weekly 279 ▪︎ Benchmarking Hive on MR3 ▪︎ Event Sourcing ▪︎ Efficiently writing to a db ▪︎ Jepsen for Dgraph ▪︎ PulsarIO ▪︎ Scheduling of notebooks at Netflix ... and more! dataengweekly.com/Data-Eng-Weekl…

Apache Airflow 1.10.0 is out ❤️🎉 !! Highlights: - New RBAC web interface in beta - First class kubernetes operator - Experimental kubernetes executor - Timezone support - Performance optimizations for large DAGs - Many GCP and S3 integration improvements - Tons of Bug Fixes

We've just released Apache Arrow 0.10.0, the biggest release yet with 4 months of work and nearly 500 issues closed. We've added 3 new languages to the project: Go, Ruby, and Rust. Read more arrow.apache.org/blog/2018/08/0…

Fortnite: 12 AWS regions, 5000 shards of Kinesis Streams, 4000 instances of EMR, and adding 2PB of data per month. datanami.com/2018/07/31/ins…

Catch up on the latest issue of Data Eng Weekly before the next one this Sunday! dataengweekly.com/Data-Eng-Weekl…

The @ApacheFlink community reached 10,000 issues in JIRA. Thanks to everyone who participated! Let's file another 10,000 feature requests/bug reports and keep the community alive! issues.apache.org/jira/browse/FL…

Big growth in flash storage: 55% year-over-year in Q1 2018. datanami.com/2018/07/26/the…

Apache Accumulo 1.9.2 was released with fixes for a few serious bugs. accumulo.apache.org/release/accumu…

Data Eng Weekly Issue #274 Lots of stream processing coverage this week—Apache Kafka, Wallaroo, Apache Samza, WSO2, and Amazon SQS + a couple of posts on Kubernetes, db monitoring + 2 new books + a proposed data ethics checklist. dataengweekly.com/Data-Eng-Weekl…

ERA5 atmospheric data is now available on S3 as a public data set. Currently available from 2008 on-wards, all 9 petabytes dating back to 1950 will be released incrementally. medium.com/planet-os/era5…

Speeding up ETL in PySpark by parallelizing db access. @joaopedro.pinheiro88/how-to-increase-data-load-speed-from-database-with-pyspark-3741cccdf928" target="_blank" rel="nofollow noopener">medium.com/@joaopedro.pin…

HDP 3.0 delivers new capabilities for the enterprise to enable agile application deployment, new #machinelearning /deep learning workloads, real-time database, & security and governance. Learn more about the enhancements, here: bit.ly/2JZms8t #BigData

Data Eng Weekly #273 is out. It was a tough one—so much great content to choose from. Coverage includes Scio, make at Propublica, Paypal's NameNode analytics, MySQL on Kubernetes, Kinesis+Lambda, data replication at Hotels.com, & much more. dataengweekly.com/Data-Eng-Weekl…