Joseph Machado

Exercise project for anyone starting in data engineering startdataengineering.com/post/data-engi… #dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark

Left anti-join is cool! Get all the data from the left table that has no matching data in the right table select t1.* from t1 left join t2 on t1.id=t2.id where https://t.co/TilIGFtVGj is null; #data #dataengineering #SQL #Database

An orchestration tool that I've been impressed with is @dagsterio. Easy setup, powerful features and great docs. Use 👇🏽 to play around with a pipeline on dagster startdataengineering.com/post/data-engi… #data #dataengineering #Python #Database #DataAnalytics

If you are interested in using "Change Data Capture" pattern for streaming ETL, check out startdataengineering.com/post/change-da… #ETL #changedatacapture #dataengineering #debezium #BigData

Preparing for SQL interviews? Do Leetcode SQL hard, sort by freq, and do the first 40 #data #dataengineering #Software #SQL

Starting a data project is a lot of work! It can be overwhelming to define the problem, set up systems, and then code! Use this DE project as a blueprint to build your own: startdataengineering.com/post/data-engi… #data #dataengineering #Database #DataAnalytics #dataviz #Python #datapipeline

Backfilling is an inevitable part of data projects. When designing your data pipelines take some time to answer the following questions 1. Does multiple backfill runs cause duplicate data? 2. Can multiple backfills be parallelized? #data #DataEngineering #datapipeline #datasets

When data to process is larger than memory, try to stream with python generators, before jumping to distributed systems! #data #dataengineering #Python #pythonlearning #Generator E.g. Stream a file(note () and not []), get diff between date cols

Learning data engineering? Build a pipeline locally. 1. Python to pull data from an API (e.g. Coincap) 2. Load data into a local Postgres container 3. Automate it with cron/task scheduler Start small, build, improve, & repeat. #data #dataengineering #pythonlearning #Python

Starting as a DE? 90% of what you will need is SQL (OLAP), python, & distributed system basics Don't overcomplicate! #data #dataengineering #SQL #Database #Python

Exercise project for anyone starting in data engineering startdataengineering.com/post/data-engi… #dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark

It can be overwhelming to start learning data engineering. I'd recommend starting with the basics of python, sql, UNIX commands, building a simple data project, update Github, Linkedin. Landing a DE job is 60% part learning and 40% marketing. See reply 👇🏽 for helpful links.

@matsonj RG35xx? I've got the purple one :) For AliExpress

in some ways you could say the future is already here

uv by @astral_sh is truly one of the best tools you can have in your toolkit as a DE. TIL: You can quickly start a jupyter notebook with it doc: docs.astral.sh/uv/guides/inte…

What is the biggest issue you face with writing tests? Let me know in the comments below. --- Follow me for more actionable data engineering content.

3. Provides new engineers making changes the confidence to move fast 4. Documents edge case handling 5. Automated checks as part of CI Do you write tests for your data pipeline?

Does pipeline code changes feel like playing Jenga? Write tests now! Save your sanity and get the highest return on time invested with tests. Here's why (well-written) tests are crucial: 👇 #data #datapipeline #datatest #unittest

Data engineers write the most complex piece of code to Upsert into tables. Here's THE command you need to know MERGE INTO/INSERT ON CONFLICT #data #dataengineering #SQL

Sign up for our free "Data Engineering 101" course to learn about data engineering core concepts, building scalable & resilient systems, data best practices, data modeling, and building projects that reflect real-world projects. startdataengineering.com/email-course/

Let me know in the comments how you use MERGE INTO! Interested in more actionable data engineering content? Checkout my website link (on my profile)

2. Re-running the pipeline with the same input will not give you the same output (i.e., not idempotent) This makes debugging challenging. However, with thoughtful design, MERGE INTO/INSERT ON CONFLICT are powerful tools every data engineer must understand. ---