Joseph Machado

Exercise project for anyone starting in data engineering startdataengineering.com/post/data-engi… #dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark

Stuck preparing for data engineering system design interviews? Use this article to guide you through the prep. startdataengineering.com/post/de_interv… #dataengineering #interview #systemdesign

Everyone says you need to know Python for data work. But what does that mean exactly? In this post I cover the key Python concepts every DE needs to know ⬇️ startdataengineering.com/post/python-fo… #dataengineering #python #datapipeline #softwareengineering #dataanalysis

Pipeline code can be generated; set yourself apart by understanding how to design pipelines. Learn how to design your pipelines based on given inputs and required outputs. startdataengineering.com/post/design-pa… #dataengineering #datawarehouse #sql #etl #datapipeline

I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out. I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really). It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely. The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture. We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying. I worry.

@sspaeti Nice, How'd you get image view inside vim? I use render markdown but no image protocols. Any recommendations?

Are you jumping from IDE to IDE? Learn vim motions. An investment that pays off for your career. It'll be here in 50 years - unlike most editors. ssp.sh/brain/vim-moti…

Your data warehouse bill is high for one reason. Full table scans. Every query reads everything, whether it needs to or not. Here are 6 storage patterns that fix this 👇 startdataengineering.com/post/data-stor…

The API is the easy part. Production is where the real learning happens.

6. Read the Spark UI: Slow stages, skewed tasks, spill to disk are all there. If you can't diagnose a hanging job, you can't own one in production. 7. Observability, audit, and lineage: You need to know what ran, when, on what data, and whether it succeeded.

Spark API is easy to learn. But to debug a hanging job, you need to know Spark internals. Here are 7 topics to know for production Spark 👇

@VicVijayakumar Thank you

@startdataeng am.gs.com/en-us/advisors…

40% of people earning $500,000+ per year say they’re living paycheck to paycheck.

Large datasets are stored as individual files in S3. Too many small files per dataset make reads expensive! Learn to: Detect it, Compact it or use table properties during insert. Read how to here 👇 startdataengineering.com/post/fix-small… #dataengineering #data #apachespark #apacheiceberg

PSA: Understand the concepts and read the docs, before using LLMs Claude sent me on a wild goose chase, hallucinations, complex setup that breaks stuff, etc Wasted a lot of time, only to realize the tool(quarto) I work with already does what I needed

Too many small files in your data lake impact performance. Detect it with Spark UI 1. Go to the stages tab, see the event timeline. 2. Many small tasks (1 task = 1 green chunk) indicate a many-small-files (or partitions) problem. Fix coming tomorrow #dataengineering

@fba YOLO SEO

Most SEO agencies will never tell you this: You can take a website to 100K visits/month in under a year. Without backlinks, a massive team nor ads. Just some YOLO SEO. Do these 7 moves and you'll be playing in easy mode, I swear. 1. Topical map first: know what you're allowed to rank for before even getting started. 2. Rewrite the existing pages before writing new ones, and make sure you comply with your topical map. 3. 90-day editorial calendar, low-hanging fruit only in the beginning. We'll scale that later. 4. 100-200 pSEO pages on a real strategy (not spam). Preferably, you have proprietary data. 5. 3-5 free tools where the market has gaps. You know the ones "Free AI generators" of all types 6. 20-50 commercial pages that match buyer intent. Landings, commercial pages, etc. 7. Weekly optimization on content + internal links. Don't stop optimizign until you see that needle moving. Ok? Want the full play? Comment "YOLO SEO" + like this post, and I'll DM the link to the full playbook.

@Tech_p001 :)

@startdataeng Keep building Keep learning bro

Starting to learn data engineering, and are not sure where to begin? Start with dbt, here’s why 1. Multi-hop data flow: Go from staging → Intermediate → Data Marts represent progressively transformed data.

@fba My website is not as big to need your services, do you have any book/course recommendation to learn SEO for a tech blog?