고정된 트윗
Just reactivated this account. Senior Data Engineer. Building in public — AWS, data pipelines, and the road to Europe. Follow along 🚀 #DataEngineering #AWS
English
DataEngineer.ai
59 posts
DataEngineer.ai
@ai_dataengineer
Senior Data Engineer · AWS | PySpark | Databricks
Bangalore,India 가입일 Ağustos 2025
193 팔로잉43 팔로워
Every week there is a new headline:
“AI will replace software engineers.”
“New AI model beats previous benchmarks.”
“Companies are reducing hiring due to AI.”
But one thing I’ve noticed is that while AI is changing how we work, it is also creating new opportunities for those willing to adapt.
A few years ago, Cloud Computing transformed the industry.
Engineers who learned AWS, Azure, and GCP stayed relevant.
Today, AI is creating a similar shift.
The question is no longer:
“Will AI affect my job?”
The better question is:
“How can I use AI to become more productive?”
For Data Engineers, AI still needs:
✅ High-quality data
✅ Reliable pipelines
✅ Data governance
✅ Scalable infrastructure
✅ Cloud platforms
Without these foundations, even the most advanced AI model cannot deliver business value.
Technology has always evolved.
The winners are usually not the people who resist change.
They are the people who learn, adapt, and grow with it.
The future belongs to engineers who combine domain expertise with AI, not those who see them as competing forces.
#AI #DataEngineering #CloudComputing #AWS #Databricks #Technology #CareerGrowth
English
Early in my Data Engineering career, I thought success meant making the pipeline run successfully.
If the job completed without errors, I considered the work done.
Then I started working on larger datasets and production environments.
That’s when I learned the real meaning of optimization.
Imagine two engineers building the same pipeline.
Engineer A processes 500 GB of data in 4 hours.
Engineer B processes the same 500 GB in 40 minutes.
Both produce the same output.
But one solution saves time, cloud costs, infrastructure resources, and helps meet business SLAs.
That’s optimization.
Optimization is not about writing clever code.
It’s about thinking differently.
Can we process less data?
Can we avoid unnecessary shuffles?
Can we partition data better?
Can we reduce cloud costs?
Can we make the pipeline scale when data grows 10x?
As Data Engineers, we’re not paid just to move data.
We’re paid to move data efficiently.
Because in the real world, business leaders don’t ask how many lines of code we wrote.
They care about:
✅ Performance
✅ Reliability
✅ Cost
✅ Scalability
A pipeline that works is good.
A pipeline that works efficiently is engineering.
#DataEngineering #PySpark #SparkOptimization #AWS #Databricks #BigData #CareerGrowth
English
Looking to connect with creative people who’re building cool things in:
🤖 AI
📱 Claude
💻 Tech
📊 Startups
🧠 Indie vibecoders
Drop what you're building:👇
English
Looking to connect with more builders on X
If you're into:
Frontend development
Design Engineering
Building SaaS
AI tools
UI/UX & animations
Next.js / React
Shipping in public
Learning by building
Drop a heyy and tell me what you're working on.
Looking to follow active builders, designers, and developers
English
Early in my career, I got an IAM error.
AccessDenied: User is not authorized to perform: s3:GetObject
I panicked. Added "Action": "*" to the policy.
It worked.
That single move is one of the most common — and dangerous — mistakes data engineers make.
IAM isn’t “someone else’s job.”
Every Glue job, every Lambda function, every Redshift cluster — needs permissions to talk to other AWS services.
If you’re writing pipelines, you’re writing IAM policies. Whether you realize it or not.
5 things every data engineer must know about IAM:
1. Users vs Roles — know the difference
👤 Users → for people (you logging into AWS console)
🤖 Roles → for services (Glue, Lambda, Redshift assuming permissions)
Your pipelines should never use a user’s credentials. Always roles.
2. Two policies, two jobs
📜 Trust policy → “WHO can assume this role?”
📜 Permission policy → “WHAT can this role do?”
A Glue job’s role trusts glue.amazonaws.com AND has permissions to read S3 + write to Glue Catalog.
3. Least privilege isn’t optional
"Action": "s3:*" on "Resource": "*" = an open door.
Scope it down:
"Action": ["s3:GetObject", "s3:PutObject"]
"Resource": "arn:aws:s3:::my-bucket/raw/*"
4. Never hardcode credentials
No access keys in your PySpark scripts. No .env files committed to Git.
Glue, Lambda, EC2 — all can assume roles automatically. Use that.
5. One role per job type
Don’t create one giant “DataEngineerRole” used everywhere.
A Glue ETL role ≠ a Lambda trigger role ≠ a Redshift COPY role.
Separate roles = smaller blast radius if something goes wrong.
That "Action": "*" shortcut I used early on?
In a real production environment, that’s the kind of mistake that shows up in a security audit — and in an interview question.
Full breakdown with real policy JSON + common mistakes → @mallinathnpatil1996/iam-for-data-engineers-the-permissions-layer-nobody-teaches-part-3-d775c98ebf78" target="_blank" rel="nofollow noopener">medium.com/@mallinathnpat…
#DataEngineering #AWS #IAM #CloudSecurity #Python #DataEngineer
English
Looking to connect with people interested in:
→ Full Stack
→ Frontend
→ Backend
→ DevOps
→ AI/ML
→ Data Science
→ Freelancing
→ Startups
If you're building, learning, coding, or growing in tech…
Say hi & let's connect 🚀
English
Looking to #connect with more people in tech/build in public 👋
Especially from:
- Web/App development
- AI/ML
- Indie Hackers
- Startups
- Growth & Marketing
- Automation
- No Code / Low Code
- Data Science
- Product Design / UI UX
- Tech in general
What are you guys building right now?
Let's connect and grow together 🤝🏻
English
🚨 The recent restrictions on advanced AI models for foreign nationals raise an important question:
Why does India need its own AI models?
For years, we have relied on global technology platforms for cloud computing, software, and now AI.
But recent events remind us that access to critical technology can change overnight due to regulations, geopolitics, or national security concerns.
This isn’t just about one company or one AI model.
It’s about technological self-reliance.
India needs its own AI ecosystem because:
✅ Control over critical technology
✅ Better support for Indian languages
✅ AI solutions tailored for Indian businesses
✅ Reduced dependence on foreign platforms
✅ Stronger innovation and research ecosystem
The goal isn’t to replace global AI models.
The goal is to ensure that Indian developers, researchers, startups, and enterprises always have access to world-class AI capabilities.
Just as India invested in digital payments, space technology, and semiconductor manufacturing, AI may become another strategic area for long-term national growth.
The future will belong to nations that not only use AI but also build it.
#AI #IndiaAI #ArtificialIntelligence #Technology #Innovation #DataEngineering #DigitalIndia
English
Hello,
I'm looking to connect with people interested in:
💻 Web Development
🚀 SaaS
🏗️ Startups
💼 Freelancing
Reply with what you're working on right now 👇
Let's connect. 🤝
#techtwitter #webDev #SaaS #startups
English
Hey tech people 👋
Looking to connect with builders interested in:
→ Frontend
→ Backend
→ Full Stack
→ DevOps
→ LeetCode
→ AI/ML
→ Data Science
→ UI/UX
→ DSA
→ Freelancing
→ Startups
If you're building, learning, coding, or growing in tech...
Drop your SaaS, startup, or project below. 🚀
Tell us:
• What it does
• Who it's for
If I genuinely like your project, I'll share it with my X audience and feature some of my favorites in future posts.
Let's discover great builders. 👇
English
Hey @X 👋,
Trying to make my feed more useful and less noise.
I want to connect with people building real things on the internet 🚀
Looking for:
💻 Developers
📱 Indie App Developers
🤖 AI Builders
☁️ Cloud & DevOps Engineers
🚀 SaaS Founders
📈 Growth & Marketing Experts
🎨 Product Builders
🧠 People obsessed with solving problems
Especially founders sharing:
• SaaS growth strategies
• User acquisition ideas
• Distribution lessons
• Marketing experiments that actually worked
Building products is hard. Growing them is harder 😂
Drop what you’re working on 👇, let's connect.
#letsconnect
English
I'm looking to #CONNECT with people interested in:
- Frontend
- Backend
- Full stack
- DevOps
- AI/ML
- Data Engineer
- UI/UX
- Freelancing
- Startup
- Saas
Say hi & Let's grow together
#BuildingInPublic
English
