Learn how Spotify ingests 1.4T+ events daily on GCP via 38K+ data pipelines, leveraging BigQuery, Dataflow, and Flyte to power ~5K dashboards and scale data-driven insights.| www.junaideffendi.com
Consultant Julien Hurault takes you inside the world of data engineering, sharing practical insights, real-world challenges, and his perspective on where the field is headed.| www.junaideffendi.com
Diving into four approaches from Spark Operator to EMR (EKS, EC2, and Serverless), sharing benchmarking results and key insights to help you choose the best option.| www.junaideffendi.com
Learn how Snapchat ingests ~2 trillions of events per day using Google Cloud Platform.| www.junaideffendi.com
Veteran data engineer Daniel Beach takes you inside the world of data engineering, sharing hard-earned insights, day-to-day challenges, and what’s on the horizon for the field.| www.junaideffendi.com
Discover how to build a complete data governance ecosystem in a Lakehouse architecture using leading open-source tools. Explore access control, metadata management, lineage, quality and more.| Junaid Effendi | Sharing knowledge for Engineers
Learn about the Data Tech Stack used by DoorDash to process hundreds of Terabytes of data every day.| www.junaideffendi.com
Join Vu Trinh as he navigates the world of data engineering, sharing insights, challenges, and emerging industry trends.| www.junaideffendi.com
Learn to deploy a secure, automated SFTP server with AWS Transfer Family & Terraform. Set up restricted users, enforce SSH & MFA, and leverage workflows for automation—scalable, secure, and efficient.| Junaid Effendi | Sharing knowledge for Engineers
Learn what data tech stack Meta leverages to process and store massive amount of data every day in their data centers.| www.junaideffendi.com
Join Yordan Ivanov as he demystifies data engineering, clarifying misconceptions about the role.| www.junaideffendi.com
This article outlines seven effective approaches to sharing knowledge, helping you become a better engineer and a team player while fostering continuous learning and collaboration within teams.| Junaid Effendi | Sharing knowledge for Engineers
Lets look into the top six ways that can help reducing your compute costs on AWS.| www.junaideffendi.com
Read how Pinterest leverages multiple data technologies for exabyte scale data lake.| www.junaideffendi.com
Look into the benefits of having a team and how to collaborate effectively leading to a great healthy engineering culture.| www.junaideffendi.com
Discover the best content from 2024, curated across various categories, featuring top creators and their exceptional work.| Junaid Effendi | Sharing knowledge for Engineers
Sharing few approaches from experience working on GenAI chatbot leveraging reporting tool as the data source.| Junaid Effendi | Sharing knowledge for Engineers
See what Stripe use in the backend to handle big data that has processed more than a trillion dollar worth of transactions.| Junaid Effendi | Sharing knowledge for Engineers
Interested in working in the data engineering landscape, check out this natural transition for Data Analyst.| Junaid Effendi | Sharing knowledge for Engineers
Sharing the experience of moving from Terraform to Asset Bundle for Databricks workflow deployment, covering challenges and benefits.| Junaid Effendi | Sharing knowledge for Engineers