Data Engineer
Posted Updated
Description
PW (PhysicsWallah) is seeking an experienced Data Engineer to design, build, and optimize scalable data infrastructure and pipelines. You will work on ETL/ELT processes, distributed data processing, and data lake and warehouse architectures to support business intelligence and analytics.
Collaborate with data science, analytics, and product teams to gather requirements and implement data-driven solutions. You will mentor juniors, drive performance tuning, cost optimization, and ensure data
Design and implement scalable, efficient ETL/ELT pipelines to ingest, transform, and process data from multiple sources. Architect and maintain robust data lake and warehouse solutions, aligning with business and analytical needs. Own the development and optimization of distributed data processing systems using Spark, AWS EMR, or similar technologies. Collaborate with cross-functional teams (data science, analytics, product) to gather requirements and implement data-driven solutions. Ensure high levels of data quality, security, and availability across systems. Evaluate emerging technologies and tools for data processing and workflow orchestration. Build reusable components, libraries, and frameworks to enhance engineering efficiency and reliability. Drive performance tuning, cost optimization, and automation of data infrastructure. Mentor junior engineers, review code, and set standards for development practices. 1+ years of professional experience in data engineering or backend systems with a focus on scalable systems. Strong hands-on experience with Python or Scala, and writing efficient, production- grade code. Deep understanding of data engineering concepts: data modeling, data warehousing, data lakes, streaming vs. batch processing, and metadata management. Solid experience with AWS (S3, Redshift, EMR, Glue, Lambda) or equivalent cloud platforms. Experience working with orchestration tools like Apache Airflow (preferred) or similar (Azkaban, Luigi).Skills
Sign in to read and join the discussion.