Senior Data Engineer - ETL/Python

Role Overview :

The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities:

- Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

- Define technical strategy, architecture, and best practices for development and data engineering.

- Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

- Construct infrastructure for efficient ETL processes from various sources and storage systems.

- Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

- Lead the implementation of algorithms and prototypes to transform raw data into useful information.

- Review code for quality, scalability, and performance.

- Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

- Develop analytical tools, programs, and reporting mechanisms.

- Create data validation methods and data analysis tools.

- Interpret data trends and patterns to establish operational alerts.

- Conduct complex data analysis and present results effectively.

- Prepare data for prescriptive and predictive modeling.

- Ensure compliance with data governance and security policies.

- Troubleshoot, debug, and resolve complex technical issues.

- Drive continuous improvement in software and data development processes, tools, and methodologies.

- Mentor and guide engineers through code reviews, technical discussions, and training.

- Ensure timely delivery of projects while maintaining high engineering standards.

- Continuously explore opportunities to enhance data quality and reliability.

- Apply strong programming and problem-solving skills to develop scalable solutions.

- Demonstrate passion for testing strategy, problem-solving, and continuous learning.

- Willingness to acquire new skills and knowledge.

- Possess a product/engineering mindset to drive impactful data solutions.

- Experience working in distributed environments with global teams.

- Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills:

- Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

- High proficiency in Python, Scala, and Spark for applied large-scale data processing.

- Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

- Solid understanding of batch and streaming data processing techniques.

- Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

- Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

- Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

- Familiarity with Agile methodologies.

- Obsession for service observability, instrumentation, monitoring, and alerting.

- Knowledge or experience in architectural best practices for building data lakes.

Qualifications:

- Bachelors degree in computer science, Engineering, Information Systems, or related field.

Shift:

US Eastern Time (5:00 PM 2:00 AM).

Interviews:

Assessment || 2 Interview rounds.