Senior AWS Data Engineer - Databricks

Summary :

Exp. range : 7+ yrs (With 4 yrs+ of relevant experience as AWS Data Engineer)

Education : B.E / MCA or equivalent

Working model : Work from office (5-day week)

This is a hands-on technical leadership role. The successful candidate will contribute directly to architecture, coding, code reviews, troubleshooting, performance optimization, and implementation while leading and mentoring the data engineering team.

Key Responsibilities :

Technical Architecture & Design :

- Lead technical architecture and design efforts for data engineering solution components to deliver business solutions

- Design and implement scalable, secure and resilient data engineering solutions using AWS and Databricks, leveraging Databricks Lakehouse architecture, Apache Spark, Delta Lake and associated data engineering capabilities.

- Evaluate and select appropriate technology stack for data engineering solution components - cloud-native services, open-source tools, data storage, data processing frameworks etc.

Data Engineering Development :

- Lead, mentor and manage a team of dynamic data engineers for development, testing, and implementation of data pipelines, ensuring quality, performance, and on-time delivery

- Responsible for detailed technical design and data mapping specifications

- Ensure adherence to industry best practices, coding standards and guidelines driving continuous improvement in code quality and delivery excellence

- Optimize data solutions for performance, cost-efficiency and scalability

- Ensure data quality and integrity for downstream systems and applications

Technical Project Management :

- Plan, prioritize and manage data engineering projects, ensuring timely delivery and quality

- Identify and mitigate risks, troubleshoot issues, handle customer escalations

- Ensure alignment with business requirements and priorities, working closely with stakeholders - both internal and customer

Collaboration and Documentation :

- Maintain comprehensive technical documentation, including architecture diagrams, high-level/low-level data integration design, data mapping specifications, database design etc.

- Work closely with peers, BI engineers and data scientists to enable advanced analytics, machine learning models and reporting solutions

Requirements :

- 7+ years of hands-on experience in delivering data engineering solutions

- Experience leading a team of 5+ data engineers, including technical mentoring, solution/design reviews, code reviews, estimation, task prioritization, technical guidance and stakeholder communication.

- AWS & Databricks: Extensive hands-on experience designing and implementing data engineering solutions using AWS and Databricks. Strong experience with AWS services such as S3, Glue, Redshift and Athena, along with Databricks Lakehouse, Apache Spark/PySpark and Delta Lake.

- Data Engineering: Proven experience designing and implementing scalable ETL/ELT pipelines using Python, PySpark and Databricks. Experience with Databricks Jobs/Workflows, cluster configuration, performance optimization and data quality.

- Data Architecture: Strong understanding of data lake, data warehouse and Lakehouse architecture patterns, including data ingestion, transformation, storage, partitioning, schema evolution and consumption.

- Databases & SQL: Expert-level SQL skills with experience in relational and NoSQL databases, including query and performance optimization.

- DevOps & CI/CD: Experience with Git/GitHub and implementing CI/CD pipelines for data engineering solutions, including deployment of Databricks notebooks/jobs and associated infrastructure.

- AWS Security: Good understanding of AWS IAM, encryption, access controls, secrets management, and secure data engineering practices.

- Strong analytical and problem-solving skills.

- Excellent communication and collaboration skills