Hybrid: Lead Data Engineer - Databricks/AWS (Locals only)

Key Responsibilities:

• Experience: 6 to 10 years Realtime experience on databricks is must.
• Collaborate as part of a development team to design and enhance large scale applications developed using Python, Spark & Pyspark .
• Realtime experience on databricks is must.
• Evaluates and plans software designs, test results and technical manuals using AWS.
• Confer with business units and development staff to understand both the business and technical requirements for producing technical solutions.
• Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
• Extend and enhance the business Data Lake.
• Create or implement solutions for metadata management.
• Solve for complex data integrations across multiple systems.
• Design and execute strategies for real-time data analysis and decisioning.
• Build robust data processing pipelines using AWS Services and integrate with multiple data sources.
• Translating client user requirements into data flows, data mapping, etc.
• Analyses and determines data integration needs and follows Agile practices.

Required Skills:

• At least 4+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python.
• At least 4 years of experience with Python, Spark & Pyspark.
• At least 3 years of experience working on AWS technologies.
• Experience of designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lamda, Kinesis using databases Aurora and Redshift.
• Experience with Spark programming (Pyspark or scala).
• Hands on experience with AWS components like (EMR, S3, Redshift, Lamdba, API Gateway, Kinesis ) in production environments.
• Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.
• Experience using ETL tools for data ingestion.
• Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle and DB.
• Ability to translate data needs into detailed functional and technical designs for development, testing and implementation.