Data Engineer

Must-Have Skills

• Spark, SQL & Python: Strong hands-on experience with Apache Spark (PySpark), SQL, and Python, including developing and optimizing complex data transformation pipelines.
• AWS Services: Basic working knowledge of AWS Glue, Lambda, and CloudFormation, with an understanding of how these services are used in data engineering workflows.
• Large-Scale Data Processing: Experience handling large volumes of data, optimizing data processing performance, and troubleshooting data pipeline issues.
• Data Ingestion & File Processing: Hands-on experience designing and managing file-based data ingestion pipelines, including understanding how files are received, validated, processed, and loaded into target systems.
• File Validation & Whitelisting: Understanding of file whitelisting/allowlisting mechanisms, file filtering, schema validation, and error handling to ensure only authorized and valid files are processed.

Good-to-Have Skills

• Databricks: Experience working with Databricks for large-scale data processing and transformation.
• GitHub Actions: Familiarity with CI/CD pipelines and deployment automation using GitHub Actions

JOB DESCRIPTION

• 6-8 years of hands on application to lead and perform the development with experience in one or more programming languages like Python, Pyspark etc.
• 4-6 years of hands on experience in development and deployment of cloud native solutions leveraging AWS Services: Compute(EC2, Lambda), Storage (S3), Database (RDS, Aurora, Postgres, DynamoDB), Orchestration (Apache Airflow, Step Function, SNS), ETL/Analytics(Glue, EMR, Athena, Redshift), Infra (Cloud Formation, Code Pipeline), Data Migration (AWS DataSync, AWS DMS), APIGateway, IAM etc.
• Expertise in the handling large data sets and data models in terms of design, data model creation, development of data pipeline for data ingestion, migration and transformation
• Strong on SQL Server, stored procedure
• Knowledge on API's , SSO, streaming technology will be nice to have.
• Looking for folks with Databricks expertise to build newer pipeline in DBX