Senior Data Engineer - Python/SQL
Summary
Senior Data Engineer who designs, builds, and maintains scalable data pipelines, ETL/ELT workflows, warehouses, lakes, and cloud data infrastructure. Core stack spans Python, SQL, PySpark/Scala, Spark/Hadoop/Kafka, Airflow/dbt, AWS/Azure/GCP, and warehouses like Snowflake and BigQuery, supporting batch and real-time processing for AI/ML teams.
Job Summary:
We are looking for an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines, data platforms and cloud-based data infrastructure. The ideal candidate should have strong expertise in data engineering, ETL/ELT processes, big data technologies, cloud platforms and data warehousing.
Key Responsibilities:
- Design, develop and maintain scalable data pipelines and ETL/ELT workflows.
- Build and optimize data processing frameworks for structured and unstructured data.
- Develop data integration solutions across multiple internal and external systems.
- Work with large datasets using distributed computing technologies.
- Design and maintain data warehouses, data lakes and lakehouse architectures.
- Implement batch and real-time data processing pipelines.
- Optimize data storage, processing performance and query execution.
- Collaborate with Data Scientists, AI Engineers, Business Analysts and Software Engineers.
- Ensure data quality, governance, security and compliance.
- Implement automated data pipeline monitoring and error-handling mechanisms.
- Manage cloud-based data infrastructure and deployment workflows.
- Support AI/ML teams by providing reliable, high-quality data pipelines.
Required Technical Skills:
- Programming: Python, SQL, PySpark, Scala
- Big Data Technologies: Apache Spark, Hadoop, Kafka
- ETL/ELT Tools: Apache Airflow, dbt, Informatica
- Cloud Platforms: AWS, Microsoft Azure, GCP
- Data Warehousing: Snowflake, Amazon Redshift, Google BigQuery, Azure Synapse
- Databases: PostgreSQL, MySQL, MongoDB
- Data Lakes: Delta Lake, Apache Iceberg, AWS S3
- Data Processing: Batch Processing, Stream Processing, Real-Time Analytics
- DevOps & Deployment: Docker, Kubernetes, CI/CD, Terraform
- Data Governance: Data Quality, Data Lineage, Metadata Management
Experience & Qualifications:
- 6 - 11 years of overall experience in Data Engineering or related fields.
- Minimum 4 - 5 years of hands-on experience in data engineering.
- Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering or a related field.
Preferred Certifications:
- AWS Certified Data Engineer - Associate
- Google Professional Data Engineer
- Microsoft Certified: Fabric Data Engineer Associate
- Snowflake SnowPro Core Certification
- Databricks Certified Data Engineer Associate
Skills
- AI
- Airflow
- Analytics
- AWS
- Azure
- Azure Synapse
- Big Data
- BigQuery
- CI/CD
- Cloud
- Data Engineering
- Data Governance
- Data Lake
- Data Lineage
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- dbt
- Delta Lake
- DevOps
- Distributed Computing
- Docker
- ELT
- ETL
- GCP
- Hadoop
- Iceberg
- Kafka
- Kubernetes
- Lakehouse
- Machine Learning
- Metadata Management
- MongoDB
- MySQL
- PostgreSQL
- PySpark
- Python
- Redshift
- S3
- Scala
- Snowflake
- Spark
- SQL
- Terraform