Job Opening For Data Platform Engineer
Posted Updated
Description
- Design, develop, build, and maintain scalable data ingestion and transformation pipelines using Python, SQL, PySpark, and ETL/ELT frameworks.
- Develop and manage batch and streaming data processing workflows to support business and analytical requirements.
- Work with modern data platforms such as Databricks, Snowflake, Microsoft Fabric, AWS, or Google Cloud.
- Build and maintain data lakes, lakehouses, and data warehouse solutions using Delta Lake, Parquet, and other modern storage formats.
- Implement data modelling, data validation, data quality checks, and data reconciliation processes.
- Develop and optimise SQL queries, Spark jobs, and data processing workflows for improved performance.
- Implement workflow orchestration using Apache Airflow or equivalent orchestration tools.
- Work with dbt for data transformation, testing, and documentation.
- Understand Kafka fundamentals and support event-driven and real-time data ingestion requirements.
- Implement incremental data loading, partitioning strategies, and schema evolution.
- Monitor pipeline execution, troubleshoot failures, identify root causes, and ensure data pipeline reliability.