Sr. Data Engineer - PySpark, Python & Cloudera (CDP)
Summary
Senior Data Engineer builds and optimizes PySpark pipelines on Cloudera CDP, designs ETL/ELT workflows, and ensures data quality and governance in a distributed environment.
- Build batch and distributed data processing solutions
- Conduct code reviews
- Create technical documentation
- Design data pipelines
- Develop ETL and ELT pipelines
- Develop data ingestion frameworks
- Ensure data quality and governance
- Monitor and troubleshoot production data pipelines
- Optimize Spark jobs
- Participate in agile scrum ceremonies
- Perform data profiling and validation
- Support continuous improvement