Sr. Data Engineer
Summary
Build and maintain scalable data pipelines in Python and PySpark on GCP, integrating diverse data sources and optimizing performance for analytics and compliance.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Python and PySpark
Build and optimize batch and near‑real‑time data processing workflows
Work on GCP‑based data platforms, integrating multiple structured and semi‑structured data sources
Develop data transformations, validations, and enrichment logic aligned to business requirements
Optimize Spark jobs for performance, scalability, and cost efficiency
Collaborate with data architects, platform teams, and downstream consumers
Implement logging, monitoring, and error‑handling within data pipelines
Ensure adherence to enterprise data governance, security, and compliance standards
Support production deployments and provide L2/L3 support as needed
Good to Have / Preferred Skills
Knowledge of Kafka or other messaging systems
Experience with Airflow / Cloud Composer
Containerization and orchestration exposure (Docker / Kubernetes)
CI/CD exposure for data pipelines
Familiarity with data warehousing and analytics use cases
Prior experience in financial services / regulated environments
na