Data Engineer
Posted Updated
Description
Data Engineer
Summary
We are seeking an experienced Data Engineer to architect, build, and maintain scalable data infrastructure within the IT Services sector. In this pivotal role, you will drive the design of robust data warehousing solutions and ensure seamless data modeling to support critical business analytics. Your expertise will be essential in managing cloud-native data pipelines, overseeing high-quality data migration strategies, and optimizing large-scale processing workflows. This position offers the opportunity to leverage cutting-edge Google Cloud technologies to transform raw data into actionable insights, directly contributing to the organization's data-driven decision-making capabilities and operational efficiency.
Responsibilities
- Design and implement comprehensive data warehousing architectures and advanced data modeling strategies to meet evolving business needs.
- Develop and maintain efficient version control workflows using Git and GitHub, integrating continuous integration and continuous deployment (CI/CD) pipelines for automated testing and deployment.
- Construct and optimize complex data processing pipelines utilizing Google Dataflow and Apache Beam for real-time and batch data transformation.
- Establish rigorous data quality frameworks and lead strategic cloud migration initiatives to ensure data integrity and security during transitions.
- Manage and scale data storage and processing environments using BigQuery and Dataproc, ensuring high performance and cost-effectiveness.
- Orchestrate scheduled workflows and dependency management through Cloud Composer (Apache Airflow) to guarantee reliable data delivery.
- Implement event-driven data ingestion and storage solutions leveraging Pub/Sub and Cloud Storage to handle high-volume data streams.
Requirements
Requirements:
- Experience: A minimum of 15 to 20 years of professional experience in data engineering and related fields.
- Core Competencies: Proven expertise in Data Warehousing and sophisticated Data Modeling techniques.
- Version Control & DevOps: Strong proficiency with Git/GitHub and the implementation of CI/CD practices.
- Big Data Processing: Hands-on experience with Google Dataflow and Apache Beam for scalable data processing.
- Data Governance: Demonstrated ability in ensuring Data Quality standards and executing successful Cloud Migration projects.
- Cloud Technologies: In-depth knowledge of Google BigQuery and Dataproc for analytics and processing.
- Workflow Orchestration: Skilled in utilizing Cloud Composer (Apache Airflow) for workflow management.
- Messaging & Storage: Familiarity with Pub/Sub and Cloud Storage for data ingestion and archival.