Big Data Engineer

Innovantes is looking for an experienced Data Engineer who has designed, architected, and implemented complex data integration and processing pipelines for enterprise BI and analytics platforms.

We need a hands-on engineer with strong expertise in Python, PySpark, distributed data processing, cloud data platforms, and complex integrations across heterogeneous data sources.

The candidate must have experience translating business intelligence and reporting requirements into scalable data architectures, integrating data from multiple enterprise systems, and building reliable pipelines that operate in production.

Experience with GCP and/or Microsoft Azure is strongly preferred.

Key Responsibilities

1. Data Architecture and Pipeline Design

• Architect end-to-end data pipelines supporting enterprise BI, reporting, analytics, and data warehousing requirements.
• Design scalable data ingestion, transformation, storage, and serving architectures across multiple data sources and destinations.
• Define appropriate approaches for batch processing, incremental ingestion, CDC, historical data loads, and data synchronization.
• Design data lake and lakehouse architectures, including raw, curated, and analytics-ready data layers.
• Make technical decisions on processing frameworks, storage formats, orchestration, data modelling, and pipeline reliability.

2. Complex Data Integrations

• Build integrations across heterogeneous enterprise systems, including relational databases, cloud platforms, and legacy systems.
• Handle diverse data formats, inconsistent schemas, changing source structures, large data volumes, and complex transformation requirements.
• Implement incremental extraction, change detection, deduplication, reconciliation, and cross-source data consistency.
• Resolve integration challenges involving API pagination, rate limits, authentication, schema evolution, late-arriving data, and failed or partial loads.
• Develop reusable ingestion frameworks and standardized integration components to accelerate future implementations.

3. Cloud Data Engineering

• Design and implement cloud-based data platforms using GCP and/or Microsoft Azure.
• Develop data pipelines using technologies such as PySpark, Apache Spark, BigQuery, Cloud Storage, Dataproc, Cloud Data Fusion, Azure Data Factory, Azure Data Lake Storage, and Azure Databricks.
• Integrate cloud storage, processing engines, relational databases, and downstream analytics platforms.
• Evaluate technology choices based on scalability, maintainability, security, processing performance, and cloud costs.

4. BI and Analytics Data Platforms

• Build and maintain curated datasets and analytical data models for Power BI and other BI platforms.
• Design data structures that support reporting performance, historical analysis, dimensional modelling, and consistent business metrics.
• Understand the complete data lifecycle from source-system extraction to transformation, warehouse modelling, and BI consumption.
• Work with analytics teams to translate reporting requirements into robust, reusable data pipelines.
• Optimize data refresh strategies, incremental processing, and downstream query performance.

5. PySpark and Distributed Processing

• Develop production-grade data processing applications using Python, PySpark, and Apache Spark.
• Optimize large-scale transformations, joins, shuffles, partitioning, caching, and file layouts.
• Troubleshoot memory issues, skewed data, slow-running jobs, and distributed processing failures.
• Work with Parquet, Avro, JSON, CSV, and other structured or semi-structured formats.
• Apply sound engineering practices to build modular, testable, maintainable pipeline components.

6. Production Operations and Reliability

• Own pipelines throughout their lifecycle: design, development, deployment, monitoring, optimization, and support.
• Implement orchestration, dependency management, retries, checkpoints, idempotency, and recovery mechanisms.
• Set up logging, alerting, data quality validation, reconciliation, and failure notifications.
• Investigate production incidents and identify permanent corrective actions.
• Implement CI/CD and version control practices for controlled pipeline releases.
• Maintain technical documentation, architecture diagrams, and operational runbooks.

Mandatory Technical Experience

• Proven experience architecting and delivering complex, production-grade data pipelines.
• Strong hands-on expertise in Python, PySpark, Apache Spark, and SQL.
• Demonstrated experience integrating multiple enterprise data sources and systems.
• Experience building data platforms specifically for BI, analytics, or enterprise reporting.
• Practical experience with GCP or Microsoft Azure data engineering services.
• Strong understanding of data lake, lakehouse, data warehouse, and dimensional modelling concepts.
• Experience with orchestration, monitoring, production troubleshooting, and performance optimization.
• Ability to independently design solutions and implement them hands-on.

Preferred Experience

• Experience across both GCP and Azure.
• BigQuery, Cloud Data Fusion, Cloud Composer, Dataproc, Azure Data Factory, Azure Databricks, or Microsoft Fabric.
• CDC, streaming data, schema evolution, and large-scale incremental processing.
• Power BI data preparation and refresh architecture.

What We Are Looking For

• An engineer who understands architecture but is equally comfortable writing and debugging production code.
• Someone who has solved real-world data integration challenges.
• Someone who can independently own complex client implementations from architecture through production support.
• Strong analytical thinking, troubleshooting ability, technical communication, and engineering discipline.

Candidates will be evaluated on their ability to design, implement, optimize, and operate complete data integration solutions.