Data Engineer (Kafka)
Description
Location: Mexico(Remote)
Experience: 8+ years
Please note: This is an independent contractor position with a maximum monthly compensation of USD $3,400. No additional benefits are included.
Job Description
· We are seeking a Lead Kafka Integration Engineer to design, develop, and support enterprise-scale data integration solutions leveraging Kafka, Databricks, and SQL Server-based MDM platforms. The role will be responsible for architecting scalable and resilient real-time data ingestion pipelines, implementing Kafka consumer services, and delivering data through Databricks Bronze, Silver, and Gold layers.
• Design and implement end-to-end data ingestion and processing solutions from Kafka into Databricks Medallion Architecture (Bronze/Silver/Gold) and SQL Server-based MDM platforms, ensuring scalability, resiliency, and high-performance data delivery.
• Architect, develop, and support enterprise-grade Kafka consumer services including consumer groups, offset management, schema validation, retry mechanisms, dead-letter queues (DLQ), monitoring, ing, security, and automated recovery capabilities.
• Establish DevOps and operational frameworks for Kafka and Databricks using CI/CD pipelines, Infrastructure as Code (IaC), environment automation, deployment standards, observability, logging, capacity planning, and production support.
• Build and optimize Kafka-to-SQL Server ingestion pipelines, data models, stored procedures, reconciliation processes, and XML backfeed services supporting master data management, downstream integrations, and data quality controls.
• Lead architecture, development, troubleshooting, performance tuning, and governance across Kafka, Databricks, SQL Server, and cloud-native data platforms while ensuring reliability, maintainability, security, and operational excellence.
• Batch & Streaming Pipelines: Design, develop, and optimize scalable ETL/ELT workflows using Azure Databricks, Azure Data Factory, and Informatica, alongside real-time data ingestion pipelines using Apache Kafka.
• Master Data Management (MDM): Implement and maintain MDM workflows, data models, and match/merge logic to establish a reliable single source of truth across enterprise systems.
• Azure Lakehouse Management: Manage and optimize data storage, partitioning, and transformation layers within ADLS Gen2 and Delta Lake to ensure high performance and cost efficiency.
• SQL & Performance Tuning: Write complex SQL queries, perform deep data analysis, and troubleshoot bottlenecks across streaming and batch workloads.
Mandatory Skills:
· Strong hands-on experience with Apache Kafka (Consumers, Consumer Groups, Offset Management, DLQ, Schema Registry, Security)
· Expertise in Databricks, Spark, Delta Lake, and Medallion Architecture (Bronze/Silver/Gold)
· Experience building real-time and batch data ingestion pipelines from Kafka to Databricks and SQL Server
· Strong knowledge of SQL Server, Stored Procedures, Data Modeling, MDM, and Data Reconciliation processes
· Experience with CI/CD pipelines, DevOps practices, Infrastructure as Code (Terraform/Bicep), and environment automation
· Expertise in monitoring, ing, observability, performance tuning, and production support
· Experience with cloud platforms such as Azure and related data services
· Experience with Master Data Management (MDM) platforms and healthcare data environments
· Knowledge of XML processing, backfeed services, and downstream system integrations
· Experience with Kafka Connect, Event-Driven Architecture, and Streaming Analytics• Azure Data Ecosystem: Hands-on experience building and managing cloud data solutions using Azure services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), and Azure Synapse Analytics.
· Databricks & Spark: Strong proficiency in Azure Databricks using PySpark, Spark SQL, and Delta Lake for scalable data processing and lakehouse architecture.
· Real-Time Streaming (Kafka): Practical experience designing, building, and troubleshooting real-time event streaming pipelines using Apache Kafka (or Azure Event Hubs for Kafka).
· Informatica & MDM: Proven experience with Informatica (IICS / PowerCenter) and Master Data Management (MDM) concepts, including data modeling, match/merge rules, and golden record creation.
· Advanced SQL: Expert-level SQL skills for complex querying, data transformation, profiling, and performance tuning across relational and cloud data warehouses.'
Desired Skills
· Exposure to containerization technologies such as Docker and Kubernetes
· Familiarity with data governance, data quality frameworks, and security best practices
· Strong leadership, stakeholder management, and mentoring capabilities
· DevOps & Orchestration: Experience with CI/CD pipelines (Azure DevOps, GitHub Actions), Infrastructure as Code (Terraform/ARM), and workflow orchestration tools.
· Programming & Scripting: Proficiency in Python or Scala beyond PySpark for automation, API integrations, and custom data processing.
· Data Governance & Security: Familiarity with Unity Catalog, Microsoft Purview, RBAC security models, and data privacy compliance standards.
· Certifications: Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Data Engineer, or Informatica certifications.'
Skills
- Analytics
- API
- Automation
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Synapse
- Bicep
- CI/CD
- Cloud
- Cloud Native
- Containerization
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Quality
- Data Reconciliation
- Databricks
- Delta Lake
- DevOps
- Docker
- ELT
- ETL
- Event Driven Architecture
- GitHub
- GitHub Actions
- Infrastructure as Code
- Kafka
- Kubernetes
- Lakehouse
- Master Data Management
- MDM
- Medallion Architecture
- Microsoft Purview
- Observability
- PySpark
- Python
- RBAC
- Scala
- Spark
- SQL
- SQL Server
- Stakeholder Management
- Terraform
- Unity
- Unity Catalog
- Workflow Orchestration
- XML
Sign in to read and join the discussion.