Software Engineer – Data Engineering
Summary
Designs and maintains real-time and batch data pipelines on Microsoft Azure using Databricks, Data Factory, Event Hubs, Spark, and PySpark, while also contributing to backend services, APIs, and software architecture. Requires 3+ years in software or data engineering with strong Python, PySpark, and SQL skills.
The Software Engineer – Data Engineering at Strichpunkt Development will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data engineering solutions, with a primary focus on building real-time and batch data pipelines using Microsoft Azure technologies.
The role involves developing data ingestion, transformation, and processing workflows that handle large volumes of structured and unstructured data. The engineer will work extensively with Azure Databricks, Azure Data Factory, Azure Event Hubs, Apache Spark, and PySpark to build modern data platforms and streaming solutions.
In addition to data engineering responsibilities, the engineer will contribute to backend development, API integrations, software architecture, and distributed systems. The role requires close collaboration with software engineers, data architects, and business stakeholders to translate business requirements into robust technical solutions.
The engineer will also be responsible for optimizing data processing performance, ensuring data quality and reliability, implementing monitoring and error-handling mechanisms, and following best practices for security, scalability, and maintainability.
Key Responsibilities
- Design, develop, and maintain scalable real-time and batch data pipelines on Microsoft Azure.
- Build and maintain streaming solutions using Azure Event Hubs, Azure Stream Analytics, and Azure Databricks.
- Develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
- Implement real-time data ingestion, transformation, and processing using Apache Spark Structured Streaming and PySpark.
- Integrate data from REST APIs, relational databases, NoSQL databases, event streams, and external systems.
- Work with Azure Cosmos DB and PostgreSQL, including direct integration with Azure Databricks.
- Design and optimize data storage solutions using Azure Data Lake Storage Gen2, Azure SQL, and Delta Lake.
- Implement data quality checks, monitoring, logging, error handling, and automated recovery mechanisms.
- Apply data modeling and lakehouse architecture principles to support analytical and operational workloads.
- Ensure data security, governance, integrity, and compliance.
- Optimize data pipelines for performance, reliability, scalability, and cost efficiency.
- Develop and maintain backend services and APIs required for data integration and processing.
- Participate in software architecture discussions, code reviews, testing, and continuous improvement initiatives.
- Collaborate with cross-functional teams to deliver reliable data and software solutions.
Required Skills & Experience
- 3+ years of professional experience in Software Engineering, Data Engineering, or a related technical field.
- Strong hands-on experience with Microsoft Azure data services.
- Proven experience designing and implementing real-time streaming and batch data pipelines.
- Strong proficiency in Python, PySpark, and SQL.
- Practical experience with Azure Databricks, Azure Data Factory, and Azure Event Hubs.
- Experience integrating Azure Cosmos DB and PostgreSQL with Azure Databricks.
- Good understanding of Apache Kafka, Spark Structured Streaming, and event-driven architectures.
- Experience with Delta Lake, data lakehouse architectures, and data modeling.
- Understanding of event-time processing, windowing, checkpointing, and handling late-arriving data.
- Experience with data integration, transformation, and large-scale distributed processing.
- Familiarity with CI/CD pipelines, Git, and Azure DevOps.
- Strong understanding of software engineering fundamentals, including OOP, design patterns, API development, and system design.
- Knowledge of software testing practices, including unit and integration testing.
- Strong analytical, troubleshooting, and problem-solving skills.
- Good English communication skills and ability to work effectively in a remote team.
Nice to Have
- Experience with Microsoft Fabric and Real-Time Intelligence.
- Experience with Azure Functions and Azure Service Bus.
- Knowledge of Kubernetes, Docker, and Infrastructure as Code (Terraform/Bicep).
- Experience with Power BI and real-time analytics dashboards.
- Familiarity with Microsoft Purview and Azure data governance.
- Experience with cloud-native architectures and distributed systems.
- Microsoft Azure certifications (DP-203 or equivalent).
Skills
- Analytics
- API
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Functions
- Azure Service Bus
- Bicep
- CI/CD
- Cloud
- Cloud Native
- Cosmos Db
- Data Engineering
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Databricks
- Delta Lake
- Design Patterns
- DevOps
- Distributed Systems
- Docker
- ELT
- ETL
- Event Driven Architecture
- Git
- Infrastructure as Code
- Integration Testing
- Kafka
- Kubernetes
- Lakehouse
- Microsoft Fabric
- Microsoft Purview
- NoSQL
- OOP
- PostgreSQL
- Power BI
- PySpark
- Python
- RDBMS
- REST
- Spark
- SQL
- Terraform
Sign in to read and join the discussion.