Senior Data Software Engineer (Azure, Spark)
Description
We are looking for a Senior Data Software Engineer to contribute to the design and development of a modern enterprise data platform focused on large-scale financial data processing. The platform is being built to deliver new capabilities rather than maintain legacy systems, with a strong emphasis on data quality, governance, and scalable processing across cloud-native technologies. The role involves working on data ingestion, transformation, modeling, and optimization within a medallion architecture, supporting both structured and complex datasets, including private markets data.
Responsibilities
- Lead the design, development, and optimization of scalable data engineering solutions on Azure, using Spark-based processing (PySpark, SparkSQL, Scala, or equivalent)
- Own end-to-end data transformation pipelines, including ingestion, transformation, storage, and analytics
- Work with Azure-native data services such as Data Factory, Databricks, and Synapse
- Support high-performance data access patterns using Cosmos DB (NoSQL API) where applicable
- Collaborate with data scientists, AI engineers, and product stakeholders to enable data-driven analytics and insights
- Mentor and guide junior engineers, setting coding standards and best practices
- Ensure data quality, security, governance, and performance across platforms
- Contribute to technical decision-making and solution architecture discussions
- Build and maintain data pipelines within a medallion architecture, supporting structured and complex datasets
- Apply CI/CD practices to automate validation and deployment of data pipelines
Requirements
- 5+ years of experience in data engineering, with a focus on large-scale data processing
- Proficiency in Microsoft Fabric, Apache Spark, and Python
- Skills in Pandas, Delta Lake, and Parquet for data processing, storage, and optimization
- Knowledge of Cosmos DB for bitemporal data storage and retrieval
- Familiarity with AWS Glue for supporting ETL workflows
- Expertise in Spark Streaming for near real-time processing with watermarking support
- Competency in CI/CD tools for building automated validation pipelines
- Background in mentoring engineers and establishing coding standards and best practices
- English proficiency at B2 level or higher
Nice to have
- Hands-on experience with Microsoft Fabric and OneLake
- Expertise in Cosmos DB (NoSQL API) specifically
- Familiarity with financial instruments and financial services data
- Familiarity with AI-assisted development tools such as GitHub Copilot and awareness of industry-standard LLMs
- Understanding of Data Science fundamentals and collaboration experience with DS teams
Benefits
We gather like-minded people:
- Top tech minds driving innovation in AI, cloud and digital platform modernization
- Supportive team and agile, startup-like culture
- Hybrid by design mode and opportunity to work remotely within Poland
- Chance to work abroad for up to 60 days annually
- Business-driven relocation opportunities
We provide growth opportunities:
- Career development programs
- Thought leadership, mentoring, soft skills and well-being programs
- Certification (Anthropic, Gemini, GCP, Azure, AWS)
- English classes
We cover it all:
- Stable pay
- Participation in the Employee Stock Purchase Plan with a 15% discount
- Benefits package (health insurance, multisport, shopping vouchers)
- Referral bonuses up to $2,000
- Offices featuring entertainment and relaxation zones, table tennis and football, free snacks, coffee and more
- Corporate, social and well-being events
Please, note:
- Benefits listed above are available to employees only
- We are open for working with Contractors. Terms of B2B cooperation agreements are agreed individually
- We will reach out to selected candidates exclusively
Skills
- Agile
- AI
- Analytics
- Anthropic
- API
- AWS
- Aws Glue
- Azure
- Azure Data Factory
- CI/CD
- Cloud
- Cloud Native
- Cosmos Db
- Data Engineering
- Data Ingestion
- Data Pipelines
- Data Quality
- Data Science
- Databricks
- Delta Lake
- ETL
- GCP
- GitHub
- Github Copilot
- LLM
- Medallion Architecture
- Microsoft Fabric
- NoSQL
- pandas
- Parquet
- PySpark
- Python
- Scala
- Spark
Sign in to read and join the discussion.