Senior Data Engineer

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Engineer based in India.

We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable data platforms that power advanced analytics and data-driven business decisions. In this role, you will develop high-performance batch and real-time data pipelines using Python, SQL, PySpark, and Databricks, working with terabyte-scale datasets and distributed processing architectures. You will contribute to modern lakehouse solutions, cloud migration initiatives, and secure data management practices across major cloud platforms. You will also automate deployment workflows and infrastructure provisioning using modern DevOps tools and CI/CD practices. Collaborating with business stakeholders and cross-functional engineering teams, you will translate complex requirements into reliable, efficient technical solutions. This remote opportunity is suited to a technically strong professional who enjoys solving complex data challenges, improving platform performance, and mentoring colleagues in a fast-paced environment.

• Design, develop, and optimize scalable ETL/ELT data pipelines for batch and real-time processing using Python, SQL, PySpark, Spark Structured Streaming, and Delta Live Tables.

• Implement and maintain Medallion Architecture, organizing data into Bronze, Silver, and Gold layers to support data quality, performance, and analytical requirements.

• Process terabyte-scale datasets using distributed computing techniques, ensuring pipeline reliability, scalability, and efficient resource utilization.

• Manage Databricks environments, including workflows, clusters, Delta Lake, and related lakehouse components.

• Implement Unity Catalog for metadata management, granular access controls, OAuth-based authentication, and enterprise-level data security and governance.

• Develop and execute cloud migration strategies for legacy databases, including Oracle, Teradata, and SQL Server, to cloud-based data platforms and storage solutions such as Google Cloud Storage, Amazon S3, Azure Data Lake Storage, Snowflake, and Amazon Redshift.

• Build and maintain CI/CD pipelines using Terraform and GitHub Actions to automate infrastructure provisioning, data pipeline deployments, testing, and monitoring.

• Develop custom data processing scripts and data-driven APIs using Python and web frameworks such as Flask.

• Use Docker to containerize data processing workloads and support portable, consistent deployment environments.

• Evaluate emerging technologies and approaches for data-intensive machine learning workloads and other advanced engineering use cases.

• Collaborate with business stakeholders, data teams, and cross-functional engineering partners to translate business needs into technical architectures and actionable data solutions.

• Apply best practices for SQL query optimization, data processing efficiency, security, monitoring, and operational reliability.

• Contribute to technical problem-solving, project ownership, knowledge sharing, and mentoring to strengthen team capabilities and delivery quality.

Requirements:

• Four to seven years of professional experience in data engineering, big data development, or a closely related technical field.

• Advanced hands-on proficiency in Python, SQL, and PySpark, including expert-level SQL query optimization for large-scale datasets.

• Strong practical experience with Apache Spark, Spark Structured Streaming, Delta Live Tables, and the design of scalable batch and real-time data pipelines.

• Demonstrated experience working with terabyte-scale datasets and distributed data processing architectures.

• Extensive working knowledge of Databricks, including Workflows, Unity Catalog, Delta Lake, and lakehouse architecture.

• Experience with at least one major cloud platform: Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure.

• Familiarity with cloud data storage and analytics services, including Google Cloud Storage, Amazon S3, Amazon Redshift, Azure Data Lake Storage Gen2, and Snowflake.

• Experience migrating data from legacy database environments, such as Oracle, Teradata, or SQL Server, to modern cloud data platforms.

• Strong understanding of infrastructure automation, Terraform, GitHub Actions, Docker, and CI/CD best practices for data engineering environments.

• Practical experience developing APIs or web-based data services using Python frameworks such as Flask.

• Strong problem-solving and analytical skills, with the ability to troubleshoot complex data processing challenges and optimize system performance.

• Excellent communication and collaboration skills, with the ability to work effectively with business stakeholders and cross-functional technical teams.

• A proactive, ownership-driven mindset, with the ability to manage priorities, work in an agile environment, and mentor other team members.

• Availability to work the US business-hours shift, from 6:00 PM to 3:00 AM IST.

• Candidates who can join immediately or have a notice period of 0–15 days are preferred.

Benefits:

• Remote working arrangement within India.

• Full-time employment opportunity.

• Opportunities to work with modern data engineering technologies, including Python, Spark, Databricks, cloud platforms, and advanced data architectures.

• Exposure to large-scale data processing, cloud migration, automation, and machine learning-related engineering initiatives.

• Opportunities for professional learning, technical development, and collaboration with colleagues from diverse backgrounds.

• A workplace culture emphasizing transparency, diversity, integrity, continuous learning, and career growth.

• Specific salary details, healthcare coverage, paid leave, bonuses, and other benefits are not specified in the available job description.

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1