Principal Data Engineer

Job Description :

As a Principal Data Engineer, you will serve as a strategic technical leader responsible for architecting, building, and evolving our enterprise-scale data infrastructure.

You will bridge the gap between high-level business strategy and deep technical execution, designing robust real-time pipelines and leading the transition toward next-generation Lakehouse architectures.

This role requires a hands-on leader who can mentor engineering teams, drive rigorous data governance, and ensure our data platforms are highly scalable, secure, and optimized for advanced analytics and machine learning.

Key Responsibilities :

- Architect, deploy, and maintain robust, fault-tolerant, and highly scalable enterprise-scale data platforms that support multi-tenant analytics and reporting.

- Lead the evaluation, selection, and implementation of cutting-edge data technologies, driving the evolution from traditional data warehouses to unified Data Lake and Lakehouse paradigms.

- Design cloud-native data architectures leveraging AWS or Azure cloud ecosystems, ensuring optimal resource provisioning, cost management, and security compliance.

- Design and implement low-latency, high-throughput real-time data ingestion and processing pipelines to support operational analytics.

- Build complex batch processing workflows handling terabyte-to-petabyte scale datasets using distributed computing frameworks.

- Diagnose and resolve performance bottlenecks across distributed systems, optimizing Spark applications, complex SQL queries, and storage layouts (e.g., partitioning, indexing, and clustering).

- Implement automated data validation, profiling, and monitoring frameworks to guarantee end-to-end data accuracy, consistency, and reliability.

- Drive data governance initiatives, establishing standards for data lineage, metadata management, retention policies, and data security (encryption, RBAC/ABAC).

- Standardize enterprise-wide workflow automation, dependency management, and scheduling using robust orchestration platforms.

- Act as a technical mentor to senior, mid-level, and junior data engineers, fostering a culture of engineering excellence, code reviews, and continuous learning.

- Collaborate closely with Product Managers, Data Scientists, Business Intelligence teams, and executive stakeholders to translate complex business requirements into scalable technical roadmaps.

Technical Skills :

- Expert-level proficiency in Apache Spark (including Spark SQL, Spark Streaming, and performance optimization).

- Strong hands-on coding expertise in Scala or Python, along with advanced, highly optimized SQL.

- Deep technical expertise with Snowflake and Databricks (including Delta Lake implementation, caching strategies, and concurrency tuning).

- Practical experience in architecting event-driven architectures using Apache Kafka (managing topics, partitions, consumer groups, and schemas).

- Extensive experience designing and managing complex directed acyclic graphs (DAGs) in Apache Airflow.

- Strong architectural footprint in either AWS (S3, EMR, Redshift, Glue, Lambda) or Azure (ADLS, ADF, Synapse, Databricks).

- Masterful understanding of data modeling techniques, including Dimensional Modeling (Kimball), Data Vault, and schema evolution strategies.

Experience :

- 9 to 15 years of dedicated experience in Data Engineering, Big Data Architecture, or Platform Engineering, with a proven track record in a product-focused or high-scale data environment.

- Bachelors or Masters degree in Computer Science, Information Technology, Software Engineering, or a related quantitative field.

- Proven experience leading architectural decisions, defining engineering best practices, and guiding engineering teams through complex delivery cycles.