Data Engineer - Databricks

Data Engineer (Databricks) - Job Description

Job Title: Data Engineer (Databricks | Python/PySpark | SQL)

Experience: 5 Years

Location: Hyderabad

Key Responsibilities:

- Hands-on experience in data-related activities such as data parsing, cleansing, quality definition, data pipelines, storage and ETL/ELT scripts.

- Expert knowledge of Databricks Lakehouse implementation and support (Delta Lake, Unity Catalog, Auto Loader, Lakeflow, Databricks SQL).

- Strong programming experience in Python, PySpark and SQL.

- Hands-on experience with data migration, including moving legacy warehouses, Hadoop or Spark workloads to Databricks.

- Experience consuming REST APIs using various authentication options (OAuth, API keys, tokens) within Databricks notebooks and jobs.

- Build, orchestrate, schedule and debug batch and streaming jobs using Lakeflow Jobs and Lakeflow Spark Declarative Pipelines.

- Design Medallion (Bronze, Silver, Gold) architectures, including data models, storage layout and effective queries.

- Optimize Spark jobs and Delta tables for performance and cost, including cluster sizing and query tuning.

- Understanding of Databricks security features such as Unity Catalog permissions, service principals and secret scopes.

- Knowledge of DevOps tools and CI/CD processes, including Databricks Asset Bundles.

- Databricks certification is highly preferred (Data Engineer Associate or Data Engineer Professional).

Technical Skills & Expertise:

Databricks Platform Expertise:

- Delta Lake: ACID transactions, schema enforcement and evolution, time travel, MERGE/upserts, OPTIMIZE, VACUUM, partitioning and liquid clustering.

- Apache Spark / PySpark: DataFrame and Spark SQL development, joins and shuffle tuning, caching, reading Spark UI and query plans, and handling data skew.

- Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables): Building batch and streaming pipelines, streaming tables, materialized views, data quality expectations and CDC with AUTO CDC.

- Lakeflow Jobs (Workflows): Creating multi-task jobs, dependencies, triggers and schedules, retries, alerts, and job monitoring.

- Lakeflow Connect and Auto Loader: Incremental file ingestion from cloud storage, schema inference and managed connectors for databases and SaaS applications.

- Structured Streaming: Streaming ingestion from Kafka or Kinesis, checkpoints, watermarks, and exactly-once processing with Delta tables.

- Unity Catalog: Catalogs, schemas and volumes, access control, data lineage, audit logs, and data sharing through Delta Sharing.

- Databricks SQL: SQL warehouses, creating tables and views, query tuning, and supporting BI dashboards.

- Compute Management: Job and all-purpose clusters, serverless compute, autoscaling, cluster policies, Photon, and cost monitoring through system tables.

Cloud (AWS) Expertise:

- S3 (Simple Storage Service): Data lake storage, partitioning, lifecycle policies, encryption at rest and in transit, and external locations for Unity Catalog.

- IAM: Roles and policies, instance profiles, and cross-account access for Databricks workspaces.

- VPC (Virtual Private Cloud): Awareness of customer-managed VPCs, subnets, NAT gateways, peering, PrivateLink and network ACLs for Databricks deployments.

- Lambda and API Gateway: Event-driven functions and REST APIs that trigger or support Databricks jobs.

- CloudWatch and CloudTrail: Monitoring, alerting, audit logging, and integration with third-party SIEM tools.

CI/CD & DevOps Tools:

- GitHub / GitHub Actions: Managing version control, branching strategies, and automating workflows for testing and deployment.

- Databricks Asset Bundles and Git Folders: Packaging jobs and pipelines as code, Git integration in workspaces, and promotion across dev, test and production.

- Jenkins / CloudBees: Building pipelines for build-test-deploy automation, plugin management, parallel execution, and integrations with SCM and artifact repositories.

- SonarQube: Static code analysis, security vulnerability checks, technical debt reporting, and integration into CI/CD.

- Terraform: Provisioning Databricks workspaces, clusters, jobs and permissions as infrastructure as code.

DevOps/MLOps & AI/ML Awareness:

- Understanding of the ML model lifecycle: data preprocessing, training, evaluation, deployment and monitoring.

- Experience supporting Data Scientists and ML Engineers in deploying models to production.

- Familiarity with MLflow experiment tracking, Unity Catalog model registry and versioning, Feature Store, Mosaic AI Model Serving, and Airflow (optional).

Technical Skills Summary:

- Core: Databricks, Apache Spark, PySpark, Python, SQL, Data Engineering, ETL, ELT, Data Pipelines, Data Migration, Data Modeling, Data Quality

- Databricks Platform: Delta Lake, Lakehouse Architecture, Medallion Architecture, Unity Catalog, Lakeflow, Lakeflow Jobs, Lakeflow Connect, Spark Declarative Pipelines, Delta Live Tables (DLT), Auto Loader, Structured Streaming, Databricks SQL, Delta Sharing, Photon, Serverless Compute

- Cloud (AWS): AWS, Amazon S3, IAM, VPC, AWS Lambda, API Gateway, Amazon CloudWatch, AWS CloudTrail

- Streaming and Integration: Apache Kafka, Amazon Kinesis, REST APIs, OAuth, Change Data Capture (CDC)

- DevOps and CI/CD: CI/CD, Git, GitHub, GitHub Actions, Databricks Asset Bundles, Jenkins, CloudBees, SonarQube, Terraform

- ML and Orchestration: MLflow, Feature Store, Mosaic AI Model Serving, MLOps, Apache Airflow

- Certifications (Preferred): Databricks Certified Data Engineer Associate, Databricks Certified Data Engineer Professional