Senior Data Engineer
Posted Updated
Description
Coforge is seeking a Senior Databricks Data Engineer in Brisbane, Australia to design, develop, and support enterprise data pipelines on AWS. The role combines platform engineering with Data BAU operations, ensuring SLA compliance, incident resolution, and continual improvement of production data ecosystems.
You will work with Unity Catalog, Delta Live Tables, serverless compute, and DBT, with experience in ITIL/ITSM practices and ticketing tools to manage incidents and changes.
Ensure Data BAU operational support for production data pipelines and data quality alerts. Apply ITIL/ITSM practices to Incident, Problem, Change, and Release management. Track SLAs and incidents using ServiceNow or Jira Service Management and document RCAs. Maintain and optimize Databricks workstreams and lakehouse pipelines in AWS (S3). Implement Unity Catalog, DLT, serverless compute, and dashboards to support analytics. Tune performance with Liquid Clustering, Z-Ordering, and Spark optimization. Manage Databricks assets with DABs and IaC via Terraform. Leverage dbt for transformations and Kafka-based streaming data pipelines. Collaborate on GitLab CI/CD pipelines and GitLab for source control. Work in Unix/Linux environments and ensure platform stability and SLA compliance. Strong hands-on experience in Data BAU operational support, managing production data pipelines, data freshness, and data quality alerts. In-depth understanding of ITIL frameworks and ITSM practices (Incident, Problem, Change, and Release management). Experience using ITSM ticketing tools (e.g., ServiceNow, Jira Service Management) to track SLAs, manage high priority/impact incidents, and document RCAs. Hands-on experience with AWS services such as S3. Advanced proficiency in Databricks Development work using pyspark and features: Unity Catalog, Delta Live Tables, Serverless compute, SQL Warehouse, dashboards, alerts. Good experience with performance tuning using Liquid Clustering, Z-Ordering, and Spark optimization. Practical experience managing workloads with Databricks Asset Bundles (DABs). Advanced experience with dbt for version-controlled data transformations. Kafka-based streaming data pipelines for real-time data integration. Proven use of Terraform for IaC to provision AWS and Databricks resources. Production-grade experience with GitLab for source control and CI/CD pipelines. Good experience with Unix/Linux environment. Preferred: Airline/Aviation domain experience; Databricks certifications.Skills
Sign in to read and join the discussion.