Sr. Data Engineer
Posted Updated
Description
Responsibilities:
- Contribute to the architecture, design, and growth of our Data Products and Data pipelines in Scala and Python / Spark while maintaining uptime SLAs.
- Develop scalable solutions, building ETL pipelines in Big Data environments (cloud, on-prem, hybrid).
- Implement the Lakehouse architecture, working with key partners to shift towards a Lakehouse-centric data platform.
- Collaborate with Data Product Managers, Data Architects, and Data Engineers to design, implement, and deliver successful data solutions.
- Maintain detailed documentation of your work and changes to support data quality and data governance.
- Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to our customers (Data Science, Data Analytics teams).
- Be an active participant and advocate of agile/scrum practice to ensure health and process improvements for your team.
- Be a problem solver; when presented with new challenges, you are expected to research and network to find solutions.
- Seek out answers to business problems and look for opportunities to automate processes and optimize costs.
- Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and
- incremental platform improvements.
Requirements:
- Bachelor's degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or a comparable field of study, and/or equivalent work experience.
- At least 5 years of data engineering experience developing large data pipelines.
- Strong hands-on Experience with Cloud technologies like AWS (S3 EMR, EC2) and building data pipelines.
- Strong algorithmic problem-solving expertise.
- Strong SQL skills and ability to create queries to extract data and build performant datasets.
- Hands-on experience with distributed systems such as Spark, Hadoop (HDFS, Hive, Presto, PySpark) to query and process data.
- Strong fundamental Scala and Python programming skills.
- Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, BigQuery).
- Solid experience with data integration toolsets (i. e, Airflow) and writing and maintaining Data Pipelines with Databricks.
- Experience in Data Modeling techniques and Data Warehousing standard methodologies and practices.
- Familiar with Scrum and Agile methodologies.
- Demonstrated excellent interpersonal and communication skills, including the ability to partner with others and build consensus in a cross-functional team toward a desired outcome.
Preferred Qualifications:
- Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, Big Query).
- Experience with Software Application development.
- Good data acumen to understand domain /functional knowledge and to be able to articulate analysis of data points in a business context for stakeholders.
- Experience working with a test-first mentality.
- Deep Understanding of AWS or other cloud providers, as well as infrastructure as code.
Tech stack:
- AWS
- Snowflake
- Spark
- Databricks
- Delta Lake
- Airflow
- Python
- Scala
Skills
- Agile
- Airflow
- Analytics
- AWS
- Big Data
- BigQuery
- Cloud
- Data Analytics
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Data Quality
- Data Science
- Data Warehousing
- Databricks
- Delta Lake
- Distributed Systems
- EC2
- ETL
- Hadoop
- Hive
- Infrastructure as Code
- Lakehouse
- PySpark
- Python
- Redshift
- S3
- Scala
- Scrum
- Snowflake
- Spark
- SQL
Sign in to read and join the discussion.