Senior/Lead Big Data Engineer (with Python experience)
Summary
Design and build scalable cloud-based data processing systems (batch/streaming) using Python/Java/Scala, ensuring high performance and alignment with business needs.
We are looking for an experienced data engineer with design and
development experience in automating scalable and high-performance data processing systems (batch and/or streaming) on the cloud (AWS preferably).
What You’ll Do
● Design our data models for optimal storage and retrieval on the cloud and to meet critical product and business requirements
● Build scalable and highly-performant distributed data processing systems as we migrate to the cloud
● Work closely with our business stakeholders to flesh out and deliver on requirements in an agile manner
● Set and evolve data standards and best practices
● Contribute to the data architecture and align it with the business and technology
● Adhere to and enforce software development best practices on the cloud in areas including but not limited to CI/CD, code reviews, automated testing, operational excellence, data quality, etc.
What You Should Have
● 4+ years of programming experience in Java/Scala/Python
● 2+ years experience using an enterprise cloud-based solution
● 2+ years implementing data processing pipelines on the cloud (batch and/or streaming)
● Advanced understanding of SQL, relational, and NoSQL databases required
● Experience with various data access patterns, streaming technology, data quality, data modeling, data performance, and cost optimization