Data Engineering and Cloud
Summary
Designs and maintains data pipelines and cloud infrastructure for autonomous driving projects, using Python, Spark, Kafka, and Azure to process sensor and simulation data.
Responsibilities:
- Design, develop, and maintain scalable and efficient data pipelines and infrastructure to support autonomous driving research and development.
- Extract, transform, and load (ETL) data from various sources, including sensors, simulations, and databases.
- Collaborate with data scientists, researchers, and engineers to understand data requirements and translate them into technical solutions.
- Develop and optimize data storage and retrieval strategies, ensuring data quality and integrity.
- Implement data governance and security measures to protect sensitive data.
- Monitor and troubleshoot data pipelines and systems to identify and resolve issues promptly.
- Continuously explore and evaluate new data technologies and tools to improve data processing efficiency and scalability.
- Mentor and guide junior data engineers in their professional development.
Mandatory Core Skills & Competencies:
- Strong proficiency in Python or another programming language commonly used in data engineering.
- Expertise in data engineering tools and frameworks, such as Apache Spark, Hadoop, Kafka, and Airflow.
- Deep understanding of data warehousing and data lake concepts.
- Experience with cloud platform (Azure) and cloud-based data services.
- Knowledge of SQL and NoSQL databases.
- Excellent problem-solving and analytical skills.
- Ability to work independently and as part of a cross-functional team.
- Strong communication and documentation skills.
Nice to Have Skills:
- Experience with machine learning and deep learning frameworks (TensorFlow, PyTorch).
- Familiarity with autonomous driving technologies and concepts.
- Knowledge of data visualization tools (e.g., Tableau, Matplotlib).
- Experience with real-time data processing and streaming technologies.
- Understanding of data quality and validation techniques.
- Experience with containerization technologies (Docker, Kubernetes).
- Familiarity with CI/CD pipelines and DevOps practices.
- Experience with data governance and compliance frameworks (e.g., GDPR, CCPA).
- Ph.D. or Master's degree in Computer Science, Data Engineering, or a related field.
- Publications in relevant conferences or journals.
BE M Tech CS
6+ years