Data Platform Engineer
Description
Responsibilities:
Design, develop, and maintain scalable and reliable data pipelines to support our growing data needs.
Build and optimize ETL processes for data extraction, transformation, and loading from various sources.
Develop and maintain data warehouse solutions, data lakes, and other data storage systems.
Implement data quality checks and validation processes to ensure data accuracy and consistency.
Collaborate with data scientists and analysts to understand their data requirements and provide them with the data they need.
Monitor and troubleshoot data pipeline performance and identify areas for optimization.
Implement data security measures and ensure compliance with data governance policies.
Evaluate and implement new data technologies and tools to improve our data infrastructure.
Create and maintain data documentation, including data dictionaries, data flow diagrams, and data models.
Automate data processes to improve efficiency and reduce manual errors.
Qualifications:
Bachelor's degree in Computer Science, Data Science, or a related field.
Strong programming skills in languages such as Python, Java, or Scala.
Proficiency in SQL and experience with relational databases (PostgreSQL, MySQL).
Experience with ETL tools and technologies (Apache NiFi, Talend, Informatica).
Experience with big data technologies (Hadoop, Spark, Kafka).
Familiarity with cloud platforms (AWS, Azure, Google Cloud).
Solid understanding of data warehousing concepts and data modeling techniques.
Experience with NoSQL databases (MongoDB, Cassandra) is a plus.
Excellent problem-solving and analytical skills.
Strong communication and collaboration skills.
Ability to work independently and as part of a team.
Strong understanding of data security and data governance.
Nice to Have:
Computer Science, Data Science, or a related field.
Experience with data visualization tools (Tableau, Power BI, Superset).
Experience with machine learning concepts and tools.
Experience with DevOps practices and tools (Jenkins, Docker, Kubernetes).
Skills
- AWS
- Azure
- Big Data
- Cassandra
- Cloud
- Data Governance
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Science
- Data Visualization
- Data Warehousing
- DevOps
- Docker
- ETL
- GCP
- Hadoop
- Java
- Jenkins
- Kafka
- Kubernetes
- Machine Learning
- MongoDB
- MySQL
- NiFi
- NoSQL
- PostgreSQL
- Power BI
- Python
- RDBMS
- Scala
- Spark
- SQL
- Superset
- Tableau
- Talend