Data Engineer Staff
Posted
Summary
Data Engineer Staff at nolimit builds and runs large-scale social media crawlers, keeping incoming data complete and timely alongside a data quality team, and works with stakeholders on data requirements. Core stack: Scala, SQL, MariaDB/Postgres, Redis, Kafka, Elasticsearch, Linux, and web scraping/API data acquisition.
Building robust large-scale social media crawler for various social media platforms. Maintain day-to-day operation of said crawlers alongside our data quality team to make sure the completeness and timeliness of incoming data. Work in team along with several stakeholders to better understand and communicate about data requirements and issues.
Key responsibilities
- Building robust large-scale social media crawler for various social media platforms
- Maintain day-to-day operation of crawlers alongside data quality team
- Ensure completeness and timeliness of incoming data
- Work with stakeholders to understand and communicate data requirements and issues
About you
- Intermediate or advanced understanding of SQL
- Experience in processing large volumes of data from social media and online media platforms
- Good experience in using Scala for production (not related to Spark)
- Intermediate or advanced understanding of several data-related technologies such as MariaDB, Postgres, Redis, Kafka, and Elasticsearch
- Ample experience using Linux-based systems
- Understanding of several data acquisition techniques such as webscraping, API-based data consumption
- Fresh Graduates Welcome