Middle+ Data Engineer
Posted Updated 7
views
Summary
Designs and maintains scalable data pipelines for a cybersecurity platform using Python, PySpark, and cloud infrastructure to process large datasets in real time.
Dataforest is looking for a Data Engineer to join an interesting software development project in the field of Cybersecurity. Our client’s platform offers Identity Insights drives real-time, multi-contextual digital forensics to protect new account opening workflows and expose fake accounts already in a customer database.
You'll be a great fit if you have:
- 3+ years of hands on in data engineering;
- 3+ years of commercial experience with Python;
- Advanced experience with SQL DBs (optimisations, monitoring, etc.);
- Advanced experience with PySpark;
- Solid understanding of ETL principles (architecture/ monitoring/ alerting/search and resolve bottlenecks);
- Familiar with AWS infrastructure (boto3, S3 buckets, etc);
- Experience working with large volumes of data;
- Understanding the principles of medallion architecture.
Would be a plus:
- Cassandra/Scylla
- PostgreSQL
- Pandas
- Experience with Kafka and Redis
What you'll do:
- Maintain data processing architecture using Python and PySpark that interacts with ScyllaDB & PostgreSQL.
- Work with Databricks infrastructure to ensure reliability, scalability, and cost efficiency.
- Proactively identify bottlenecks and suggest technical improvements.
We ensure your growth with:
- Working in a fast-growing company;
- Great networking opportunities with international clients, challenging tasks;
- Personal and professional development opportunities;
- Competitive salary fixed in USD;
- Paid vacation and sick leaves;
- Flexible work schedule;
- Friendly working environment with minimal hierarchy;
- Team building activities, corporate events.