Senior Data Engineer (Big Data Pipelines)
Summary
Senior Data Engineer for a data analytics firm (hired via placement agency Placements24): design, build, and maintain scalable batch and real-time data pipelines, warehouses, and lakes on cloud platforms for analytics and ML. Fully remote; stack includes Spark/Hadoop, SQL, Python/Scala/Java, and Airflow/Prefect.
About the Role
Our client, a cutting-edge data analytics firm, is seeking a highly skilled Senior Data Engineer to join their distributed, fully remote team. This role is critical for designing, building, and maintaining robust data pipelines and infrastructure that support the company's advanced analytics and machine learning initiatives. You will work with vast datasets, implement efficient data processing solutions, and ensure data quality and accessibility for data scientists and analysts. As a remote-first organization, strong communication, self-discipline, and experience with collaborative remote tools are paramount. This is an exceptional opportunity to work with state-of-the-art big data technologies from anywhere.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines for batch and real-time data processing.
- Build and maintain data warehousing solutions and data lakes on cloud platforms.
- Implement ETL/ELT processes to ingest, transform, and load data from various sources.
- Ensure data quality, integrity, and reliability across all data systems.
- Collaborate with data scientists and analysts to understand their data requirements and provide efficient data access.
- Develop and maintain data infrastructure, monitoring performance and proactively addressing issues.
- Implement data governance best practices and ensure compliance with data privacy regulations.
- Automate data processes and workflows using scripting and orchestration tools.
- Evaluate and integrate new data technologies and tools to improve data processing capabilities.
- Participate in code reviews and contribute to architectural decisions for data platforms.
- Effectively communicate technical solutions and data insights within a remote team environment.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
- Minimum of 6 years of experience in data engineering or a related role.
- Proven experience in designing and building large-scale data pipelines using big data technologies (e.g., Spark, Hadoop, Hive).
- Proficiency in SQL and experience with various relational and NoSQL databases.
- Experience with cloud data platforms (e.g., AWS EMR, Redshift, Glue; Azure Data Factory, Synapse; GCP Dataflow, BigQuery).
- Strong programming skills in Python, Scala, or Java.
- Experience with data warehousing concepts and ETL/ELT tools.
- Familiarity with data orchestration tools like Airflow or Prefect.
- Understanding of data modeling and database design principles.
- Excellent problem-solving, analytical, and debugging skills.
- Strong communication and collaboration skills, essential for remote work.
- Demonstrated ability to work independently and manage priorities effectively in a remote setting.
Benefits
Our client offers a highly competitive salary, comprehensive health benefits, performance bonuses, and ample opportunities for professional development, including advanced training and certifications. This is a fully remote position, allowing for excellent work-life balance and flexibility. The company cultivates a strong remote work culture with regular virtual team meetings, collaborative online platforms, and opportunities for significant career growth. Join a leading data-driven organization and build the future of data infrastructure from anywhere.