Senior Data Engineer
Description
Data Engineer
Atlanta, GA (Hybrid)
- We are currently seeking a Data Engineer experienced with cloud conversion/migration, with a preference for Teradata to AWS migrationThe Engineer must have Knowledge of Databricks.
- Must have demonstrable hands-on experience in creating/designing/implementing end-to-end data pipeline solutions. (Data ingestion, Data extraction, Data transformation, Data enrichment, Data loading, Data validation, Data processing, Data monitoring and maintenance, Data governance and security)
- In-depth understanding of ETL and data ingestion processes used in large-scale data warehouses
Position Overview: The Data Engineer is a skilled professional responsible for creating end-to-end data pipelines, managing data ingestion, transformation, enrichment, loading, validation, monitoring, and maintenance processes. This role plays a crucial part in building and optimizing data architecture, ensuring seamless data flow between various systems, and supporting data-driven decision-making. The Data Engineer should possess expertise in cloud migration, Databricks, ETL processes, and data governance, particularly for large-scale data warehouses.
Key Responsibilities:
- End-to-End Data Pipelines: Design, implement, and maintain robust data pipelines that cover the entire data lifecycle, from data ingestion to data loading, and ensure the efficient flow of data through the entire process.
- Data Ingestion: Develop and manage data ingestion processes to collect data from different sources, such as databases, APIs, logs, and other data repositories, ensuring data is collected in a reliable and scalable manner.
- Data Transformation: Perform data transformations and data enrichment to convert raw data into a structured and usable format, preparing it for storage and analysis.
- Data Loading: Implement data loading procedures to efficiently store processed data into data warehouses, data lakes, or other storage systems, ensuring data quality and integrity.
- Data Validation: Design and implement data validation mechanisms to verify data accuracy, consistency, and completeness, ensuring the data is reliable for analytical purposes.
- Data Processing: Utilize various data processing technologies and frameworks to handle large-scale data volumes efficiently and effectively.
- Data Monitoring and Maintenance: Establish monitoring processes to ensure data pipelines run smoothly, and proactively address any issues that arise. Regularly maintain and optimize data pipelines for better performance and scalability.
- Cloud Migration: Plan and execute cloud migration strategies, transferring data and applications to cloud-based platforms while ensuring security, scalability, and cost-efficiency.
- Databricks Expertise: Leverage Databricks and Apache Spark to build scalable and high-performance data processing workflows, providing insights and analytics on large datasets.
- ETL and Data Ingestion Processes: Understand and implement ETL (Extract, Transform, Load) processes used in large-scale data warehouses, ensuring efficient data movement and integration.
- Data Governance and Security: Implement data governance best practices and security measures to protect sensitive data and ensure compliance with data privacy regulations.
Qualifications and Skills:
- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
- Proven experience as a Data Engineer or similar role with a strong focus on data pipeline development.
- Expertise in creating end-to-end data pipelines, including data ingestion, extraction, transformation, loading, and validation.
- Proficiency in cloud migration, particularly with cloud platforms like AWS, Azure, or Google Cloud.
- Experience with Databricks and Apache Spark for large-scale data processing and analytics.
- In-depth knowledge of ETL processes and best practices used in large-scale data warehouses.
- Knowledge of big data platforms such as Hadoop, Spark, HBase, and Teradata
- SQL Databases: Oracle DB, DB2 - On-Prem and AWS (preferred)
- Strong programming skills in languages like Python, SQL, or Java for data manipulation and scripting.
- Knowledge of distributed data storage systems, data lakes, and data warehousing concepts.
- Excellent problem-solving and analytical skills, with an ability to handle complex data-related challenges.
- Strong communication and collaboration skills to work effectively with cross-functional teams.
Nice to have
- Precisely Connect
- Fivetran Transformations
- Prophecy
As a Data Engineer, the individual plays a pivotal role in designing and implementing robust data pipelines and facilitating smooth data integration, empowering organizations to harness the power of data for informed decision-making and business growth.
Benefits (employee contribution):
- Health insurance
- Health savings account
- Dental Insurance
- Vision insurance
- Flexible spending accounts
- Life insurance
- Retirement plan
All qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.
Skills
Sign in to read and join the discussion.