Data Engineer (Snowflakes, Databricks)

Related Work Experience

• At least 3 years of experience in data engineering, data analytics, or related fields.

• Proficiency in Python and Java, with experience in other programming languages as an advantage.

• Hands-on experience with data manipulation and processing tools such as pandas, dplyr, and Apache Spark.

• Proficiency in ETL/ELT processes and the development, orchestration, and maintenance of data pipelines.

• Strong proficiency in SQL and experience with NoSQL databases.

• Hands-on experience with Databricks for data engineering, large-scale data processing, and pipeline development.

• Hands-on experience with Snowflake for data warehousing, data transformation, query optimization, and data management.

• Hands-on experience with Google Cloud Platform (GCP) and its data engineering services.

• Experience with big data storage and processing technologies such as Apache Spark, Hadoop, Hive, MongoDB, Snowflake, and similar platforms.

• Experience working with cloud-based or server-based data processing environments, particularly GCP, AWS, or Azure.

Knowledge

• Comprehensive understanding of data manipulation tools such as pandas, dplyr, and Apache Spark.

• In-depth knowledge of big data frameworks and technologies, including Apache Spark, Hadoop, and Databricks.

• Strong understanding of data warehousing concepts, including Snowflake architecture, data modeling, and query performance optimization.

• Knowledge of GCP data services, particularly BigQuery, Cloud Storage, Dataflow, Dataproc, and Cloud Composer.

• Familiarity with AWS data services, particularly AWS Glue and AWS Lake Formation.

• Understanding of ETL/ELT design patterns, data pipeline orchestration, and workflow automation.

• Familiarity with business intelligence tools such as Tableau, Power BI, Amazon QuickSight, or Looker Studio.

• Awareness of data visualization libraries and packages such as Dash, Plotly, Matplotlib, ggplot, and Folium.

• Understanding of machine learning libraries and tools, such as scikit-learn, caret, and MATLAB, is an advantage.

• Knowledge of data governance, data quality management, cloud security, access control, and data protection best practices.

Skills

• Problem-Solving: Ability to address complex technical challenges and deliver efficient, scalable data solutions.

• Technical Proficiency: Ability to develop and optimize data pipelines using Databricks, Snowflake, GCP, Python, SQL, and related technologies.

• Communication: Clear and concise communication skills when collaborating with team members and stakeholders.

• Collaboration: Ability to work effectively with cross-functional teams to achieve shared goals.

• Time Management: Capacity to manage tasks, prioritize deliverables, and meet deadlines.

• Attention to Detail: Careful and accurate handling of data to ensure quality, consistency, and integrity.

• Client Focus: Ability to understand business needs and align data solutions with business objectives and decision-making requirements.

• Adaptability: Willingness to learn and adapt to new tools, technologies, and processes in a rapidly changing environment.