Data Engineer

Job Summary Design, develop, and maintain scalable data infrastructure and pipelines to support data-driven products. Collaborate cross-functionally to ensure data accuracy, integration, and compliance in a cloud-based big data environment. Responsibilities Design, develop, and deploy data tables, views, and marts across data warehouses, operational data stores, data lakes, and data virtualization platforms to support business needs Extract, clean, transform, and manage data flows, including web scraping techniques, to ensure high data quality and availability Build, launch, and maintain efficient large-scale batch and real-time data pipelines using data processing frameworks for reliable data delivery Integrate and consolidate disparate data silos in scalable and compliant ways to enable unified data access Collaborate with project managers, data architects, business analysts, frontend developers, designers, and data analysts to build scalable, data-driven products Develop backend APIs and manage databases to support application functionality and performance Apply Agile methodologies with continuous integration and delivery to accelerate development cycles Engage in pair programming and code reviews to maintain code quality and knowledge sharing Required competencies and certifications Proficient in data cleaning and transformation using tools such as SQL, pandas, or R to ensure data accuracy and consistency Skilled in building ETL pipelines with technologies like SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container tasks, Event bridge, AWS Glue, or Spring Experienced in database design and management across various systems including SQL, PostgreSQL, AWS S3, Athena, MongoDB, Postgres/GIS, MySQL, SQLite, Volt DB, and Cassandra Knowledgeable in cloud platforms such as AWS, Azure, or Google Cloud for big data engineering Experienced in building production-grade data pipelines and ETL/ELT data integration workflows Understanding of system design, data structures, and algorithms to optimize data processing Familiar with data modeling, data access methods, and storage infrastructures like Data Mart, Data Lake, Data Virtualization, and Data Warehouse for efficient data retrieval Familiar with REST APIs and web protocols to support data integration and application communication Familiar with big data frameworks and tools such as Hadoop, Spark, Kafka, and RabbitMQ Experienced with web scraping technologies including Beautiful Soup, Casper.JS, Phantom.JS, Selenium, and Node.js for customized data extraction Knowledgeable about data governance policies, access control, and security best practices to ensure data compliance Comfortable scripting in languages such as SQL and Python for automation and data manipulation Proficient in both Windows and Linux development environments for versatile deployment and testing Demonstrates interest in bridging engineering and analytics teams to enhance data-driven decision-making