Software Lead- Data Engineering

<p><span style="font-size:inherit;"><strong>Job Description: Tech Lead Engineer at GEP</strong></span></p><p><span style="font-size:inherit;"><strong>What you will do</strong></span></p><p><span style="font-size:inherit;">As a Tech Lead of Data Engineering at GEP, you will be the technical authority responsible for our global data acquisition and processing ecosystem. You will bridge the gap between business strategy and technical execution, architecting scalable web scraping systems on Azure and transforming raw data into interactive insights using Streamlit. Beyond technical mastery, you will drive engineering excellence, define technology roadmaps, and empower a high-performing team to deliver high-quality, actionable data for our global clients. You will:</span></p><ul><li><span style="font-size:inherit;"><strong>Azure Cloud Architecture:</strong> Lead the design and implementation of cloud-native data pipelines within the <strong>Azure ecosystem</strong> (Azure Functions, Data Factory, Blob Storage, and Container Instances), ensuring high availability and cost-efficiency.</span></li><li><span style="font-size:inherit;"><strong>Data Visualization & Internal Tools:</strong> Drive the development of interactive dashboards and internal data applications using <strong>Streamlit</strong> to provide stakeholders with real-time visibility into data quality, pipeline health, and market trends.</span></li><li><span style="font-size:inherit;"><strong>Technical Strategy:</strong> Architect resilient, large-scale scraping solutions and orchestration workflows using <strong>Apache Airflow</strong>, ensuring they align with global business goals and security standards.</span></li><li><span style="font-size:inherit;"><strong>Engineering Excellence:</strong> Establish and enforce best practices for Python development, CI/CD, and data modeling. Transition the team from functional code to robust, observable systems.</span></li><li><span style="font-size:inherit;"><strong>Innovation & Roadmap:</strong> Evaluate cutting-edge technologies (e.g., Spark, Databricks, LLMs for parsing) to stay ahead of anti-scraping measures and enhance data accuracy.</span></li><li><span style="font-size:inherit;"><strong>Team Growth & Mentorship:</strong> Lead the technical growth of the team, conducting deep-dive code reviews, defining technical career paths, and fostering a culture of technical excellence.</span></li><li><span style="font-size:inherit;"><strong>Governance & Compliance:</strong> Oversee data validation frameworks to ensure "one single version of the truth," maintaining compliance with global ethical standards and data privacy (GDPR/LGPD).</span></li></ul><p><span style="font-size:inherit;"><strong>What you should bring</strong></span></p><p><span style="font-size:inherit;">Education and Experience</span></p><ul><li><span style="font-size:inherit;">Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field</span></li><li><span style="font-size:inherit;"><strong>Knowledge</strong> in Data Engineering, with a strong background in designing and leading data infrastructure projects with<strong> previous experience in technical leadership or architect.</strong></span></li><li><span style="font-size:inherit;">Experience building and deploying production-grade data applications</span></li></ul><p style="margin-left:40px;"><br /><br /></p><p><span style="font-size:inherit;">• Languages</span></p><p style="margin-left:40px;"><span style="font-size:inherit;">o Fluent in English (spoken and written) is required due to collaboration with global teams. Spanish is a plus.</span></p><p><span style="font-size:inherit;"><strong>Other Skills</strong></span></p><ul><li><span style="font-size:inherit;">Azure Mastery: Deep knowledge of Azure services for data engineering and compute (e.g., Azure Container Registry, Azure Key Vault, Managed Identities).</span></li><li><span style="font-size:inherit;">Streamlit & Data Apps: Proven experience in building and deploying Streamlit applications to simplify complex data for non-technical users.</span></li><li><span style="font-size:inherit;">Expert Python: Advanced knowledge in asynchronous programming (asyncio), automation, and data processing.</span></li><li><span style="font-size:inherit;">Advanced Scraping: Mastery of Scrapy, Selenium, Playwright, and managing complex proxy rotations.</span></li><li><span style="font-size:inherit;">Orchestration: Expert-level experience with Apache Airflow for managing complex DAG dependencies and scheduling.</span></li><li><span style="font-size:inherit;">Ecosystem: Experience with Docker, Kubernetes, and CI/CD pipelines (GitHub Actions/Azure DevOps).</span></li></ul><p><span style="font-size:inherit;">Leadership & Soft Skills</span></p><ul><li><span style="font-size:inherit;">Strategic Thinking: Ability to balance technical debt with the urgency of business delivery.</span></li><li><span style="font-size:inherit;">Global Communication: Fluent in English and Portuguese (mandatory). Spanish is a plus.</span></li><li><span style="font-size:inherit;">Stakeholder Management: Exceptional ability to present technical solutions as business value to stakeholders and clients.</span></li></ul><p><span style="font-size:inherit;"><strong>Closing Statement</strong></span></p><p><span style="font-size:inherit;">As GEP experiences active and rapid growth, we’re looking for passionate individuals who want to flourish along with us — helping to pave the way to a brighter future.</span></p><p><span style="font-size:inherit;"><strong>Are you one of us?</strong></span></p><p></p><p><span style="font-size:inherit;">GEP is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, ethnicity, color, national origin, religion, sex, protected veteran status, disability status, or any other characteristics protected by federal, state or local law. We are committed to hiring and valuing a global diverse work team. GEP is proud to be </span></p>