Senior Software Developer

Role & responsibilities

• Design, develop, and maintain scalable web scraping and crawling solutions using Python.
• Extract data from complex websites, including static, dynamic, and JavaScript-rendered websites.
• Work with tools and frameworks such as Scrapy, Selenium, Playwright, BeautifulSoup, and Requests.
• Handle website anti-bot mechanisms, CAPTCHA challenges, IP blocking, proxies, cookies, and session management.
• Develop and integrate REST APIs for data extraction and processing.
• Analyze website structures, network requests, API endpoints, and JSON responses.
• Optimize scraping scripts for performance, scalability, and reliability.
• Troubleshoot crawling failures, identify root causes, and implement permanent fixes.
• Manage large-scale crawling jobs, scheduling, monitoring, and error handling.
• Ensure data accuracy, completeness, and consistency through appropriate validation mechanisms.
• Collaborate with QA and operations teams for smooth project execution.
• Provide technical guidance and support to junior developers.

Preferred candidate profile

• Experience in large-scale e-commerce website crawling.
• Knowledge of retail websites and product data extraction.
• Knowledge of cloud environments and distributed crawling.
• Understanding of website request/response cycles and browser developer tools.
• Ability to independently handle end-to-end scraping projects.
• Strong debugging and analytical skills.
• Good problem-solving and troubleshooting abilities.
• Ability to estimate development timelines and meet delivery commitments.
• Ownership of assigned projects and production issues.
• Ability to guide junior team members and share technical knowledge.