Key Responsibilities
- Develop and maintain web crawling and data extraction systems
- Optimize automation pipelines for performance and reliability
- Design scalable solutions for large-scale data processing
- Collaborate with data teams to ensure data quality and integrity
- Implement best practices in error handling and monitoring
Requirements
- 5+ years of experience in automation and web crawling
- Proficiency in Python and frameworks like Scrapy or BeautifulSoup
- Experience with distributed systems and parallel processing
- Knowledge of HTML, CSS, and JavaScript for web interaction
- Strong problem-solving skills and attention to detail