Key Responsibilities
- Design, develop, and maintain scalable big data pipelines
- Process and analyze large datasets using Hadoop and Spark
- Optimize data storage and retrieval for performance and cost efficiency
- Collaborate with data scientists to implement machine learning models
- Ensure data integrity, security, and compliance with regulations
- Monitor and troubleshoot data processing workflows
Requirements
- 5+ years of experience in big data engineering
- Proficiency in Python, Hadoop, and Spark
- Strong understanding of SQL and data warehousing concepts
- Experience with cloud platforms like AWS
- Problem-solving skills and ability to work with large-scale datasets