As a Senior Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems using Apache Spark and other big data technologies. You will work closely with cross-functional teams to develop and implement data governance policies and ensure data quality and integrity. You will also be responsible for implementing CI/CD pipelines to automate data processing and deployment. Experience with cloud-based data platforms, such as AWS, is highly desirable. The ideal candidate will have a strong background in software engineering and a passion for working with big data technologies.
Key Responsibilities:
- Design and develop large-scale data processing systems using Apache Spark and other big data technologies.
- Collaborate with cross-functional teams to develop and implement data governance policies.
- Implement CI/CD pipelines to automate data processing and deployment.
- Ensure data quality and integrity by implementing data validation and quality control processes.
- Work closely with data scientists and analysts to understand business requirements and develop data solutions.
Requirements:
- 5+ years of experience in software engineering with a focus on big data technologies.
- Strong background in Apache Spark and other big data technologies.
- Experience with cloud-based data platforms, such as AWS.
- Strong understanding of data governance policies and data quality control processes.
- Excellent communication and collaboration skills.