Key Responsibilities
- Design, develop, and maintain scalable data pipelines for batch and real-time data processing using Azure (preferred) or AWS cloud platforms.
- Build and optimize ETL processes to ensure high-quality, secure, and efficient data flow across systems.
- Collaborate with cross-functional teams to translate business requirements into robust data models and solutions.
- Implement data quality, data governance, and data security standards throughout the data lifecycle.
- Develop and maintain documentation, including design and mapping documents.
- Conduct code reviews, unit testing, and peer reviews to ensure code quality and compliance.
Requirements
- 5–7 years of experience in data engineering, with a strong background in building and managing large-scale data pipelines.
- Hands-on experience with Azure Data Services (Data Factory, Data Lake, Synapse, Databricks, etc.) is preferred.
- Proficiency in SQL, Python, or Scala for data processing and transformation.
- Experience with data warehousing (e.g., Snowflake, SQL Server, MongoDB) and real-time databases.
- Strong understanding of data architecture, data ingestion, curation, and consumption patterns.