Senior/Lead Azure Data Engineer

Job Description

We are seeking a highly skilled Senior/Lead Azure Data Engineer to design, develop, and optimize large-scale cloud-based data solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory (ADF), Azure Databricks, PySpark, and Autosys, with hands-on experience building scalable data pipelines, data transformation frameworks, and enterprise-grade data platforms.

The candidate will collaborate with business stakeholders, architects, and development teams to deliver high-performance data solutions that support analytics, reporting, regulatory, and operational requirements.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Data Factory (ADF).
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop and maintain batch and near real-time data integration workflows.
• Implement data quality, validation, reconciliation, and monitoring frameworks.
• Schedule, monitor, and troubleshoot jobs using Autosys and cloud-native orchestration tools.
• Design and implement robust ETL/ELT solutions leveraging Azure cloud services.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Architects and Solution Architects to define enterprise data models and standards.
• Support production deployments, incident management, and performance tuning activities.
• Ensure adherence to security, governance, and compliance standards.
• Mentor junior engineers and provide technical leadership for project delivery.
• Participate in code reviews, architecture discussions, and best practice implementation.

Required Qualifications

• Bachelor’s or master’s degree in computer science, Information Technology, Engineering, or related field.
• 8+ years of experience in Data Engineering and Data Warehousing.
• 5+ years of experience with Azure Data Platform technologies.
• Strong hands-on experience with:
• Azure Data Factory (ADF)
• Azure Databricks
• ADLS
• MYSQL
• PySpark
• SQL
• Autosys
• Experience designing and implementing ETL/ELT frameworks.
• Strong understanding of distributed computing and Spark architecture.
• Experience working with large-scale structured and semi-structured datasets.
• Proficiency in Python and SQL programming.
• Experience with source control tools such as Git/Azure DevOps.