• Design & Implementation: Architecting and implementing scalable data pipelines for both batch and real-time processing, leveraging Databricks and Azure Data Factory.
• Development & Optimization: Crafting and optimizing data workflows using Python, PySpark, and SQL on the Databricks platform.
• Real-Time Solutions: Developing Spark Structured Streaming solutions for real-time data processing and analytics.
• Quality & Performance: Designing data models, ensuring data quality and governance, and monitoring/tuning Spark jobs for optimal performance.
• Mentorship & Standards: Providing guidance to junior engineers, participating in code reviews, and contributing to technical standards and best practices.

Technical Foundation:

• Experience: A minimum of 5+ years in data engineering or a related field.
• Databricks Specialist: At least 1+ year of strong, hands-on Databricks experience is essential.
• Coding Expertise: Expert-level proficiency in Python and SQL.
• Spark Proficiency: Strong hands-on experience with Apache Spark and PySpark.
• Azure Data: Experience with Azure Data Factory for orchestration and familiarity with Azure cloud services (Data Lake, Synapse, Event Hubs, etc.).

Architecture & Optimization:

• Solid understanding of Databricks concepts, including CDC, Delta Lake flow jobs, and optimizing storage patterns using Azure Data Lake.
• Understanding of data modeling, data warehousing, and data lake concepts.
• Experience with version control systems (Git) and CI/CD practices.

Great to Have (Bonus Skills):

• Experience with streaming technologies (Apache Kafka, Azure Event Hubs).
• Knowledge of machine learning workflows and MLOps practices.
• Experience with On-Prem SQL servers and related technologies.
• Azure Data Engineer Associate certification or similar cloud certifications.

Contract Details

• Title: Senior Data Engineer
• Term: 6 months Contract with a chance of extension
• Location: Auckland – requiring at least 3 days work from the office.

Apply Now

If you are a talented Senior Data Engineer seeking an immediate and challenging contract, apply now or contact Amaan for more details: call on 0220607986 or Email at amaan.kazmi@randstaddigital.co.nz.

At Randstad, we are passionate about providing equal employment opportunities and embracing diversity to the benefit of all. We actively encourage applications from any background.