Azure Data Engineer

Job summary

Design, build, and maintain scalable automated batch and streaming data pipelines using Azure, Databricks, and Delta Live Tables. Develop data architectures and models, orchestrate and monitor pipeline execution, implement data quality and CI/CD practices, and participate in Agile delivery.

Job details

Advance Data Engineer REMOTE Full Time This job posting is for an existing, active vacancy and We are looking to hire Advance Data Engineer in Canada (WFH), immediately who has strong experience in Python, PyTest, PySpark , Databricks and Delta Live Tables. Role Overview: Mandatory Skill: Programming Languages: Proficient in Python, PyTest and PySpark with a strong understanding software engineering best practices. Oops Concept: Strong Knowledge of It. Cloud Computing: Utilize Azure cloud-based data platforms, specifically leveraging Databricks and Delta Live Tables for data engineering tasks, while effectively utilizing services related to storage, compute, and security. Data Pipelines: Design, build, and maintain robust and scalable and automated data pipelines for batch and streaming data ingestion of data and processing (Data bricks workflow). Data Architecture and Modeling: Design and implement robust data models and architectures that align with business requirements and support efficient data processing, analysis, and reporting. Orchestration: Utilize workflow orchestration tools to automate data pipeline execution and dependency management. Monitoring and Alerting: Integrate monitoring and alerting mechanisms to track pipeline health, identify performance bottlenecks, and proactively address issues. Strong Agile principles: Utilize Agile development methodologies, actively participating in sprint planning, daily stand-ups, sprint reviews, and retrospectives. Be flexible and adaptable to changing requirements and priorities throughout the project lifecycle. Unity Catalog Good to Have: Github Actiom Datagog exposure Data Quality: Implement data quality checks and balances throughout the data pipeline, including profiling, validation, and root cause analysis, to ensure data accuracy, completeness, and consistency. CI/CD: Implement continuous integration and continuous delivery (CI/CD) practices for automated testing and deployment of data pipelines.

What you’ll do

Design, build, and maintain scalable automated batch and streaming data pipelines using Azure, Databricks, and Delta Live Tables. Develop data architectures and models, orchestrate and monitor pipeline execution, implement data quality and CI/CD practices, and participate in Agile delivery.

Requirements

The role requires strong proficiency in Python, PyTest, and PySpark, along with software engineering best practices and experience with Azure data platforms, Databricks, and Delta Live Tables. It also calls for knowledge of data modeling, orchestration, monitoring, data quality, CI/CD, and Agile practices; Unity Catalog is listed as a good-to-have skill.

Listed skills

• Microsoft Azure · Preferred
• CI/CD · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Python
• PyTest
• PySpark
• Azure
• Databricks
• Delta Live Tables
• Data Pipeline Engineering
• Batch And Streaming Data Processing
• Data Architecture And Modeling
• Workflow Orchestration
• Monitoring And Alerting
• Agile Methodologies
• Unity Catalog
• GitHub Actions
• Data Quality
• CI/CD

Job areas

• Technology
• Data & Analytics
• Software
• Engineering

More jobs from Ardent SoftSol Inc.