Advance Data Engineer

Job summary

Position Name – Advance Data Engineer Type of hiring – Fulltime Location – Remote Canada Job Description : Must hav e:Programming Languages: Proficient in Python, PyTest and PySpark with a strong understanding software engineering best practice s.Cloud Computing: Utilize Azure cloud-based data platforms, specifically leveraging Databricks and Delta Live Tables for data engineering tasks, while effectively utilizing services related to storage, compute, and securit y.Data Pipelines: Design, build, and maintain robust and scalable and automated data pipelines for batch and streaming data inge…

Job details

Position Name – Advance Data Engineer Type of hiring – Fulltime Location – Remote Canada Job Description : Must hav e:Programming Languages: Proficient in Python, PyTest and PySpark with a strong understanding software engineering best practice s.Cloud Computing: Utilize Azure cloud-based data platforms, specifically leveraging Databricks and Delta Live Tables for data engineering tasks, while effectively utilizing services related to storage, compute, and securit y.Data Pipelines: Design, build, and maintain robust and scalable and automated data pipelines for batch and streaming data ingestion of data and processing (Data bricks workflow ).Data Architecture and Modeling: Design and implement robust data models and architectures that align with business requirements and support efficient data processing, analysis, and reportin g.Orchestration: Utilize workflow orchestration tools to automate data pipeline execution and dependency managemen t.Monitoring and Alerting: Integrate monitoring and alerting mechanisms to track pipeline health, identify performance bottlenecks, and proactively address issue s.Strong Agile principles: Utilize Agile development methodologies, actively participating in sprint planning, daily stand-ups, sprint reviews, and retrospective s.Be flexible and adaptable to changing requirements and priorities throughout the project lifecycl e.Unity Catal og Nice to Ha ve:Github Act ionDataDog expos ureData Quality: Implement data quality checks and balances throughout the data pipeline, including profiling, validation, and root cause analysis, to ensure data accuracy, completeness, and consisten cy.CI/CD: Implement continuous integration and continuous delivery (CI/CD) practices for automated testing and deployment of data pipelin es.

What you’ll do

The Advance Data Engineer will design, build, and maintain robust and scalable data pipelines for batch and streaming data ingestion. They will also implement data models and architectures that align with business requirements and support efficient data processing.

Requirements

Candidates must be proficient in Python, PyTest, and PySpark, with experience in Azure cloud platforms, particularly Databricks. Strong knowledge of data pipeline design, orchestration, and Agile methodologies is also required.

Listed skills

• Software · Preferred
• Microsoft Azure · Preferred
• CI/CD · Preferred
• Automated testing · Preferred
• analysis · Preferred
• Execution · Preferred
• Pipeline · Preferred
• Accuracy · Preferred
• Development · Preferred
• Agile · Preferred
• batch · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Python
• PyTest
• PySpark
• Azure
• Databricks
• Delta Live Tables
• Data Pipelines
• Data Architecture
• Data Modeling
• Orchestration
• Monitoring
• Agile
• Data Quality
• CI/CD