Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!

Big Data Engineer (PySpark / Spark / Kafka)

Toronto, ON
Hybrid
Posted Nov 4, 2026
1 position

Opens LinkedIn

Employment type: Full-time
Posting language: English
Office presence: 3 days per week
Application method: Apply on LinkedIn (LinkedIn sign-in required)

Job summary

Design, develop, and optimize scalable batch and real-time data pipelines and Kafka-based streaming applications using PySpark, Spark, Hadoop, and Apache NiFi. Build and maintain ETL/ELT workflows, troubleshoot and improve pipeline performance, and support data quality, governance, security, deployments, and production releases.

Job details

Job Description: Big Data Engineer (PySpark / Spark / Kafka) Work Mode: Hybrid – 3 Days a Week in Office Job Overview We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks. Key Responsibilities Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi. Build and maintain batch and real-time data processing solutions. Develop and support Kafka-based streaming applications and event-driven architectures. Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets. Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting. Implement data ingestion solutions from databases, APIs, files, and streaming sources. Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines. Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions. Support platform upgrades, deployments, testing, certification, and production releases. Ensure data quality, governance, security, and operational excellence across data platforms. Mandatory Skills PySpark Apache Spark (Spark SQL, DataFrames) Apache Kafka Hadoop Ecosystem (HDFS, Hive, YARN) Apache NiFi SQL Python Preferred Skills Spark Streaming Airflow / Oozie Hive Scala Jenkins, Bitbucket, Git JIRA, Confluence Cloud Platforms (GCP/AWS/Azure) Data Warehousing Concepts and Dimensional Modeling

What you’ll do

Requirements

The role requires hands-on expertise in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, Apache NiFi, SQL, and Python. It also calls for experience with large-scale data ingestion and transformation, batch and real-time processing, and collaboration on enterprise data platforms; preferred skills include streaming, workflow orchestration, cloud platforms, and data warehousing.