itjobs.ca Logo
AceStack logo

Principal Databricks Data Engineer / Toronto, ON (Onsite) - FTE

AceStack1 day ago
Toronto, Ontario, Canada
Senior Level
Full-Time

About the role

Role: Principal Databricks Data Engineer Location: Toronto, ON (Onsite) Experience: 10+ years Type: Full Time Permanent

Key Requirements ** 12–18 years of overall data engineering experienc e 8+ years of experience in enterprise Data Warehouse and Data Lake platform s* 5+ years of hands-on experience with Databricks and Spark at scal e* Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehous e* Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lak e* Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipeline s* Ensure data reconciliation, audit integrity, and consistency during migratio n* Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architecture s* Implement layered architecture including Raw/Landing, Curated/Conformed, and Semantic/Consumption layer s* Modernize traditional EDW platforms into scalable lakehouse architecture s* Strong experience in finance and risk data models including General Ledger, Sub-ledger, financial hierarchies, and risk exposure models (credit, liquidity, market risk )* Enable reporting use cases including aggregation, drill-down, and drill-back capabilitie s* Build and manage semantic/consumption layers for BI, reporting, and analytic s* Define business metrics, dimensions, hierarchies, and KPI s* Experience with Databricks SQL, Delta tables, and dbt or similar framework s* Develop and optimize large-scale data pipelines using PySpark, Spark SQL, and Delta Lak e* Implement Medallion architecture (Bronze, Silver, Gold layers )* Optimize workloads using Z-ORDER, OPTIMIZE, caching, and cluster configuration s* Implement data governance, data quality frameworks, reconciliation controls, and exception handlin g* Establish data lineage and metadata managemen t* Ensure data security, access control, and compliance standard s* Experience with cloud platforms such as AWS or Azur e* Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOp s* Familiarity with orchestration tools such as Airflow or Databricks Workflow s* Experience with dbt is a plu s* Act as a technical authority and lead architecture decision s* Mentor and guide senior engineers and establish engineering standard s* Strong stakeholder management with finance, risk, analytics, and governance team s* Ability to translate complex data structures into business-ready insight

s Nice to Have *** Experience in BFSI, Capital Markets, or regulatory reporti ng Exposure to SAP Finance, Oracle Financials, or S/4HA NA Experience supporting AI/ML workloa ds* Databricks or cloud certificatio

ns

About AceStack

Software Development