Sr Data Engineer -

Job summary

Design and optimize scalable data and machine learning platforms using Databricks and Spark. This includes owning end-to-end data pipelines, ML workflows, and implementing RAG and LLM-based solutions.

Job details

Job Summary We are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems. Key Responsibilities Design and implement scalable ETL pipelines using Databricks & Spark Build Lakehouse architecture using Delta Lake Develop and deploy ML models using MLflow Implement MLOps pipelines for training, testing, and serving models Optimize cluster performance and reduce compute cost Build RAG and LLM-based solutions using Mosaic AI Integrate analytics with BI tools (Power BI, Tableau) Implement data governance using Unity Catalog Collaborate with Data Scientists and Business teams Ensure data quality, security, and compliance Required Skills Mandatory 5+ years of Databricks & Apache Spark Strong Python & PySpark Experience with Delta Lake & Lakehouse MLflow & MLOps experience Cloud platform (AWS/Azure/GCP) Git & CI/CD Preferred Experience with LLMs & Generative AI RAG pipelines & Vector Databases Deep Learning frameworks Databricks certifications Power BI integration Requirements Job Summary We are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems. Key Responsibilities Design and implement scalable ETL pipelines using Databricks & Spark Build Lakehouse architecture using Delta Lake Develop and deploy ML models using MLflow Implement MLOps pipelines for training, testing, and serving models Optimize cluster performance and reduce compute cost Build RAG and LLM-based solutions using Mosaic AI Integrate analytics with BI tools (Power BI, Tableau) Implement data governance using Unity Catalog Collaborate with Data Scientists and Business teams Ensure data quality, security, and compliance Required Skills Mandatory 5+ years of Databricks & Apache Spark Strong Python & PySpark Experience with Delta Lake & Lakehouse MLflow & MLOps experience Cloud platform (AWS/Azure/GCP) Git & CI/CD Preferred Experience with LLMs & Generative AI RAG pipelines & Vector Databases Deep Learning frameworks Databricks certifications Power BI integration

What you’ll do

Design and optimize scalable data and machine learning platforms using Databricks and Spark. This includes owning end-to-end data pipelines, ML workflows, and implementing RAG and LLM-based solutions.

Requirements

Requires over 5 years of experience with Databricks, Apache Spark, and Python. Candidates should be proficient in Delta Lake, MLOps, and cloud platforms, with a preference for experience in Generative AI.

Listed skills

• Microsoft Azure · Preferred
• Power BI · Preferred
• Production · Preferred
• CI/CD · Preferred
• Data analysis · Preferred
• Compliance · Preferred
• Machine learning · Preferred
• Amazon Web Services · Preferred
• Google Cloud · Preferred
• Integration · Preferred
• Git · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Databricks
• Apache Spark
• Python
• PySpark
• Delta Lake
• MLflow
• MLOps
• AWS
• Azure
• GCP
• Git
• CI/CD
• LLMs
• Generative AI
• RAG
• Vector Databases

Job areas

• Data & Analytics
• Technology
• Software
• Engineering
• Consulting

More jobs from iLink Digital