Databricks Developer

Key Responsibilities:

• Build production-grade AI applications using Databricks capabilities.
• Design and implement data and feature pipelines (Delta Lake, Auto Loader, Spark).
• Develop and maintain MLflow-based model training and inference systems.
• Utilize Hugging Face and foundation models, including fine-tuning when needed.
• Build production processes using Databricks Workflows and Jobs.
• Implement APIs and model serving solutions.

Maintenance and Operations:

• Participate in application change management (version control, configuration changes, controlled deployment processes).
• Participate in incident management (incident triage, root cause analysis, corrective actions).
• Monitor and optimize the performance of production processes (clusters, jobs, streaming processes).
• Implement proactive maintenance practices (monitoring, alerts, capacity management).
• Contribute to application documentation (architecture, data flows, user guides, operational procedures).

Collaboration:

• Work closely with other experts in the team.
• Participate in architecture design and the continuous improvement of the production environment.

Mandatory Skills and Experience:

• Databricks Technical Expertise: Spark (Python / PySpark / SQL), Delta Lake (optimization, schema evolution, time travel), Auto Loader and Structured Streaming, MLflow (tracking, registry, model serving), Databricks Workflows / Jobs, Git-based development (Repos).
• AI Expertise: Experience in developing and deploying machine learning models to production, experience utilizing Hugging Face models, experience with foundation model fine-tuning or RAG solutions, experience with API-based inference.
• Other Skills: Experience with Azure services.