Databricks Expert Lead

Hays IT Contracting is a cooperation based on B2B rules. We connect IT and Engineering specialists with the most interesting technological projects on the market.

Join the group of 500 satisfied contractors working for Hays’ clients

For our client we are currently looking for candidates for the position of:

Databricks Expert / Lead

Work model: hybrid, 8 days/month at the office

Location: Kraków

Job type: B2B (with Hays Poland)

Rate: 200 – 240 PLN net/h

We’re looking for a hands-on expert who can own both the platform layer and the application layer on Databricks: designing and running robust infrastructure and enabling high quality data products on top. You’ll lead installation, configuration, governance, security, performance, and deep integration with Microsoft Azure services—as the technical authority for the Databricks platform.

Key Responsibilities:

Platform Architecture & Build

• Design, deploy, and harden Databricks workspaces in Azure (VNet injection, private endpoints/Private Link, workspace peering, access connectors) in line with our Azure Databricks Technical Standard and allowed resource providers.

• Set up clusters/instance pools, jobs & workflows, and CI/CD (Azure DevOps/GitHub) for reliable, repeatable platform operations.

Security, Governance & Compliance

• Implement Unity Catalog for data governance (metastore, catalogs, schemas, volumes), enforce cluster policies, credential passthrough, and secret/credential management with Azure Key Vault and Managed Identities.

• Align naming conventions and platform configuration to the Azure Technical Standard; document platform baselines and guardrails.

Azure Integration

• Integrate with ADLS Gen2/Blob, Azure Data Factory, Event Hubs, and observability tooling (Azure Monitor, Log Analytics, Application Insights) to support batch, streaming, and orchestration needs.

Data Products & Enablement

• Enable teams to build high quality solutions in Python, SQL, Spark; champion Delta Lake formats, ingestion frameworks, and optimization techniques across lakehouse workloads.

• Establish platform best practices, documentation, onboarding, and training.

Performance, Reliability & Cost

• Tune workloads (autoscaling, caching, shuffle strategies, runtime selection) and optimize DBU/compute use; drive capacity planning and SRE style incident/availability management.

Stakeholder Leadership

• Partner with data engineering, security, and infra teams; provide architectural guidance and technical decision records (e.g., Delta Lake vs. alternatives) and lead migration enablement when relevant.

• Experience with lakehouse migrations (e.g., Hadoop/Cloudera or Datastage to Databricks) and related tooling decisions (streaming with Kafka/Flink, ingestion via Lakebridge, etc.).

Required Qualifications:

• 10+ years in data/platform engineering; 3–5+ years administering Databricks at enterprise scale (production, multi workspace, multi tenant).

• Deep hands on with Unity Catalog, Delta Lake, Jobs/Workflows, Repos, cluster policies, secret scopes; strong Python/SQL/Spark.

• Proven Azure expertise: VNets, Private Link, NSGs, Key Vault, Managed Identities, ADLS Gen2, Data Factory, Event Hubs, Monitor/Log Analytics.

• Infrastructure as Code (Terraform/Bicep/ARM) and CI/CD (Azure DevOps/GitHub) for platform automation.

• Strong grounding in security/governance and compliance practices; design with standards first (Azure Databricks Technical Standard).

• Excellent communication; able to lead designs, reviews, and cross team enablement.

Nice to Have

• Certifications: Databricks Certified Data Engineer Professional/Administrator, Azure Solutions Architect Expert, Azure Security Engineer.

Hays Poland sp. z o.o. is an employment agency registered in a registry kept by Marshal of the Mazowieckie Voivodeship under the number 361.