Databricks Technical Lead Job at Ender-IT, Chicago, IL

MDVhdEdHN2VWN1FVc0oxTEhCKzJJQ3M3elE9PQ==
  • Ender-IT
  • Chicago, IL

Job Description

Title: Databricks Technical Lead

Location: Chicago, IL

Duration: 12+ Months

Role description

We're looking for a Databricks Technical Lead who can guide the design and build-out of our data engineering and transformation platforms. This person will be the go-to expert on Databricks, Delta Lake, and Spark, and will help shape how data flows across our organization — from ingestion all the way through our curated layers.

This is a hands-on leadership role. You won't just review work — you'll help solve hard problems, mentor engineers, set standards, and make architecture decisions that will influence the platform for years.

What You'll Do

Lead the technical direction for Databricks-based data pipelines and frameworks.

Design and review patterns for ingesting, transforming, and publishing data (Bronze → Silver → Gold).

Define best practices around Delta Lake, schema evolution, SCD handling, and metadata-driven transformations.

Provide technical oversight across multiple engineering squads.

Work with architects, data modelers, quality engineers, and operations teams to ensure pipelines are built the right way.

Mentor data engineers and help elevate the overall engineering capability.

Oversee Unity Catalog governance, including RBAC, lineage, and schema enforcement.

Help troubleshoot complex performance issues and guide teams on tuning and optimization.

Support integration with orchestration tools and CI/CD processes.

What You Bring

Several years of hands-on Spark experience and deep expertise with Databricks (Workflows, Delta Lake, Repos, Unity Catalog).

Strong understanding of data engineering patterns, especially medallion architecture.

Comfortable designing metadata-driven systems using YAML/JSON or similar configuration styles.

Solid knowledge of AWS data services (S3, Glue, IAM, or equivalents).

Experience with structured streaming and Auto Loader is a plus.

Ability to lead and mentor engineers, give constructive feedback, and set engineering standards.

Strong communication skills — able to explain complex ideas in a clear and approachable way.

Recruiter Checklist — Databricks Technical Lead

Must-Have Skills

5+ years Databricks experience at a senior/lead level

Strong Spark (PySpark + Spark SQL), not just SQL users

Deep understanding of Delta Lake (OPTIMIZE, VACUUM, compaction, file layout)

Designed or owned a medallion architecture

Experience with schema evolution & SCD Type 1/2 handling

Hands-on Unity Catalog experience (permissions, lineage, governance)

Built or maintained metadata-driven frameworks

Streaming experience (Auto Loader / Structured Streaming)

Experience with Airflow, Glue, or Databricks Workflows

Working knowledge of cloud services (AWS)

Leadership Indicators

Has led or mentored engineering teams

Performs architecture/design reviews

Sets standards and frameworks

Communicates clearly with tech and non-tech teams

Red Flags

Only familiar with Databricks notebooks, not platform engineering

No real Spark optimization experience

No Unity Catalog exposure

Has not worked on metadata-driven patterns

Cannot explain SCD or schema evolution confidently

Job Tags

Similar Jobs

Motion Recruitment

Administrative Assistant Job at Motion Recruitment

 ...administrative staff with overflow work, including word processing, data entry and internet research tasks. Takes messages for department...  ...skills are required. Qualifications Entry-mid level experience Can-do attitude Approachable and customer service... 

Edensign

Lead Generative AI Engineer (Diffusion Models, 3D, VLM) Job at Edensign

Company Description Edensign is building the future of AI-powered visual and spatial engine. Backed by the Harvard Innovation Labs, were creating next-generation intelligent systems that merge generative AI, 3D understanding, and spatial intelligence to transform how...

Compunnel Inc.

Microbiology Technician -- PATDC5698129 Job at Compunnel Inc.

Title: Microbiology Technician Location: Athens, GA Contract duration : 12 Months Contract Job Description: Performs operations necessary for quality testing. Performs generally supervised, routine tasks, following SOPs, which may include environmental ...

Net2Source (N2S)

Sr Network Architect (Level 3 & 4 support) Job at Net2Source (N2S)

Title- - Sr Network Architect (Level 3 & 4 support) Location- Brea, CA, Need Locals, Onsite 5 days Duration: 5 months Serve as Tier 3/4 support resource (IT Escalation). On Call Rotation. Primary: Palo Alto, Cisco ISE, Wireless, SDWAN Secondary: F5, ZScaler...

Pyramid Consulting, Inc

Social Media Coordinator & Content Specialists Job at Pyramid Consulting, Inc

 ...Immediate need for a talented Social Media Coordinator & Content Specialists . This is a 05+ months contract opportunity with long-term potential and is located in Dallas TX(Hybrid). Please review the job description below and contact me ASAP if you are interested...