Architect Databricks

Role: Architect Databricks
Position Type: Full-Time Contract (40hrs/week)
Contract Duration: 12 Months+
Work Schedule: 8 hours/day (Mon-Fri)
Work Hours: Start at 3:30 am EST
Location: 100% Remote (Candidates can work from anywhere in India)

We are seeking an experienced Databricks Architect to design and lead the implementation of a modern, enterprise-grade data platform built on the Databricks Lakehouse architecture. This role focuses on establishing architectural standards, defining governance frameworks, and driving strategic decisions that enable scalable, secure, and high-performance analytics across the organization.

Key Responsibilities

• Architecture & Strategy : Define the overall Databricks platform architecture, including workspace design, Unity Catalog governance model, storage patterns, compute strategies, and integration with broader enterprise data ecosystems.
• Governance & Security : Design and implement comprehensive governance frameworks within Unity Catalog, including multi-catalog strategies, metastore configuration, identity federation, fine-grained access controls, attribute-based access control (ABAC), data classification, and compliance requirements.
• Technical Leadership : Establish architectural patterns and best practices for medallion architecture, Delta Lake optimization, data mesh principles, pipeline orchestration, and cross-domain data sharing.
• Platform Design : Architect solutions for compute resource allocation (all-purpose clusters, job clusters, SQL warehouses, serverless), autoscaling strategies, cost optimization, and workload isolation across teams and environments.
• Data Modeling & Standards : Define enterprise data modeling standards, schema design patterns, slowly changing dimensions (SCD) strategies, CDC architectures, and data quality frameworks that scale across multiple domains.
• DevOps & Automation : Design CI/CD architecture for Databricks assets, including Git-based development workflows, automated testing frameworks, deployment pipelines, and infrastructure-as-code patterns using Terraform or similar tools.
• Performance & Optimization : Architect solutions for query optimization, storage layout strategies (partitioning, Z-ordering, liquid clustering), caching, materialized views, and monitoring/observability across the platform.
• Integration Architecture : Design integration patterns with upstream source systems (APIs, databases, streaming), downstream analytics tools (Power BI, Tableau), and adjacent cloud services (Azure Synapse, AWS services, etc.).
• Collaboration & Enablement : Partner with engineering teams, data teams, and stakeholders to translate business requirements into architectural blueprints; mentor engineers on implementation of architectural patterns.
• Innovation & Evaluation : Stay current with Databricks platform evolution, evaluate new features (Lakehouse Federation, AI/ML capabilities, streaming enhancements), and drive adoption of capabilities that deliver business value.

Required Skills & Experience

• 7+ years of experience in data architecture, data engineering, or platform engineering roles, with at least 3+ years focused on Databricks platform architecture.
• Expert-level knowledge of Databricks platform components: Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL Warehouses, MLflow, and Databricks SQL.
• Deep expertise in Unity Catalog governance, including metastore design, catalog/schema strategies, permission models, data lineage, and multi-workspace/multi-cloud patterns.
• Strong architectural background in cloud platforms (Azure, AWS, or GCP), including storage services, identity management (Azure AD, AWS IAM), networking, and security best practices.
• Proven experience designing enterprise-scale data architectures, including medallion/multi-hop architectures, data mesh patterns, domain-driven design, and data product frameworks.
• Advanced proficiency in SQL, Python/PySpark, and data modeling techniques (dimensional modeling, Data Vault, normalized schemas).
• Hands-on experience with infrastructure-as-code (Terraform, ARM templates, CloudFormation) for platform configuration and governance automation.
• Strong understanding of DevOps practices, CI/CD pipelines, version control strategies, and automated testing for data platforms.
• Experience with performance tuning, cost optimization, and capacity planning for large-scale data platforms.

Preferred Qualifications

• Databricks certification (e.g., Databricks Certified Data Engineer Professional, Solutions Architect).
• Cloud certifications such as Azure Solutions Architect, AWS Solutions Architect, or GCP Professional Data Engineer.
• Experience designing multi-tenant architectures with secure data isolation, cross-tenant data sharing, and compliance controls.
• Background in streaming architectures using Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Exposure to machine learning operations (MLOps), feature stores, model serving, and AI/ML platform architecture.
• Experience with data mesh implementations, federated governance, and distributed data ownership models.
• Knowledge of analytics platforms (Power BI, Tableau, Looker) and their integration patterns with Databricks.
• Familiarity with domain-specific data models in education (CEDS), healthcare, finance, or operational domains.
• Experience with real-time CDC patterns , change data capture tools (Debezium, Qlik, Fivetran), and event-driven architectures.

Leadership & Soft Skills

• Strategic thinking with ability to balance long-term architectural vision with pragmatic, incremental delivery.
• Exceptional communication skills to articulate complex architectural concepts to technical and non-technical stakeholders, including executive leadership.
• Proven ability to influence and drive consensus across multiple teams and organizational levels.
• Mentorship and enablement mindset to uplift engineering teams through knowledge sharing, documentation, and hands-on guidance.
• Strong problem-solving capabilities with a focus on root cause analysis and sustainable solutions.
• Commitment to quality , including comprehensive documentation, architectural decision records (ADRs), and knowledge transfer.

Position Summary

This role is ideal for a seasoned architect who thrives on designing elegant, scalable solutions and wants to shape the foundation of a next-generation Lakehouse platform. The Databricks Architect will serve as the technical authority for the platform, driving architectural excellence, governance maturity, and innovation that enables the organization to unlock the full value of its data assets across analytics, data science, and operational use cases