Hiring Databricks Architect
Srikanth
Job Title: Databricks Architect
Location: [Remote/Hybrid – US]
Experience: 14+ Years
Work Authorization: USC/GC
Job Summary:
We are seeking a highly experienced Databricks Architect to lead the design and implementation of a scalable Lakehouse architecture supporting payer analytics, regulatory reporting, and advanced healthcare use cases. This role will play a critical part in enabling data-driven decision-making across value-based care, risk adjustment, and quality measures such as HEDIS.
The ideal candidate will have deep expertise in Databricks, Apache Spark, AWS cloud services, and healthcare data standards, along with a strong focus on data governance, security, and performance optimization.
Key Responsibilities:
- Lead the design and implementation of Databricks Lakehouse architecture for payer analytics and reporting
- Architect scalable and high-performance data pipelines using Apache Spark and Delta Lake integrated with AWS services
- Enable advanced analytics use cases including value-based care, risk adjustment, HEDIS measures, and population health
- Leverage Gen AI and Agentic AI capabilities to accelerate code development, testing, and deployment processes
- Ensure data security, governance, and compliance (HIPAA, PHI) using Unity Catalog, encryption, and private networking
- Define and enforce data architecture standards, best practices, and governance frameworks
- Collaborate with business and analytics teams to translate requirements into robust data solutions
- Optimize performance, scalability, reliability, and cloud cost efficiency for large-scale healthcare data workloads
- Provide technical leadership, mentorship, and guidance to engineering teams
- Drive CI/CD, automation, and DevOps practices within the data platform
Required Skills & Qualifications:
- 10+ years of experience in data engineering, data architecture, or related roles
- Strong hands-on experience with Databricks Lakehouse platform
- Deep expertise in Apache Spark (PySpark/Scala/SQL) and Delta Lake
- Extensive experience with AWS services (S3, Glue, Lambda, Redshift, etc.)
- Strong understanding of data governance and security frameworks, including HIPAA and PHI handling
- Experience with Unity Catalog and secure data access patterns
- Familiarity with healthcare data standards such as FHIR, HL7, EDI (X12)
- Experience designing and optimizing large-scale distributed data systems
- Strong problem-solving, communication, and stakeholder management skills
Preferred / Good to Have:
- Experience implementing AWS HealthLake, Clinical Data Platforms, or Healthcare Data Lakes
- Exposure to Terminology Servers and Master Patient Index (MPI) solutions
- Experience with real-time/streaming pipelines (Kafka, Kinesis)
- Knowledge of data quality frameworks and observability tools
- Certifications in Databricks or AWS
- Experience working in payer/provider healthcare environments
Key Competencies:
- Strategic thinking and architecture design
- Leadership and mentoring
- Stakeholder collaboration
- Innovation with emerging technologies (AI/ML, GenAI)
- Strong focus on data quality, compliance, and governance