Onsite/Hybrid Job opportunity for the role of AWS Databricks Data Engineer in Los Angeles CA (Hybrid)

[Rahul Pandey]

Greetings,

This is Rahul from Quantum world Technologies; I am working as Senior Technical Recruiter in this company. I have a onsite/hybrid Job Opportunity with one of our clients. Please share your resume if you are interested in the job details given below:

Job Title: AWS Databricks Data Engineer

Location: Los Angeles CA  (Hybrid)

Job Summary

 

Key Responsibilities

1. Data Pipeline Development

• Build and maintain scalable ETL/ELT pipelines using Databricks on AWS.
• Leverage PySpark/Spark and SQL to transform and process large, complex datasets.
• Integrate data from multiple sources including S3, relational/non-relational databases, and AWS-native services.

2. Collaboration & Analysis

• Partner with downstream teams to prepare data for dashboards, analytics, and BI tools.
• Work closely with business stakeholders to understand requirements and deliver tailored, high‑quality data solutions.

3. Performance & Optimization

• Optimize Databricks workloads for cost, performance, and efficient compute utilization.
• Monitor and troubleshoot pipelines to ensure reliability, accuracy, and SLA adherence.
• Apply query optimization, Spark tuning, and shuffle minimization best practices when handling tens of millions of rows.

4. Governance & Security

• Implement and manage data governance, access control, and security policies using Unity Catalog.
• Ensure compliance with organizational and regulatory data‑handling standards.

5. Deployment & DevOps

• Use Databricks Asset Bundles for deployment of jobs, notebooks, and configuration across environments.
• Maintain effective version control of Databricks artifacts using GitLab or similar tools.
• Use CI/CD pipelines to support automated deployments and environment setups.

 

Technical Skills (Required)

• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Workflows, Delta Live Pipelines, Databricks Runtime, etc.).
• Proven ability to implement robust PySpark solutions.
• Hands‑on experience with Databricks Workflows & orchestration.
• Solid knowledge of Medallion Architecture (Bronze/Silver/Gold).
• Significant experience designing or rebuilding batch‑heavy data pipelines.
• Strong background in query optimization, performance tuning, and Spark shuffle optimization.
• Ability to handle and process tens of millions of records efficiently.
• Familiarity with Genie enablement concepts (understanding required; deep experience optional).
• Experience with CI/CD, environment setup, and Git-based development workflows.
• Solid understanding of AWS cloud, including:
  • IAM
  • Networking fundamentals
  • Storage integration (S3, Glue Catalog, etc.)

 

Preferred Experience

• Experience with Databricks Runtime configurations and advanced features.
• Knowledge of streaming frameworks such as Spark Structured Streaming.
• Experience developing real-time or near real-time data solutions.
• Exposure to GitLab pipelines or similar CI/CD systems.

 

Certifications (Optional)

• Databricks Certified Data Engineer Associate / Professional
• AWS Data Engineer or AWS Solutions Architect certification

 

 

Thanks & Regards

Rahul Pandey

rahul....@quantumworldit.com

Senior Technical Recruiter