Databricks Engineer / Senior Data Engineer --- Plano, TX & Columbus, OH
Shekhar K
Job Location : Plano, TX & Columbus, OH
Job Duration : Long Term Contract
Required Experience : 10+ years
Job Description:
We are seeking an experienced Databricks Engineer with strong expertise in modern data engineering, large-scale data migration, and Databricks Lakehouse architecture. The ideal candidate will have hands-on experience migrating PySpark and Hive workloads from AWS EMR or legacy Hadoop platforms to Databricks while ensuring data quality, governance, security, scalability, and performance.
The candidate should possess strong experience with PySpark, Spark SQL, Delta Lake, Unity Catalog, Spark Declarative Pipelines (SDP), and enterprise-scale data engineering best practices.
Key Responsibilities :
• Design, develop, and maintain scalable data pipelines using Databricks.
• Develop modern ETL/ELT solutions using PySpark and Spark SQL.
• Lead migration initiatives from AWS EMR and legacy Hadoop environments to Databricks Lakehouse.
• Migrate PySpark workloads from AWS EMR to Databricks.
• Convert Hive-based ETL jobs to Databricks using Delta Lake.
• Modernize legacy data warehouse and Hadoop ecosystems into Databricks Lakehouse Architecture.
• Convert and optimize Hive SQL, Spark SQL, and PySpark workloads.
• Build scalable and reusable data transformation frameworks.
• Develop and optimize Spark Declarative Pipelines (SDP).
• Perform end-to-end data validation, reconciliation, and quality assurance.
• Design and implement enterprise data quality frameworks.
• Optimize Databricks jobs for performance, scalability, reliability, and cost efficiency.
• Work closely with architects, data engineers, analysts, and business stakeholders.
• Manage CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure compliance with enterprise governance, security, and data management standards.
Required Qualifications :
• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.
• 10+ years of Data Engineering experience.
• 3+ years of hands-on Databricks experience.
• Strong programming skills in Python and SQL.
• Strong understanding of Spark architecture and distributed computing.
• Experience working in Agile/Scrum environments.
Preferred Qualifications :
• Databricks Certified Data Engineer (Associate or Professional)
• Experience with enterprise data governance and cataloging
• Knowledge of Lakehouse Architecture
• Experience with multi-cloud environments
• Strong understanding of enterprise security best practices
Share the Below Required Mandatory Details & Resume :
- Legal Name (First/Last):
- Phone (Primary and secondary):
- Candidate Email:
- Current Location (City, State):
- Work Authorization / Visa Status :
- Interview Availability Time Slots:
- LinkedIn URL:
- Education Details ( Bachelors / Masters , University Name , Location, Year of Pass out ) :
- Availability once Confirmed:
- Total years of Work Experience in USA :
- Over All Years of Work Experience :
- Open to Relocate ( Yes / No ) :
- Expected Bill rate C2C Hourly :
- Employer Contact Information :
