Job Opening for Senior Data Engineer – Databricks ETL/ELT
Sindhu Jayabal
Hi,
Hope you are doing great. I am Sindhu, Technical Recruiter from Intellisoft Technologies. Please take a look at the JD below and if you are interested in this opportunity, please reply with your updated resume.
Job Title: : Senior Data Engineer – Databricks ETL/ELT
Duration: Contract
Location: Austin, TX
Job Description:
We are looking for a hands-on Senior Data Engineer to design, build, and operationalize ETL/ELT pipelines in Azure Databricks for manufacturing data and analytics use cases. This role will focus on scalable ingestion, transformation, curation, and delivery of trusted data products that support reporting, analytics, and downstream AI/ML solutions. The work aligns with GM’s current direction of modernizing legacy data pipelines into cloud-native Databricks patterns, using governed Bronze/Silver/Gold data models, CI/CD automation, and standardized operational practices.
The role
As a Senior Data Engineer, you will be responsible for design, development, testing, deployment, and automation of intelligent manufacturing data pipelines across cloud and hybrid environments. You will work across architecture, product, systems, manufacturing, and platform teams to ensure reliable and scalable delivery of high-quality data products. This is a technical leadership role with strong hands-on responsibility, helping define engineering patterns, improve delivery speed and quality, and enable long-term modernization of GM manufacturing data platforms.
What this role will do
• Design and develop ETL/ELT pipelines in Azure Databricks to support ingestion, transformation, and curation of manufacturing data.
• Build reusable Bronze, Silver, and Gold data pipelines and help establish scalable medallion architecture patterns for manufacturing use cases.
• Enable ingestion from a variety of enterprise and plant-facing sources using the appropriate pattern, including managed connectors, DLT, Auto Loader, and structured streaming where needed.
• Partner with architecture, product, manufacturing, and analytics teams to implement ingestion, curation, and downstream consumption pipelines using Python, Scala, Spark, SQL, and Databricks-native capabilities.
• Build CI/CD automation for Databricks data products, including code validation, deployment workflows, code review readiness, and operational handoff.
• Optimize reliability, observability, and production readiness of pipelines through testing, monitoring, alerting, documentation, and runbook-driven operations.
• Support migration from legacy Hadoop, Kafka, and on-prem ETL assets into standardized cloud-native Databricks and Event Hub patterns.
• Participate in operational support and on-call rotation as part of sustaining a production data platform.
Key responsibilities
• Design, build, and maintain scalable batch and streaming data pipelines in Databricks.
• Implement ingestion and transformation pipelines for structured and semi-structured data from manufacturing and enterprise systems.
• Develop reusable frameworks for ETL/ELT, data quality, schema evolution, and curated data products.
• Build and support Databricks jobs, workflows, and deployment bundles across development, test, and production environments.
• Collaborate with stakeholders to define data contracts, model data for consumption, and deliver governed tables and volumes in Unity Catalog.
• Improve performance and cost efficiency of Spark and Databricks workloads through tuning, partitioning, and engineering best practices.
• Mentor other engineers and help raise the team’s overall engineering maturity in Databricks, CI/CD, and production operations.
• Model GM core values and help drive execution, accountability, and cross-functional delivery.
Required qualifications
• Bachelor’s degree in Computer Science, Software Engineering, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience.
• Strong experience building and supporting ETL/ELT pipelines and data platforms in cloud environments.
• Strong proficiency in Python, SQL, Spark, and Databricks for data engineering use cases.
• Experience with Azure Databricks, including notebooks, jobs/workflows, Unity Catalog, and production deployment patterns.
• Experience with CI/CD pipelines, GitHub-based workflows, and automated deployment of data products.
• Strong understanding of medallion architecture, data ingestion patterns, and batch/streaming tradeoffs.
• Experience working across architecture, product, and business teams to deliver reliable production data pipelines.
• Strong troubleshooting, communication, and technical leadership skills.
Preferred qualifications
• Experience with Scala, Java, Azure Event Hub, Kafka, or Pulsar in enterprise data pipeline environments.
• Experience with DLT, Auto Loader, structured streaming, and metadata-driven pipeline frameworks in Databricks.
• Experience with monitoring, observability, support operations, and production readiness reviews.
• Experience modernizing legacy ETL jobs or migrating on-prem data assets to cloud-native platforms.
• Familiarity with manufacturing data, plant-floor systems, industrial telemetry, or manufacturing analytics use cases.
• Experience working with globally distributed teams and multiple concurrent initiatives.
Thanks & Regards
Sindhu
IT-Recruiter,
Intellisoft Technologies Inc.,
Office: 972 756 1212 X 177,
11494 Luna Road, Ste 280
Farmers Branch, TX -75234
Confidentiality & Disclaimer: The information contained in this electronic message along with any attachments is private and confidential and is intended for the exclusive use of the addressee(s). If you are not the intended recipient and/or have received this communication in error, please notify the sender immediately and then delete from system and destroy all copies of this message along with any attachments. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited and may be unlawful. The contents of this email are subject to any terms and conditions agreed between Intellisoft Technologies inc and the recipient, and do not necessarily represent the views, opinions or policies of Intellisoft Technologies inc