Share profiles along with Job ID number
Job Summary:
The Worker is responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role supports the organization’s data engineering, machine learning, and analytics initiatives that rely on large-scale data processing.
Responsibilities:
Design and develop scalable data pipelines and ETL/ELT workflows for structured and unstructured data.
Optimize Apache Spark jobs for performance and cost efficiency.
Integrate Databricks solutions with Azure Data Factory and other Azure services.
Automate deployments using CI/CD pipelines and version control systems.
Design and maintain data models, schemas, and database structures to support analytical and operational workloads.
Evaluate and implement data lake and data warehouse solutions (e.g., Azure Data Lake Storage).
Ensure data quality, governance, and security using Unity Catalog, Delta Lake, and best practices.
Implement data validation, encryption, access controls, and auditing to maintain compliance.
Collaborate with cross-functional teams including data scientists, analysts, and stakeholders.
Troubleshoot, debug, and enhance data processing and integration workflows.
Required Skills & Qualifications:
8+ years of experience in:
Designing and implementing ETL/ELT workflows for large-scale data.
Data modeling, governance, validation, and quality assurance.
Working with Azure Cloud Platform, Azure Data Lake Storage, and Azure Data Factory.
Python, R, and SQL for data manipulation and analysis.
DevOps, CI/CD pipelines, and version control systems.
Implementing data security, encryption, and access controls.
Agile development in multicultural environments.
Troubleshooting and debugging complex data workflows.
5+ years of hands-on experience with:
Apache Spark architecture (RDDs, DataFrames, Spark SQL).
Databricks notebooks, clusters, jobs, and Delta Lake.
Performance tuning and integration with Azure services.
Preferred Qualifications:
Experience with MLflow, Scikit-learn, or TensorFlow.
Databricks Certified Associate Developer for Apache Spark.
Microsoft Certified: Azure Data Engineer Associate.
--
Thanks & Regards
Hangouts: sek...@tekwings.com / usekh...@gmail.com
Tekwings Requirements Email group : https://groups.google.com/d/forum/tekwings_requrements_group1
LinkedIn Group: https://www.linkedin.com/groups/10421204/
LinkedIn: https://www.linkedin.com/in/sekhar-u-27b11a166/