High Priority Position: Lead Databricks Engineer at SFO/Remote
Babu S
Hope you are doing great today...
This is BABU, BDM FROM PANTAR SOLUTIONS INC, we are an Information Technology and Business Consulting firm specializing in Project-based Solutions and Professional Staffing Services. Please have a look at below position which is with our Client and let me know your interest ASAP. I would really appreciate if you could send me your MOST RECENT UPDATED RESUME:
Lead Databricks Engineer
SFO / Remote
Long Term Contract
Consultant LinkedIn profile must have been created before 2018
Need 8-10+yrs of IT Exp. Profiles || Need Passport number, I94, Travel History documents during submission for TN1/H1/GC EAD/GC
Roles and Responsibilities:
· Collaborate closely with stakeholders to comprehend data requirements, contributing to the design and development of robust Databricks-based solutions aligned with organizational objectives.
· Engineer and document intricate data architectures, models, and infrastructure needs, ensuring scalability, security, and optimal performance within Databricks environments.
· Provide expert guidance and leadership to cross-functional teams, including data engineers, analysts, and scientists, ensuring adherence to best practices throughout project cycles.
· Demonstrate comprehensive expertise in Databricks, focusing on Spark-based data processing, ETL workflows, and relevant frameworks.
· Design, implement, and manage complex data pipelines, ETL processes, and data integration strategies to facilitate efficient data flow within Databricks clusters.
· Continuously monitor and fine-tune Databricks clusters, workloads, and queries to enhance performance, cost efficiency, and scalability.
· Uphold stringent security standards and compliance requisites within Databricks solutions, implementing robust access controls and encryption measures.
· Develop comprehensive technical documentation encompassing architecture diagrams, data flow representations, and procedural guides to support project teams and stakeholders.
· Serve as the primary technical liaison for clients, ensuring transparent communication of project progress, technical decisions, and offering insights into best practices.
· Collaborate closely with data engineers, scientists, and stakeholders to ensure seamless end-to-end data solutions meeting project objectives.
· Stay abreast of the latest trends and innovations in data engineering, Databricks, and related domains, recommending and implementing new tools and methodologies to enhance data processes.
Skills and Experience:
· Possession of a minimum of 5+ years of hands-on experience in data engineering and analytics.
· High Proficiency in Databricks, specializing in data processing, and analytics leveraging Spark.
· Strong grasp of data engineering concepts, encompassing data modelling, ETL processes, and data pipeline development.
· Advanced proficiency in SQL for querying, transformation, and analysis of diverse datasets.
· Expertise in Python programming for data manipulation, scripting, and automation within Databricks environments.
· Strong familiarity with Apache Airflow for orchestrating and scheduling ETL workflows.
· Knowledge of big data technologies such as Spark, Hadoop, and related frameworks.
· Understanding of data security best practices and compliance standards.
· Proficient in data modelling principles and data warehousing concepts.
· Experience with version control systems like Git.
· Exceptional communication and collaboration skills to effectively engage with cross-functional teams
PLEASE NOTE:
If for any reason this does not interest you or you felt uncomfortable by any part of this email, I sincerely apologize. Please consider this E-mail as a request for referrals and feel free in forward this email to anyone whom you might find a fit.