Lead Big Data Engineer - Remote Role
0 views
Skip to first unread message
Recruiter
10:49 AM (13 hours ago) 10:49 AM
to CORP TO CORP CONSULTANT
Lead Big Data Engineer - Remote Role
Location: Remote
Employment Type: Long-Term Opportunity
Key Skills: CDC, Databricks, Azure, Kafka, PySpark, Confluence
Position Requirements
We are looking for a Lead Big Data Engineer with strong expertise in streaming and modern data platforms to join our team and collaborate with our client, Deloitte. In this role, you will lead the design and delivery of real-time data solutions, scalable data platforms, and event-driven architectures, working closely with engineering teams, architects, and business stakeholders. You Are An experienced Lead Data Engineer / Lead Data Platform Engineer Strong in: Streaming data architecture Event-driven systems Designing real-time and batch data pipelines Hands-on with: Apache Kafka and Confluent Platform / Confluent Cloud Databricks and lakehouse architectures Change Data Capture (CDC) patterns Proficient in: Python, SQL (Scala is a plus) Experienced in building solutions using: Spark (PySpark/Scala), Delta Lake Knowledgeable in: Kafka topics, partitions, schema design Schema Registry, Kafka Connect, ksqlDB Experienced in: Cloud platforms (Azure / AWS / GCP) Data modeling and performance optimization Skilled in collaboration with: Architects Product Owners Cross-functional engineering teams A strong leader with ownership mindset and excellent communication skills You Will Define and drive data platform architecture and engineering standards Lead a team of data engineers and support technical decision-making Design and implement: End-to-end data ingestion frameworks Streaming and batch data pipelines Build and optimize: Real-time data processing solutions Event-driven systems based on Kafka ecosystem Develop and maintain data platforms on Databricks Implement CDC pipelines to replicate transactional data into analytics platforms Drive cloud migration and data platform modernization initiatives Ensure solutions meet requirements for: Scalability Reliability Security and governance Collaborate with stakeholders to align technical solutions with business needs Support continuous improvement and data-driven decision-making
🔹 Required Skills & Experience:
✔ Strong experience as a Lead Data Engineer / Data Platform Engineer
✔ Expertise in streaming data architecture and event-driven systems
✔ Hands-on experience with Apache Kafka and Confluent Platform/Cloud
✔ Deep knowledge of Databricks, Lakehouse Architecture, Spark (PySpark/Scala), and Delta Lake
✔ Strong programming skills in Python and SQL (Scala is a plus)
✔ Experience with CDC patterns and real-time data ingestion frameworks
✔ Knowledge of Kafka topics, partitions, schema design, Schema Registry, Kafka Connect, and ksqlDB
✔ Experience with Azure, AWS, or GCP cloud platforms
✔ Strong data modeling, performance optimization, and leadership skills
Location: Remote
Employment Type: Long-Term Opportunity
Key Skills: CDC, Databricks, Azure, Kafka, PySpark, Confluence
Position Requirements
We are looking for a Lead Big Data Engineer with strong expertise in streaming and modern data platforms to join our team and collaborate with our client, Deloitte. In this role, you will lead the design and delivery of real-time data solutions, scalable data platforms, and event-driven architectures, working closely with engineering teams, architects, and business stakeholders. You Are An experienced Lead Data Engineer / Lead Data Platform Engineer Strong in: Streaming data architecture Event-driven systems Designing real-time and batch data pipelines Hands-on with: Apache Kafka and Confluent Platform / Confluent Cloud Databricks and lakehouse architectures Change Data Capture (CDC) patterns Proficient in: Python, SQL (Scala is a plus) Experienced in building solutions using: Spark (PySpark/Scala), Delta Lake Knowledgeable in: Kafka topics, partitions, schema design Schema Registry, Kafka Connect, ksqlDB Experienced in: Cloud platforms (Azure / AWS / GCP) Data modeling and performance optimization Skilled in collaboration with: Architects Product Owners Cross-functional engineering teams A strong leader with ownership mindset and excellent communication skills You Will Define and drive data platform architecture and engineering standards Lead a team of data engineers and support technical decision-making Design and implement: End-to-end data ingestion frameworks Streaming and batch data pipelines Build and optimize: Real-time data processing solutions Event-driven systems based on Kafka ecosystem Develop and maintain data platforms on Databricks Implement CDC pipelines to replicate transactional data into analytics platforms Drive cloud migration and data platform modernization initiatives Ensure solutions meet requirements for: Scalability Reliability Security and governance Collaborate with stakeholders to align technical solutions with business needs Support continuous improvement and data-driven decision-making
🔹 Required Skills & Experience:
✔ Strong experience as a Lead Data Engineer / Data Platform Engineer
✔ Expertise in streaming data architecture and event-driven systems
✔ Hands-on experience with Apache Kafka and Confluent Platform/Cloud
✔ Deep knowledge of Databricks, Lakehouse Architecture, Spark (PySpark/Scala), and Delta Lake
✔ Strong programming skills in Python and SQL (Scala is a plus)
✔ Experience with CDC patterns and real-time data ingestion frameworks
✔ Knowledge of Kafka topics, partitions, schema design, Schema Registry, Kafka Connect, and ksqlDB
✔ Experience with Azure, AWS, or GCP cloud platforms
✔ Strong data modeling, performance optimization, and leadership skills
Reply all
Reply to author
Forward
0 new messages