LOCAL - Senior Data Engineer – PySpark and Databricks in New York City, NY (12+ Years) - Hybrid
VINEET SHARMA
Please share resume to vineet...@hire-in.com
- US Citizen
- LinkedIn Profile
- 12+ Years
- Only local
Job Title: Senior Data Engineer – PySpark and Databricks
Location: New York City, NY (Hybrid – 3 Days Onsite)
Experience Required: Minimum 12+ Years (Strict Requirement)
Local / Nearby NYC Candidates Only
LinkedIn ID Required for Submission
Job Summary
We are seeking a highly experienced Senior Data Engineer with strong hands-on
expertise in PySpark and Databricks to support large-scale data engineering
initiatives for the Federal Reserve Board (FRB). The ideal candidate will have
a deep background in building, optimizing, and modernizing enterprise data
pipelines and distributed processing systems in a cloud environment.
This role requires 12+ years of experience in data engineering, strong
technical leadership, and the ability to collaborate with cross-functional
teams. Candidates must be able to work onsite 3 days each week in NYC.
Responsibilities:
Data Engineering & Development
Design, build, and maintain scalable, high-performance data pipelines using
PySpark and Databricks.
Develop and optimize ETL/ELT processes for structured and unstructured
datasets.
Build and enhance data ingestion frameworks, streaming pipelines, and batch
workflows.
Databricks & Spark Optimization
Utilize Databricks notebooks, Delta Lake, and Spark SQL for data
transformations.
Optimize PySpark jobs for performance, cost-efficiency, and scalability.
Troubleshoot Spark performance issues and implement best practices.
Data Architecture & Modeling
Work with architects to design data lake/lakehouse solutions.
Implement data modeling standards, schema management, and data quality
frameworks.
Maintain and improve data governance, metadata, and lineage processes.
Collaboration & Delivery
Partner with data scientists, analysts, and business teams to support
analytical requirements.
Translate business needs into technical solutions and deliver production-ready
datasets.
Participate in Agile ceremonies, sprint planning, and code reviews.
Required Skills & Qualifications
Mandatory Requirements
- 12+ years of professional experience in Data Engineering (no exceptions).
- Strong hands-on expertise in PySpark (advanced level).
- Deep proficiency with Databricks (development + optimization).
- Strong knowledge of Spark SQL, Delta Lake, and distributed data processing.
- Solid experience in ETL/ELT design, large-scale data pipelines, and performance tuning.
- Experience working in cloud environments (AWS, Azure, or GCP).
- Excellent communication and documentation skills.
Preferred Skills
Prior experience in banking, finance, or federal organizations.
Experience with CI/CD tools (Git, Jenkins, Azure DevOps).
Knowledge of data governance, security, and compliance frameworks.
Additional Information
Work Mode: Hybrid – 3 days onsite in NYC (mandatory).
Only local or nearby candidates will be considered due to onsite requirement.
Excellent opportunity to work with a major federal client on high-impact data
engineering initiatives.
|
|