Only Local to NC, Lead Data Engineer
P R
Location: Charlotte, NC (Onsite), Only Locals
Duration: Long-term Contract
Job Description:
• We are seeking a Lead Data Engineer to design, architect, and implement robust data platforms within the AWS ecosystem. The ideal candidate will have strong leadership skills and extensive hands-on experience in ETL development, big data processing, and cloud-based data engineering, with a deep understanding of RDBMS systems and data formats.
Key Responsibilities:
• Lead the design, development, and deployment of data pipelines using AWS Glue, PySpark, and Python.
• Architect scalable data lake and data warehouse solutions on AWS.
• Work with RDBMS systems such as PostgreSQL and SQL Server for data extraction and transformation.
• Manage and optimize data ingestion from multiple sources in formats like JSON, Parquet, Avro, and CSV.
• Implement best practices for data governance, quality, and performance optimization.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Lead and mentor data engineering teams in coding, debugging, and optimization.
• Utilize IBM DataStage and AWS services (Glue, S3, Lambda, Athena, Redshift) for efficient data integration.
• Support infrastructure automation using Terraform or CloudFormation.
Required Skills:
• 10+ years of experience in Data Engineering, with at least 3+ years in a Lead role.
• Strong proficiency in Python and PySpark for data processing.
• Expert-level experience with AWS Glue, S3, Athena, Lambda, and Redshift.
• Strong knowledge of RDBMS (PostgreSQL, SQL Server).
• Deep understanding of data formats (JSON, Parquet, ORC, Avro, CSV).
• Hands-on experience with DataStage for ETL/ELT workflows.
• Strong analytical, problem-solving, and team leadership skills.- R