Big Data Engineer (Onsite) // Columbus, IN
Shankar
Hi,
Please revert me on Shankar.a...@3sbc.com
Role: Big Data Engineer (Onsite Role)
Columbus,
IN
6+ Months
Supports, develops and maintains a data and analytics platform. Effectively and
efficiently process, store and make data available to analysts and other
consumers. Works with the Business and IT teams to understand the requirements
to best leverage the technologies to enable agile data delivery at scale.
Responsibilities:
· Implements and automates
deployment of our distributed system for ingesting and transforming data from
various types of sources (relational, event-based, unstructured).
· Designs and implements Spark
Structured Streaming & API workflow
· Implements methods to
continuously monitor and troubleshoot data quality and data integrity issues.
· Implements data governance
processes and methods for managing metadata, access, retention to data for
internal and external users.
· Develops reliable, efficient,
scalable, and quality data pipelines with monitoring and alert mechanisms that
combine a variety of sources using ETL/ELT tools or scripting languages.
· Develops physical data models
and implements data storage architectures as per design guidelines.
· Analyzes complex data elements
and systems, data flow, dependencies, and relationships in order to contribute
to conceptual physical and logical data models.
· Participates in testing and
troubleshooting of data pipelines.
· Develops and operates large
scale data storage and processing solutions using different distributed and
cloud based platforms for storing data (e.g. Data Lakes, Hadoop, Hbase,
Cassandra, MongoDB, Accumulo, DynamoDB, others).
· Uses agile development
technologies, such as DevOps, Scrum, Kanban and continuous improvement cycle,
for data driven application, attends daily stand-ups.
Skills:
Knowledge of the latest technologies in data engineering is highly preferred
and includes:
- Exposure to Big Data open source
- Hands on experience in Spark Structured Streaming & API
workflow
- SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka, Microsoft Azure
Databricks
- ADLF and Azure analytics
- SQL query language
Thanks & Regards,
Shankar | Sr. Technical Recruiter
Email: Shankar.a...@3sbc.com
Work: 281-823-9222 Ext 517 | Direct: 409-404-0333
11271 Richmond ave,Suit #107,#108,Houston,TX-77082
3S Business Corporation
An E-Verified Company
****Best way to reach me through email****