Published on May 8, 2025. Modified on May 22, 2025.
Job Description : Big Data Engineer - Senior Consultant
About Talent Worx :
Talent Worx is a recruitment consulting firm that partners with leading companies to find skilled technology professionals.
We are currently seeking a Big Data Engineer - Senior Consultant for one of our clients, a prominent organization leveraging cutting-edge data technologies.
About the Role :
Talent Worx is looking for an experienced and highly skilled Big Data Engineer at the Senior Consultant level.
This role is based in Pune or Chennai and requires deep expertise in big data technologies, particularly within the Hadoop ecosystem, Spark, and Scala, along with a strong understanding of database concepts and Dev Ops practices.
As a Senior Consultant, you will contribute to the design, development, and implementation of robust and scalable big data solutions, working on challenging projects that involve large-scale data processing and analytics.
Location : Pune/Chennai, India
Experience : 6 to 9 Years
Responsibilities :
- Design, develop, and maintain scalable and efficient big data pipelines and solutions using technologies like Hadoop, Hive, Spark, and Scala.
- Implement streaming and batch processing applications using technologies such as Spark, Storm, or Kafka.
- Integrate with various relational and No SQL databases and apply data distribution principles effectively.
- Work hands-on with the Hadoop ecosystem components, including HDFS, Map Reduce, Hive, Pig, Impala, Spark, Kafka, Kudu, and Solr.
- Develop and consume APIs, working with data formats like JSON, XML, and Hypermedia.
- Apply strong development and automation skills to streamline processes.
- Implement and manage aspects of the Dev Ops lifecycle, including source control, continuous integration, and deployments.
- Write and maintain code primarily in Scala, leveraging a strong background in Java.
- Analyze complex data requirements and translate them into technical solutions.
- Collaborate with cross-functional teams, including data scientists, analysts, and other engineers.
- Troubleshoot and optimize big data applications for performance and scalability.
- Stay updated with the latest trends and advancements in big data technologies.
Requirements :
Technology :
- Proficiency in Big Data technologies (Hadoop, Hive, Spark, Scala).
- Experience with PL/SQL and/or No SQL (Must Have) :
- 6-9 years of experience working with Hadoop and other big data technologies.
- Experience with Spark, Storm, Kafka, or equivalent streaming/batch processing and event-based messaging systems.
- Experience with relational and No SQL database integration and data distribution principles.
- Hands-on experience with components of the Hadoop eco-system (HDFS, Map Reduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr).
- Experience with API development and the use of JSON/XML/Hypermedia data formats.
- Strong development and automation skills.
- Experience with all aspects of Dev Ops (source control, continuous integration, deployments, etc.).
- 5+ years of hands-on experience as a Scala developer, with a previous background in Java.
Qualifications :
- Strong academic record, ideally with a bachelor's degree in engineering, mathematical, or scientific background.
- Strong Communication skills.
- Self-Motivated.
- Willingness to learn and adapt to new technologies.
- Excellent planning and organizational skills.
Knowledge/Experience (Preferred) :
- Experience in Core Banking functionality for generating various hand-offs.
- Experience with containerization and related technologies (e.g., Docker, Kubernetes).
- Comprehensive knowledge of the principles of software engineering and data analytics.
- Knowledge of agile (scrum) development methodology is a plus.
- Experience with Cloudera/Hortonworks distributions or AWS EMR, S3 is a plus.
