Big Data Engineer - (Hadoop/Hive/python/Spark/Scala)
Description For Big Data Engineer - (Hadoop/Hive/python/Spark/Scala)
Capco is seeking a Senior Data Engineer to join their team in Pune, India. This role focuses on building and maintaining scalable data pipelines and API integrations to handle increasing data complexity. The ideal candidate will have 5-8 years of experience working with big data technologies like Hadoop, Spark, and Kafka, along with strong programming skills in Python, Java, and SQL. The position involves designing data management systems, developing analytics applications, and providing technical leadership. You'll work closely with analytics and business teams to improve data models and foster data-driven decision-making across the organization. The role requires both technical expertise and strong communication skills, making it perfect for engineers who want to make an impact in a data-driven environment. This position offers the opportunity to work with cutting-edge big data technologies while taking on technical leadership responsibilities.
Responsibilities For Big Data Engineer - (Hadoop/Hive/python/Spark/Scala)
- Design, construct, install, test and maintain highly scalable data management systems & Data Pipeline
- Ensure systems meet business requirements and industry practices
- Build high-performance algorithms, prototypes, predictive models, and proof of concepts
- Research opportunities for data acquisition and new uses for existing data
- Develop data set processes for data modeling, mining and production
- Integrate new data management technologies and software engineering tools into existing structures
- Create custom software components and analytics applications
- Install and update disaster recovery procedures
- Collaborate with data architects, modelers, and IT team members on project goals
- Provide senior level technical consulting to peer data engineers
Requirements For Big Data Engineer - (Hadoop/Hive/python/Spark/Scala)
- Bachelor's degree in computer science, Engineering, or related field, or equivalent work experience
- Proven 5-8 years of experience as a Senior Data Engineer or similar role
- Experience with big data tools: Hadoop, Spark, Kafka, Ansible, chef, Terraform, Airflow, and Protobuf RPC
- Expert level SQL skills for data manipulation (DML) and validation (DB2)
- Experience with data pipeline and workflow management tools
- Experience with object-oriented/object function scripting languages: Python, Java, Go lang
- Strong problem solving and analytical skills
- Excellent verbal communication skills
- Good interpersonal skills
- Ability to provide technical leadership for the team
