Staff Software Engineer, Data Ingestion
Description For Staff Software Engineer, Data Ingestion
BrightEdge is seeking a Staff Software Engineer specializing in Data Ingestion to join their R&D Core Engineering team. This critical role focuses on designing and implementing robust data collection strategies and scalable pipelines. The position offers the flexibility of remote work and is based in Hyderabad. As a Staff Engineer, you'll be at the core of the data ecosystem, developing analytical software solutions that ensure timely, accurate, and complete data delivery for insights and products. The role requires extensive Python expertise, strong distributed systems knowledge, and experience with cloud platforms. You'll work with cutting-edge technologies including Kubernetes, Kafka, and various cloud services, while being responsible for maintaining high-performance data pipelines. The ideal candidate should have 6+ years of software development experience and be comfortable with both real-time and batch data processing. This role offers an opportunity to work on complex data engineering challenges while contributing to the company's core infrastructure. The position includes on-call responsibilities and requires a balance of technical expertise with system design capabilities.
Responsibilities For Staff Software Engineer, Data Ingestion
- Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python
- Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.)
- Implement data transformation and cleansing logic during ingestion to ensure data quality
- Monitor and troubleshoot data ingestion pipelines
- Collaborate with database engineers to optimize data models
- Evaluate and propose new technologies or frameworks
- Develop and implement self-healing mechanisms for data pipelines
- Define and uphold SLAs and SLOs for data freshness, completeness, and availability
- Participate in on-call rotation for critical data pipeline issues
Requirements For Staff Software Engineer, Data Ingestion
- 6+ years experience in software development industry from computer science background
- Extensive experience in developing production-grade applications with Python
- Experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.)
- Strong understanding of distributed systems concepts
- Experience with major cloud providers (AWS or GCP)
- Solid understanding of relational databases
- Experience with monitoring tools
- Proficiency with Git
- Experience with Docker and Kubernetes (Plus)
- Experience with real-time data processing using Kafka, Flink, Spark Streaming (Plus)
Jobs Related To BrightEdge Staff Software Engineer, Data Ingestion
