Software Engineer, Data Acquisition
Description For Software Engineer, Data Acquisition
OpenAI, a leading AI research and deployment company, is seeking a Senior Software Engineer to join their Data Acquisition team within the Pre-training organization. This role is crucial in managing web crawling and GPTBot services, working closely with Data Processing, Architecture, and Scaling teams. The ideal candidate will have 5+ years of industry experience, strong expertise in distributed systems, and proficiency in Kubernetes.
As a Software Engineer in Data Acquisition, you'll lead projects in web crawling, data ingestion, and search. You'll develop highly scalable systems handling petabytes of data, implement data indexing algorithms, and maintain backend services for data storage. Collaboration is key, as you'll work with various teams to ensure smooth data flow and system operability, including the legal team for compliance matters.
OpenAI offers a competitive salary range of $310K – $385K, along with generous equity and comprehensive benefits. These include medical, dental, and vision insurance, mental health support, a 401(k) plan with 50% matching, unlimited time off, paid parental leave, and an annual learning stipend.
Join OpenAI in their mission to ensure general-purpose AI benefits all of humanity. They value diversity and are committed to creating an inclusive environment. If you're passionate about pushing the boundaries of AI capabilities and want to work on cutting-edge technology that can shape the future, this role at OpenAI presents an exciting opportunity to make a significant impact in the field of artificial intelligence.
Responsibilities For Software Engineer, Data Acquisition
- Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search.
- Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability.
- Work closely with the legal team to handle any compliance or data privacy-related matters.
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
- Architect and implement algorithms for data indexing and search capabilities.
- Build and maintain backend services for data storage, including work with key-value databases and synchronization.
- Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks.
- Conduct and analyze experiments on data to provide insights into system performance.
Requirements For Software Engineer, Data Acquisition
- BS/MS/PhD in Computer Science or a related field.
- 5+ years of industry experience in software development.
- Experience with large web crawlers a plus
- Strong expertise in large stateful distributed systems and data processing.
- Proficiency in Kubernetes, and Infrastructure-as-Code concepts.
- Willingness and enthusiasm for trying new approaches and technologies.
- Ability to handle multiple tasks and adapt to changing priorities.
- Strong communication skills, both written and verbal.
Benefits For Software Engineer, Data Acquisition
- Medical, dental, and vision insurance for you and your family
- Mental health and wellness support
- 401(k) plan with 50% matching
- Unlimited time off and 13 company holidays per year
- Paid parental leave (20 weeks) and family-planning support
- Annual learning & development stipend ($1,500 per year)
- Generous equity
