Principal Software Engineer

Microsoft's Web Data Platform Crawler team is seeking a Principal Software Engineer to help scale their global web-scale Index platform. This role is fundamental to all Search and AI scenarios, with the system currently crawling tens of billions of URLs daily. The position offers an exciting opportunity to work on containerizing services and building the next generation of crawler systems.

The role involves working with distributed systems at massive scale, implementing machine learning models for effective resource usage, and balancing the demands of web crawling with server load management. With the growing importance of Large Language Models (LLMs), this position is critical in gathering web-scale data for both training and serving use cases.

As a Principal Engineer, you'll be working on hyper-scale distributed systems, focusing on increasing throughput and global deployment while maintaining intelligent crawling practices. The role requires expertise in big data technologies or scalable backend systems, with opportunities to influence the architecture of systems that power Microsoft's search and AI capabilities.

The position offers competitive compensation, comprehensive benefits, and the chance to work with cutting-edge technology at one of the world's leading tech companies. You'll be part of a team that values growth mindset, innovation, and collaboration, working in a hybrid environment that allows up to 50% work from home.

This is an excellent opportunity for experienced engineers who want to tackle complex technical challenges at scale, contribute to critical infrastructure, and shape the future of web crawling and data collection for AI applications. The role combines technical leadership with hands-on development, requiring both deep technical expertise and the ability to guide and influence technical decisions across teams.