Senior Data Engineer - CoreAI

CoreAI is Microsoft's central organization building the foundational platforms and tools to power AI experiences across products including GitHub Copilot, Azure AI Foundry, and Copilot Studio. The organization is expanding rapidly and is establishing a world-class data engineering platform to deliver analytical insights to the company leadership and product owners.

As a Senior Data Engineer in CoreAI, you'll be at the heart of transforming data into actionable intelligence and fueling AI-driven innovation. This highly visible role places you as both a technical leader and visionary collaborator, where your architectural insight and hands-on expertise will enable product teams to turn complex datasets into breakthrough business value.

The role requires expertise in implementing and optimizing data pipelines and distributed data systems (Azure Data Lake, Spark, Synapse, SQL databases), along with modern data engineering languages and frameworks (PySpark, SQL, dbt). You'll be responsible for designing large-scale data architectures, ensuring data governance and trust, enabling analytics, and collaborating across functions.

Key responsibilities include:

• Designing, building, and optimizing large-scale, robust data pipelines and architectures
• Following best practices for data quality, lineage, security, and compliance
• Partnering with data scientists, analysts, and business leaders
• Working closely with product managers, software engineers, and AI researchers

The ideal candidate should have 5+ years of experience in developing and deploying live production systems, strong communication skills, and understanding of machine learning concepts. Experience with Microsoft-specific tools like Fabric, Geneva, SQL/Cosmos, Kusto, and Power BI is preferred.

This role offers competitive compensation ($119,800 - $234,700 base pay, higher in SF/NYC), comprehensive benefits, and the opportunity to work on cutting-edge AI technology that impacts Microsoft's global products and services.