Senior Software Engineer - Distributed Data Systems

Databricks is seeking a Senior Software Engineer to join their Runtime team, focusing on distributed data systems. This role is at the forefront of building next-generation distributed data storage and processing systems that power modern data analysis and machine learning workloads.

The position involves working on critical projects including Apache Spark™, the industry-standard framework for big data processing, Delta Lake storage management system, and Data Plane Storage services. You'll be responsible for developing systems that can outperform specialized SQL query engines while providing sophisticated programming abstractions for diverse workloads from ETL to data science.

As a senior engineer, you'll contribute to significant projects like Delta Pipelines, which aims to simplify the orchestration and operation of thousands of data pipelines. You'll also work on performance engineering, building next-generation query optimizers and execution engines that are fast, scalable, and robust.

The role requires strong technical expertise in distributed systems, with at least 5 years of production-level experience in Java, Scala, or C++. You should have a solid foundation in algorithms and data structures, along with experience in big data systems like Apache Spark™ and Hadoop.

Databricks offers a competitive compensation package ranging from $166,000 to $225,000 USD, along with equity, performance bonus eligibility, and comprehensive benefits. The company serves over 10,000 organizations worldwide, including 50% of Fortune 500 companies, making this an opportunity to work on technology that powers some of the world's largest data and AI initiatives.

The position is based in San Francisco, California, where you'll join a team of passionate engineers working to solve complex technical challenges in data infrastructure. This is an excellent opportunity for someone who is motivated by delivering customer value and impact, and comfortable working towards multi-year visions with incremental deliverables.