Databricks Engineer

ShyftLabs is a data product company founded in 2020 that delivers digital solutions for Fortune 500 companies, focusing on creating value through innovation.
Data
Senior Software Engineer
Hybrid
5+ years of experience
Enterprise SaaS

Description For Databricks Engineer

ShyftLabs, a dynamic data product company established in 2020, is seeking an experienced Databricks Engineer to join their team. This role presents an exciting opportunity to work with Fortune 500 companies, designing and implementing cutting-edge big data solutions using the Databricks Unified Analytics Platform.

The position requires a seasoned professional with 5+ years of experience in Databricks and Apache Spark, combining technical expertise in SQL, Python, and cloud platforms with a strong understanding of data engineering principles. As a Databricks Engineer, you'll be responsible for architecting and optimizing data pipelines, implementing robust ETL workflows, and ensuring high-performance data architectures.

The role offers a unique blend of technical challenges, working with state-of-the-art technologies including Apache Spark, Delta Lake, and modern cloud platforms. You'll be instrumental in driving data-driven insights and implementing data governance practices while collaborating with cross-functional teams of data scientists, analysts, and engineers.

ShyftLabs provides a supportive environment for professional growth, offering competitive compensation, comprehensive insurance benefits, and extensive learning opportunities. The hybrid work model at either Noida or Hyderabad office provides flexibility while maintaining collaborative team dynamics.

This position is ideal for someone passionate about big data technologies, with strong problem-solving abilities and experience in data security and compliance. The role offers the opportunity to work on innovative solutions that help accelerate business growth across various industries, making it an excellent choice for professionals looking to make a significant impact in the data engineering field.

Last updated 13 hours ago

Responsibilities For Databricks Engineer

  • Design, implement, and optimize big data pipelines in Databricks
  • Develop scalable ETL workflows to process large datasets
  • Leverage Apache Spark for distributed data processing and real-time analytics
  • Implement data governance, security policies, and compliance standards
  • Optimize data lakehouse architectures for performance and cost-efficiency
  • Collaborate with data scientists, analysts, and engineers to enable advanced AI/ML workflows
  • Monitor and troubleshoot Databricks clusters, jobs, and performance bottlenecks
  • Automate workflows using CI/CD pipelines and infrastructure-as-code practices
  • Ensure data integrity, quality, and reliability in all pipelines

Requirements For Databricks Engineer

Python
Kubernetes
  • Bachelor's or Master's degree in Computer Science, Data Engineering, or related field
  • 5+ years of hands-on experience with Databricks and Apache Spark
  • Proficiency in SQL, Python, or Scala for data processing and analysis
  • Experience with cloud platforms (AWS, Azure, or GCP) for data engineering
  • Strong knowledge of ETL frameworks, data lakes, and Delta Lake architecture
  • Experience with CI/CD tools and DevOps best practices
  • Familiarity with data security, compliance, and governance best practices
  • Strong problem-solving and analytical skills with an ability to work in a fast-paced environment

Benefits For Databricks Engineer

Medical Insurance
  • Competitive salary
  • Strong insurance package
  • Learning and development resources

Interested in this job?

Jobs Related To ShyftLabs Databricks Engineer

Senior Data Developer

Senior Data Developer position at Global Lending Services, building scalable data pipelines and analytics solutions in a hybrid work environment in Greenville, SC.

Senior Data Engineer

Senior Data Engineer position at Sanity.io, building and maintaining scalable data infrastructure and ETL pipelines for a modern content operating system.

Data Architect, Professional Services

Senior Data Architect position at AWS Professional Services, focusing on cloud architecture, data solutions, and customer success with 3+ years experience required.

Senior BI Developer

Senior BI Developer position at cBEYONData focusing on SAP BusinessObjects and federal financial reporting, requiring 5-8 years experience and on-site work in Arlington, VA.

Senior Data Engineer

Senior Data Engineer position at Topography Health, focusing on healthcare data infrastructure, EHR integrations, and HIPAA-compliant data pipelines.