I applied online. The process took 2 weeks.

The interview was mainly focused on my technical experience and project background. The interviewer started with my introduction and career journey, followed by questions on my day-to-day responsibilities and technology stack. The discussion then covered:

• PySpark
• Spark architecture and optimization
• SQL
• Databricks
• Azure
• AWS concepts
• incremental loading
• SCD
• data engineering practices

There were also follow-up questions based on my resume and projects to understand my hands-on experience. Overall, it was a technical and project-focused interview with several in-depth follow-up questions.

Interview Questions Asked

• Tell me about yourself / give your introduction.
• You were with TCS only for three months. What was the reason you left TCS?
• Why did you join TCS in the first place?
• Tell me more about AWS.
• What exactly did you do with AWS?
• What is your day-to-day role?
• What is your day-to-day technology stack?
• Which cloud technology are you primarily using?
• What can be the other types of data architectures apart from Medallion Architecture?
• What is the use of Bronze, Silver and Gold layers?
• What is a Data Product?
• What is the difference between Schema-on-Read and Schema-on-Write?
• What is a Broadcast Join?
• What are Broadcast Variables?
• What transformations have you performed in PySpark?
• What are the different DataFrame, RDD and Dataset abstractions in Spark?
• Which languages support RDD, DataFrame and Dataset?
• Does Scala support DataFrames? (This was used as a follow-up/challenge to your answer.)
• What is the difference between groupBy, aggregateByKey and reduceByKey?
• What is Upsert?
• What is SCD Type 2?
• How does Upsert work with SCD Type 2?
• Explain Spark execution.
• How are the execution plans derived, and what is the use of those plans?
• What is Adaptive Query Execution (AQE)?
• Is AQE automatically enabled, or can we disable it?
• How do you disable AQE?
• Do you have any learning/exposure to KQL?
• What is KQL used for?
• What AWS services have you worked with?
• Do you know what AWS Lambda is?
• What is Amazon EMR?
• What AWS services are used for data processing?
• Are you currently serving your notice period?
• What is your official notice period?
• If your notice period is three months, when can you actually be released?