The interviewer joined ten minutes late. He was a director of engineering who showed off his previous companies where he had worked. He only seemed knowledgeable. We discussed the projects that I had worked on in depth. He didn't know a few basic things, such as that a Parquet file, when read from a source, can't be overwritten back to the same source, or that when you join a Spark DF with a Cassandra table, the full table doesn't load into memory. Another question on Spark was that I had a 100TB file with numbers and I had to sort it. I gave two solutions, but he wanted to hear a solution on Spark 3.5, which I guess only he knew.

Tell me about yourself and projects you've worked on.