The interview process consists of two phone screens: a technical phone screen and a programming-oriented phone screen.

The technical phone screen covered many questions that basically boil down to: "Do you know what's going on on your systems?"

For example, what can you glean from the Apache logs on a webserver, and how would you know if performance was being impacted (e.g., too many users hitting the server, or not enough resources allocated).

There were also operating system level questions. A few off the top of my head:

• If you have an executable program (a binary) and you made a copy of that program, then changed permissions on the copy, would a diff show that the file had been changed?

• When you run a program from the shell, why doesn't the program log you out when it's done running? If you wanted this behavior, how would you run the program? (Answer: exec)

• Talk me through what happens when you make an SSH connection to a remote machine. Be specific, such as the identification string exchange, algorithm negotiation, and key exchange.

• Name as many TCP flags as you can. (URG, ACK, PSH, RST, SYN, FIN - mnemonic: Unskilled Attackers Pester Real Security Folks.)

• What protocol(s) does DNS use when you run an nslookup? (Answer: normally UDP, but TCP is used for zone transfers and if a record is too long to be returned via UDP)

• Describe the difference between TCP and UDP, including the advantages and disadvantages of both.

• When I try to connect to a remote machine using (for example) SSH, how does SSH know how to get to that remote machine? (Be able to describe routing, default routes, and hostname lookup.)

The programming portion of the interview tests your ability to program in the scripting language of your choice. You can use common languages such as Perl, Python, Ruby, or PHP. You cannot use Bash or other shell interpreters (no sh, ksh, csh, etc.).

This part of the interview tripped me up a little, as most of my programming is oriented towards systems engineering problems. I write scripts to parse logs, distribute files, and perform backups, etc. I don't do a lot of CS-type programming. Unfortunately, the screener threw several of these types of problems at me, and it kinda threw me for a loop. I was able to solve these problems, but I'm sure I didn't instill the screener with a lot of confidence.

• Write a Perl program that prints a 12x12 multiplication table matrix.

• Write a program that reverses the contents of a file, byte for byte.

• Write a program that counts from 1 to 100. For each number, print a certain string if the number is evenly divisible by 6. Print a different string if the number is evenly divisible by 4. Print yet another string if the number is evenly divisible by 24. If none of these cases match, print the number.

• Write a program that descends through a directory tree and prints all files. (Hint: recursion is your friend here.)

• Given an Apache log file, print the timestamp (hour, minute, and second) followed by the number of times any log entry occurs during that time. (Hint: if you're programming in Perl, a hashed array works great here.)

You need to distribute a terabyte of data from a single server to 10,000 nodes and then keep that data up to date.

It takes several hours to copy the data to just one server. How would you do this so that it didn't take 20,000 hours to update all the servers?

Also, how would you make sure that the file wasn't corrupted during the copy?