The interview process began with a call to Human Resources, followed by a technical call.

Next was a one-hour coding screener.

The final stage involved five one-hour technical interviews conducted on the same day.

During these final interviews, one interviewer consistently appeared bored. He was yawning, stretching his arms, and reclining his chair.

He then asked about using a pirate infrastructure of 10,000 bots for web scraping without permission. I was incredulous that he would suggest engaging in illegal activities.

In another programming interview, I explained to the Engineer that I generally prefer not to use regular expressions for validations like IP addresses, as hidden bugs are easily introduced. I also mentioned my preference to avoid using yield when working with threads, as errors can be difficult to debug. It seemed as though he perceived this as a lack of knowledge regarding regular expressions or yield.

When I asked if I should write unit tests for the solution, I was told it was not necessary. However, I later received feedback that I did not code unit tests during the interview. This was very disappointing.

Imagine your CEO has rented a farm of 10,000 bots (illegally controlled machines) and wants to scrap a site.

How can you avoid detection by the legitimate owners, both in the bot machines and on the platform being scraped?

Ensure that each URL is visited only once. Split the load across all 10,000 nodes.