I'm at a stage in my career where I'm trying to figure out exactly what I want to do and where I want to go. Something I've thought about a lot about is specializing (e.g. going deep into ML), but I've also heard advice that you should be careful of going too narrow and pigeonholing yourself. Is that latter scenario really possible and how do I navigate myself along this spectrum of staying broad <-> specializing?
I think the advice to avoid pigeonholing isn’t bad, there is good intent, but I think what isn’t included is what that would mean. To me this means you can no longer work in another area. I really don’t see that as a risk. To me the choices are broad diversification, narrow diversification, specialization, or very deep specialization. I think that digging into ML, as long as you’re still coding, working on training infrastructure and so on, is narrow diversification or specialization depending on exactly what you’re working on.
If you decided to go way down a hole on one specific image recognition algorithm, and that was your life for 2-3 years, you may poke your head up and not be as ready to take on general system design problems or coding services. You could resharpen them, but they wouldn’t just be at “production quality”.
The broad generalization would be more like… “full stack”, where you’re jumping from storage concerns to front end to model training to internal tools and really never getting deep on anything but having a lot of broad working knowledge. I think for some startups or startup-like teams this flexibility can be helpful, but I think not having a specific area of strength (but you probably already do at a senior level) can also be a detriment when it comes to moving to a new team or company.
What do you do? A little of this, a little of that Ok, but what big project did you land? None, I just filled the gaps all over the place!
Sometimes this is really valued, but enunciating why you’re great and that you COULD land something big is harder to show.
So… if you like ML, try it. You won’t totally lose your other skills if you keep an eye on trends and don’t completely rathole on one specific technology.
To clarify a common misconception, technology isn't the only thing software engineers can specialize in - You can specialize in behaviors as well. I talk about this in-depth here.
Lee gave a great example of pigeonholing, and I agree with him in that it's not something you really need to worry about. As long as you're striving to build good software and be a good teammate, you will develop more fundamental skills that will carry over with you across your career and the different companies you end up at. I talk about these more fundamental skills and how to build them up in this Q&A from a Meta engineer.
In terms of how to know when you're pigeonholing, it's when you deeply tie yourself to a specific technology. For example, I don't think "Android" or even "Android UI" is too specific - It is not likely these will go away anytime soon. However, let's say you only want to work on code that deals with v2.1.2 of the Android Retrofit networking library - This will be very hard to carry over to another job.
On the flip side, I have seen software engineers get punished by specializing in something they're not truly passionate about out of some manufactured pressure that they need to do it to advance their career. This is corny, but I 100% believe it: To advance your career, just purely do something you genuinely enjoy. Find your own path. Don't just pick something because it feels trendy. If you want to be a generalist tech lead, do it. If you want to go deep on mentorship like I did, do it. If you want to build the future of AR/VR, do it. Just make sure you're doing something you really love and care about.