Finding more scope internally vs. swapping company
I've been a Data Engineer for most of my career and my observation is that scope as a Data Engineer can plateau and therefore I see a lot more L4/5 DE's than L6+. I think it is because you don't impact the bottom line directly and regularly.
At FAANG's I've worked at so far, finding new scope can be difficult even when you are working with stakeholders: it is "easier" to scope/build a product (i.e. SWE work) and show metrics of success to add value vs building a data pipeline which may be limited to them having a reporting need for example which often isn't the case especially in a more established firm.
I moved into a partner facing DE role to help more with scope/stakeholder exposure. The highest impact project I worked on so far is influencing an internal team to change the way we measure a particular metric. This involved mostly stakeholder management and nothing more complex than SQL queries from a technical standpoint. While it was fulfilling, this is also something I 'stumbled' upon and is rare due to challenges like partner scope/vision is limited/slow (their leadership can change and therefore you projects/ideas can), technical challenges of automating things because of larger concerns (e.g. privacy, lack of infra on their side which you have no control over) and so on (you generally have even less control than an internal DE).
In my current role, I am generally able to derive projects, but (in my opinion) they are limited in scope/value: i.e. build a pipeline, deliver an analysis. Therefore, even though the projects 'ticks the boxes' for an L5, it is not really driving a 'transformation' as an L6+ would. I also directly asked my manager what are some of the hardest problems we have, and have been told we have a lot, yet, I'm not hearing or seeing them.
Given the situation, would you:
- Move to a SWE role internally at FAANG for a more established path 'up' (not sure this resolves the scope problem especially at FAANG as I think SWE-DE's can almost be even harder to get to L6+ on because they generally lack stakeholder visibility and focus on more top down work?).
- Seek roles outside of FAANG where the scope of the work is already scoped to L6+ e.g. Airbnb so the 'heavy lifting' has been done in terms of scope.
- Refine your scoping strategy within you own team, and if so, how?
Note: my motivation is to thrive at work, this isn't for a promo, just incase the post comes across as promo-focused. :)
Discussion
(2 comments)- 1
a month agoTech Lead @ Robinhood, Meta, Course HeroGiven the market, I'm leaning towards Option #3 of "Refine your scoping strategy within your own team":
- Interviewing sucks right now and you're at Google, one of the greatest companies on Earth. As I'm writing this, Google is at ATH with $155/share. You're already in an incredible situation relative to almost everyone else.
- Google is infamous for hard L4 -> L5 promotions so switching to SWE doesn't seem like a great solution, at least in the short and even medium-term. However, if you're really passionate about SWE and have fallen out of love with DE, you should 100% switch.
In terms of refining scoping strategy, I believe your manager is telling the truth when they mention that there's a lot of problems. However, there are 2 reasons why they aren't being more specific with you about these problems:
- Not enough time - Distilling all the noise, chaos, and complaints you get as a manager into something actionable takes a lot of effort and managers are extremely busy people.
- They are forcing you to grow - If an L5 project is handed to you, it ceases to be an L5 project. Being able to identify problems and create scope on your own is crucial for L5 and even more important at L6.
If you can't find a meaty problem to take on, I recommend going full on user researcher mode (UXR). Schedule 45 minute to 1 hour 1 on 1s with anyone on your team and across sister teams who is willing to talk to you and is connected to the data layer in some way. Tell them that you are here to help and want to hear all the complaints they have. Compile the notes into a master doc and see what patterns you can extract. There was a very high-performing L5 infra engineer I worked with at Instagram who did just that to great success as I describe in-depth here: "How can I come up with big initiatives, especially those at a Staff level?"
Here's a potential freebie on my end too: Are there any logging events you can delete? At a giant FAANG company like Meta or Google, I'm sure there's a lot of deprecated logs that aren't being used by anyone. By killing them, you will save the company tons of money. One of the big workstreams I created in partnership with an L5 Data Engineer at Instagram was identifying all the deprecated logs related to Instagram Stories. The savings from killing all those events was $25 million+ per year.
I'll be releasing my promotion course in the next week as well which will talk about these concepts in far more detail.
- 125 days agoMid-Level Data Engineer [L4] [OP]Google
Great reply, thanks!
- My preference is staying here (even long term) but just didn't want to get "stuck" as an L4 long-term with limited scope. The options also refer to switching internally e.g. there are a lot more SWE roles at Google (and really interesting roles internally) and its easier to switch as an L4.
- It depends. SWE TC is best in the company relative to other job ladders internally and a lot of DE work is done as a SWE (job ladder) depending on the team (but more on the technical side). I've not fallen out of love with DE work, it is more just a longer term career choice and, more importantly, scope.
The 2 reasons you wrote are spot on. My manager is both very busy (he is L7+) and is trying to help me grow which I appreciate. Therefore I am trying to take this positively and figure out ways of growing in the team.
I followed your UXR approach as I do find this effective and a great way to learn. The general issue I'm facing is each sub-team is fairly disconnected in terms of scope (for valid reasons) and the data layer lives 'outside' my team to have much influence over. Within my own team, we also deal with our own partners who have their own setup/challenges so getting something sync'd between partners is tricky. That said, if I could get some ideas, then I am sure I could figure something out around that (which is why I was prodding my manager). The article linked is great and is very helpful in helping me re-assess things.
The logging events to delete is a good example of a project I explored (in another team) at Google. Though we had a path forward, given the way the tables are stored didn't really have much impact financially (because the cost is negligible). However, improving the way the search of the tables was surfaced was possible, which was a win (and I worked with another team to implement this). Again, a use case I sort of 'stumbled upon' rather than actively discovered as a problem.
I look forward to going through the promotion course. :)
