Credit Karma

Good perks. Awesome company events. The business model is still good. Lots of big data use cases in important domains. Good enough infrastructure supports. Can potentially be a great place for a data engineer to learn and to grow.

I only focus on data engineering and advanced analytics, as I do feel I have the technical authority to comment.

There are lots of tech debts. Quite a few managers at some very important positions are clueless about the future. They have been struggling to just prevent things from falling apart and to maintain the status quo.

They are not good enough at telling the quality of data engineering work to reward the right people accordingly. This is the basic requirement for a healthy environment to make things improve over time, but it takes a lot to have it right.

Engineers initiated data projects with good visions are almost nonexistent because there are not that many experienced and good data engineers who have stayed long enough to get to the bottom of the debt-infested systems.

Even if they do, they are blocked from knowing the broader context due to heavy management involvement aiming to address the complicated coordination process. This leads to data systems that are not as simple/elegant as they can be, which reinforces more complicated management processes and the practice of micromanaging. And this is a vicious cycle.

Suboptimal data systems may work okay when the company was at a smaller scale by resorting to more manual processes, but the same level of quality can easily push things beyond return at the current scale. This is yet another less recognized scaling problem specifically for data engineering.

There have been major confusions in important data projects not resolved for a long time, and the managers just started working on something else. Over time, data engineers are increasingly shielded from the real problems but only exposed to the managers' limited and confused perceptions. As a consequence, some important but hard problems are left unsolved forever. And it will keep dragging down the quality of the entire system. This can be a major roadblock for new business partnerships and revenue expansion.

At this point, and after some hard lessons, there seem to be good enough alignments about what was not good enough and what should be the goal. However, the much harder problem of how to get there is far from being clear.

At its current scale, it is not possible to incrementally and gradually improve the overall data system's quality. Major improvements are needed for multiple subsystems, and some important subsystems have been missing entirely.

The existing, heavily centralized management model has not been working efficiently enough to scale things with enough quality, and it will not suddenly work with all the carried-over debts and without some structural modifications. This can be a serious risk for a data-driven company.

C-level should have started feeling the urge, but seems not decisive enough yet to make stronger moves, other than keep giving new data leadership the benefit of doubt.

The effectiveness of the data system is many years behind companies with similar profiles, which is a lot in the tech world, and the gap is widening. Time is against us. This is urgent. The trial-and-error cost for high-level leadership in an all-in data engineering organization is too expensive.

Please try some alternative strategies in parallel, such as allocating some resources under strong technical leaders with a light-weighted management structure. Put data engineers and product teams to work together directly in smaller, autonomous groups and focus on making high-quality, end-to-end vertical solutions first.

Spread the data engineering risk across multiple organizations at a higher level. The centralized structure works only when the center is proved to be solid, not when it itself has been in doubt.