I have 2 years of experience.
In my current job I'm only working on research tickets without writing any new code or adding new features, I work as a contractor for a US based company (Finech).
While the WLB and my manager are great I rather be in a place where I can get feedback on the code I wrote and learn from it.
If your goal is to grow as a Software Engineer, you'd be better served by finding opportunities for writing code and getting better at it. While working on side projects can be helpful, it's not a proxy for coding experience at the job.
Yes, if you're not writing code, and you want to stay as an individual contributor SWE, I think this will hurt your career.
Instead of jumping to the conclusion that you need to switch jobs, though, I wonder if you can start to take on coding work at your fintech job now. That'll be much easier (and you have trust built up) compared to finding a coding role at a brand-new company where you might struggle if you haven't coded in a while.
The reason to switch would be if you don't think you'll have any kind of growth/change in your contractor role. e.g. if you simply don't have permission to push code to the repo, that's a difficult barrier to overcome.