Choosing an off the shelf solution v. building from scratch

What are some useful tactics to efficiently search for and evaluate whether a tool/library is right for the job, compare alternatives or decide to build a solution from scratch?

What I find tricky is that differences between alternatives only become clear when the software becomes complex (which is often too late to go back) and that edge cases are unexpected and show up late once the tool is too entangled with your code.

What I have tried is:

1. Ask AI (obvious), works well for suggesting alternatives, but pretty terrible at helping you pick the right one (not a big fan of relying on AI for things that can not be tested immediatly as it is really good at sounding knowledgeable even when it is not).
2. Architect the software as to make the choice reversible (if possible at all will still involve much rewriting)
3. I thought about trying to implement edge cases first but this assumes you already have exprience with whatever it is your are building.

I guess more broadly the question is how to approach decision making when testing alternatives is too expensive/time consuming and where prototyping is not very effective.