First of all, thanks for the detailed question. Its a scenario I can completely relate to and am facing something of this sort right now. I would attempt to answer your questions, but changing the order in the responses ( from the original order you provided)
(3) How can I think about de-risking this attempt of the project and improving the probability of success?
If the upgrade has failed a few times in the past, I would start by asking the past owners on the blockers they faced and documenting the same. Have the conditions changed from the past that we are attempting this project again? In either case however, trying to understand the nature of the blockers and what are the possible paths forward will help. Mentioning a couple of examples I can think of,
The answers to the above questions ( and the ones you come up with respect to your project) can help estimating the risk and take a more informed guess at the success rate. The idea is "why did the previous attempts fail and what has changed?"
(2) How can I come up with broad estimates and capacity requirements for the external and internal milestones if I'm not clear on what these milestones would require for completion?
The only way to remove ambiguity from a complex task is to break the complex task/milestone into smaller tasks. Again, going back to the previous owners and looking up the documentation would help. For milestones that are so ambiguous that we are unable to break them into simpler tasks, a strategy I would use is to try ( or seek someon's help ) and correlate the previous owners' PRs to the milestones, and figure out how the previous owner sought to break the problem. A rule of thumb that my team has followed in the past is if the estimate of a task is more than 2 days, it probably needs to be broken down further ( of course, exceptions can be there). This exercise sucks but it generally sucks more without these finer estimates.
(1) How can I identify smaller, independent high-level milestones that are relevant for external stakeholders?
For this part, I would approach them ( perhaps cut a low/med sev ticket if too many stakeholders) and inquire from them about two things:
The answer to these questions would help in coming up with timelines and might pave way for more questions ( which lead to finer milestones )
Another question you might want to ask your leadership: why do you need to do this project, now? If multiple teams failed at in the past and the probability of success is low, why does your leadership want the team to do it? How long do you really have to do it? They may have very good reasons and it could help you evaluate your strategies.