What’s Your Process for Estimating Tasks Accurately?

There's 3 separate questions here, but it really all comes down to experience.

Pulling The Data

The first level is your personal experience. If you have worked on task A before and then you get task B which is ~75%+ similar, you know that task B will (probably) take roughly the same amount of the time.

The next level is the experience of others, which is what you'll need to rely on when you're new, either new to a team/company/domain or new to software engineering overall (i.e. you are a junior/early mid-level engineer). This is why it's so important to rope people in early and proactively when you're scoping things out. For larger projects, this should be some sort of system design doc.

So what if you're a new engineer on a team, and you're the only engineer in your space? (very common in early startups) Then you just have to do things to hard way - Try out a bunch of stuff, mess up estimates greatly, and build out your experience pool through a lot of trial-and-error pain 😅

Evolving

You can get estimates wrong even with a lot of data - This is very common in Big Tech environments where the sheer scale leads to a ton of natural curveballs. If an estimate ever goes wildly wrong, you should do some reflection, ideally with some sort of retro meeting for larger projects.

Even for smaller interactions like the one you had, you should ask a ton of followup questions and deeply understand why their estimate was so much higher. After you get the reasons (as a tech lead, the most common case for overly aggressive estimates is missed edge cases), ask yourself if you would have been able to come up with those angles yourself or if they were truly something you didn't know at all. Both are common, but I have seen that for many cases, the more junior engineer could have seen what I saw if they had thought harder about all the possible flows upfront.

Learning The Right Way

To make the construction of your "experience bank" effective, I highly recommend following the Lego Model and compartmentalizing all your experience into certain components and patterns: How The Best Software Engineers Learn A Tech Stack - The Lego Model

If you don't do this, it'll be hard for you to translate your experience into new projects unless they're exactly the same. If you see the world through distinct legos, you can mix and match pieces to come up with estimates even if a project has something wildly different. For example, if a new project has components A, B, C, D, and E where you have seen A-D but not E, you can do something like this:

1. Make estimates for A-D and sum them up
2. For E, just do a rough ballpark based on vibe and add on that 50% buffer I've mentioned many times (e.g. 2 weeks -> 3 weeks)

Yes, E is a wild card, but your estimate is probably ~80% accurate as you identified the 4 other components within your lego kit.

An example of a lego for me as a mobile engineer is this:

1. Have a screen that shows a list of items
2. You pull the items from an API
3. Some collection of features among CRUD are supported (pagination, edit, delete, etc)

I have built this lego 75 trillion times with all the different parameters (i.e. sub-legos), so I know exactly how long it'll take whenever I get any feature/project like this.

Envision The Code

This is less related to experience, but you should always try to envision the code as much as you can, ideally line-by-line. I give this advice a lot (e.g. envision how you would write the code yourself when reviewing a teammate's PR and look for the delta), because it's really helpful. Clarity leads to action and higher quality of execution - In this case, it will help you do the estimate as your brain will start hypothetically feeling the difficulty and volume of the code. And if there's some areas where you're like "Crap, I have 0 idea how this would look", then you know to add more buffer to the estimate.