1

How to make up for mistakes?

Profile picture
Software Engineering Intern at Taro Community3 months ago

A few days ago, I almost broke production.

Our company doesn’t have any formal CI/CD protocols. When we ‘push to production’ that would mean testing code on our local computers, then going onto our single production machine, clone our changes, test on the production machine, then merge to main. The dev team was incredibly small, but this process started failing when more people worked on the codebase.

My manger strongly prefers merging one large feature all at once as opposed to merging changes bit by bit. Even though I pushed for the ‘one diff, one thesis’ approach, he pushed back on it. He would rather clone my entire feature all at once over making small changes.

A week and a half ago, I finished my main feature. My code successfully worked on the production machine, but my manager kept pushing back on actually landing the code to production. I kept pushing for my manager to push my code but he kept extending the due date. Now that I’m close to my internship, he is asking me to push code.

The issue is that and another dev were trying to push their features to production. The other developer went first since his feature was smaller, then it was my turn. When it came to merging my code, I made errors in resolving merge conflicts, which resulted in errors. I tried ‘undoing’ the commits by reverting changes (when I should’ve done a reset), which caused the commit history to become longer and more complex, making the situation worse. After rolling back commits, somehow even the old code wasn’t working.

My team and I are working to resolve the issue, but I was really upset throughout the whole process.  I’m frustrated at my own incompetence with version control, but I’m also frustrated with my manager for not listening to my suggestions, as I think this would’ve been prevented if I had integrated my code earlier and in smaller chunks.

My team knows that it was a version control issue (as opposed to a code issue), especially since I showed them on two different occasions that my code works.

How should I handle this situation? I definitely feel like I should apologize but how can I make up?

43
2

Discussion

(2 comments)
  • 2
    Profile picture
    Eng @ Taro
    3 months ago

    Sorry to hear that your manager pushed back on you merging your code in smaller chunks. One issue with merging in your larger diff last, is that the burden lies on you to make sure all of your new code works with the code that was merged in since you started working on your code. When you can merge in your code in a piecemeal way, the burden is shared among your teammates. You are in the right that your manager should have let you merge in code a little at a time.

    I'll say that everyone in software engineering goes through a rite of passage where they have to wrestle with version control, so you aren't alone :). The two most common issues I see are: accidentally resetting a work in progress branch and dealing with a ton of merge conflicts. It can seem like the world is collapsing in on you when these things happen, but both are recoverable, and you'll get through them.

    When it came to merging my code, I made errors in resolving merge conflicts, which resulted in errors. I tried ‘undoing’ the commits by reverting changes (when I should’ve done a reset), which caused the commit history to become longer and more complex, making the situation worse. After rolling back commits, somehow even the old code wasn’t working.

    One way to avoid too many issues with version control is to keep the history as simple as possible. When I want to merge in my branch, I will squash all of my commits into a single commit. Then, I will fetch all of the latest changes from the remote repo. Then, I'll rebase my one commit on top of all of the latest changes.

    You mentioned that you had to roll back multiple commits. I think squashing all of your commits together into one commit would have helped a lot in this case. If there was an issue, you know that your commit would have led to it, so you would have only had to revert your commit.

    Our company doesn’t have any formal CI/CD protocols. When we ‘push to production’ that would mean testing code on our local computers, then going onto our single production machine, clone our changes, test on the production machine, then merge to main. The dev team was incredibly small, but this process started failing when more people worked on the codebase.

    Since you're an intern, the CI/CD process is mostly out of your hands because the scope is too large for the short internship time frame. But, it does sound like an interesting opportunity if you were to return with a full-time offer.

    How should I handle this situation? I definitely feel like I should apologize but how can I make up?

    This is a pretty common scenario, so I wouldn't worry too much about it. I would just make sure that you are diligent with making sure that everyone else's changes still work after you are able to resolve the merge conflicts and get your changes on the main branch.

  • 0
    Profile picture
    Tech Lead/Manager at Meta, Pinterest, Kosei
    3 months ago

    The best way to recover is to share your learnings and have this mistake actually benefit the rest of the team. "Turn lemons into lemonade" 🍋

    Right a detailed post-mortem sharing what happened and what you would have done differently. Share what commands you would have run to avoid the mistake.