For your context regarding to this question: This past month, I had a painful release experience, required debugging production issue introduced by another cross-functional team right before the New Year Holidays Release. I found out that that this team does not have good QA process and would ship out this new feature assuming that QA complete varying degrees of success. Their release process seems to be broken and it ultimately lead to production bugs and issues that requires hot fixes. I joined the retro release meeting with Dev & QA leads. And we all agree that QA should have not signed off the feature release until it's done with regression testing. and future new feature must require proper review and End-to-End testing on Non-Production Env. However, the issue that this new team raise is that they do not have proper resource and maturity to perform all of the necessary QA testing, and their QA engineers could not test out their feature in the non-prod env, so therefore they cannot catch the bugs before going to production env. TLTR, here's a Question**: Appreciate any thoughts or suggestion on how to work with this new team (a cross-functional inside my Organization)? Given the context I learned that this new team lack of QA resource and engineering maturity and scoping issues. I feel uncomfortable to approve and review this team's changes, as it seems very likely that similar mistakes and defect will get introduced by this team until they fix the QA process. I don't think I want to see any more new bugs added to Production and not discovered during the QA phase.

There's another question (perhaps you discussed in the retro): why was this feature shipped before the New Year holiday? Given that many people are out of office, it seems like poor timing for the team to plan it that way. Or perhaps it wasn't planned. To your question: I think you should push hard for that team to be more self-sufficient. If they truly can't test until something goes into production (that seems kinda lame though...), I'd ask for the following from them: what steps have they taken that other teams are not negatively impacted by their releases? what monitoring do they have, and how quickly can they turn off features? do they have a thorough runbook to diagnose issues if they come up? There should also be a discussion about the cost (either revenue or user experience) for this -- if it's an internal tool, for example, maybe you can be more lenient.

How to work with a new, cross-functional team that would introduce new defect to production and not discovered during the QA phase?

Anonymous User at Taro Community2 years ago

For your context regarding to this question: This past month, I had a painful release experience, required debugging production issue introduced by another cross-functional team right before the New Year Holidays Release. I found out that that this team does not have good QA process and would ship out this new feature assuming that QA complete varying degrees of success.
Their release process seems to be broken and it ultimately lead to production bugs and issues that requires hot fixes.
I joined the retro release meeting with Dev & QA leads. And we all agree that QA should have not signed off the feature release until it's done with regression testing. and future new feature must require proper review and End-to-End testing on Non-Production Env.
However, the issue that this new team raise is that they do not have proper resource and maturity to perform all of the necessary QA testing, and their QA engineers could not test out their feature in the non-prod env, so therefore they cannot catch the bugs before going to production env.
TLTR, here's a Question: Appreciate any thoughts or suggestion on how to work with this new team (a cross-functional inside my Organization)? Given the context I learned that this new team lack of QA resource and engineering maturity and scoping issues. I feel uncomfortable to approve and review this team's changes, as it seems very likely that similar mistakes and defect will get introduced by this team until they fix the QA process. I don't think I want to see any more new bugs added to Production and not discovered during the QA phase.

4848 Views

33 Comments

Discussion

(3 comments)

0
Anonymous User [OP]
•Taro Community
2 years ago
I also checked the Flow for current QA process for Production HotFix is following:

Check for impact of the bug/issue on the system. And if is of high severity and high priority, call for a hotfix.

Had a retrospective meeting and find the root cause for the defect and also make sure that we have test scenario's and test cases ready for regression.

Perform a regression on Stage Environment for the hotfix and make sure that build is stable and no new defects introduced because of the defect.

QA gives approval for moving build to PROD Environment and performs regression testing and sign off for the release.
1
Rahul Pandey
•Meta, Pinterest, Kosei
2 years ago
There's another question (perhaps you discussed in the retro): why was this feature shipped before the New Year holiday? Given that many people are out of office, it seems like poor timing for the team to plan it that way. Or perhaps it wasn't planned.

To your question: I think you should push hard for that team to be more self-sufficient. If they truly can't test until something goes into production (that seems kinda lame though...), I'd ask for the following from them:

what steps have they taken that other teams are not negatively impacted by their releases? what monitoring do they have, and how quickly can they turn off features?

do they have a thorough runbook to diagnose issues if they come up?

There should also be a discussion about the cost (either revenue or user experience) for this -- if it's an internal tool, for example, maybe you can be more lenient.
1
Rahul Pandey
•Meta, Pinterest, Kosei
2 years ago
Ask for this legwork to be done by that team, and I'd also involve various managers as appropriate (depends on how contentious the relationship is).

Certainly you should keep your manager in the loop, and broadcast any decisions made to everyone who attended the retro.