Root Cause Analysis:
During a planned service routing migration, a required configuration update for one service was missed. As a result, traffic for that service was routed incorrectly. Because other services remained unaffected and the overall configuration appeared valid, existing monitoring did not detect the issue. The root cause was human error during the configuration update process.
Preventive Measures:
To reduce the risk of similar issues, we will consistently apply the four-eyes principle to all configuration changes and introduce a mandatory post-implementation verification step to ensure all intended updates have been applied correctly before a change is considered complete.