Yesterday afternoon this forum, most other websites at the centre and email were temporarily out of service - the Interweb died!!
An inquest was held last evening and the verdict was
A network service restart was triggered by our Operations team to implement a minor server reconfiguration in order to resolve some benign IPv6 related errors. Ordinarily speaking, this should not be service affecting and due to its minor nature was not deemed to pose sufficient risk to warrant an out of hours maintenance window.
Unfortunately we got the risk assessment wrong; the operation unexpectedly caused ARP errors for a selection of IP addresses / containers which were therefore temporarily unrouteable.
The problem was quickly detected, diagnosed, and corrected after approximately 11 minutes downtime in total.
We take service reliability very seriously, and are sorry that our actions let you down on this occasion. We have already put operational measures in place to minimise the risk of similar incidents in future.
