This weeks network outage

We sincerely and deeply regret the inconvenience caused to our customers as a result of the disruption of connectivity services this week.

Our network is operational – if you are without connectivity still, please turn off your fibre ONT for no less than 10 minutes, power back on and wait up to 5 minutes for connectivity to re-establish.

Please review the below video from our Founder, Shaun Fisher:

Transcript:

Hi, Shaun here – Founder of Vetta Group.

I wanted to jump on video here to apologize for the recent network outages which you’ve felt on our network over the last week, and especially yesterday evening.

Believe me when I say that myself and my whole team here take an immense amount of personal ownership and emotional investment in our network and the services we provide, and while I am proud of how our staff combat these outages as a combined team effort, we truly feel the frustration, inconvenience and pain experienced by yourselves as our customers when being without your critical connectivity and we are absolutely devastated to have our network entail this on your business and home connections. Needless to say, we do not take the responsibility of being your telecommunications provider lightly.

The triggers and causes of these events range in extreme fashion, from third party actions, through to significant simultaneous systems outages triggered by what in these early hours post event appear to be a range of independent and badly timed events requiring a cohesive team approach to tackle, all of which will be independently reviewed and investigated.

My absolute and committed focus right now will be not only to my team, but to fully understand and comprehend the triggers, causes and effects of the last week both internal and external to ensure that our modernized network which was deployed earlier this year can be the confidence inducing network we all know it will be into the future.

For our impacted customers, we owe it to you to make this right, and while our focus right now needs to be ensuring our network is robust, resilient and performant, it is too early to speculate on the series of root causes which acted as triggers to these events, but when these conclusions do come through, we will be ensuring you are provided with as much information, reporting, and reassurance that you need as much as quite frankly equally we also want to establish as well, to continue to be your provider of choice. These will be distributed both through post mortem incident reports on our status page, as well as further information through either this page you’re watching this on (on our website), or linked to through this post, depending on where you’re watching this video from.

From the bottom of our hearts, thank you for your continued support, and we look forward to continuing to innovate, push connectivity further and challenge the status quo of telecommunications in New Zealand.

mā te wā

Update 19th June 2024:

We thank all customers for their patience while we’ve been working both internally and with external consultants to fully review our May 2^nd outage.

Our post incident report for this incident is now available, and can be obtained by emailing faults@quic.nz from your registered account email address. This incident report ultimately shows the colliding 4 events at once which resulted in this incident, briefly being:

• A customer-facing core network port being incorrectly allocated, and customer device incorrectly configured; and
• A dark fibre vendor connecting two of our key datacentres performing unannounced works at a fibre distribution hub; and
• A cluster freeze of our core compute cluster; and
• An undocumented bug introduced by our vendor which reset licensing across all virtual network elements.

As ever, lessons have been learned on how to protect against this in the future, and a thorough dive by external consultants has been completed to ensure our network is in the right state, and heading in the right direction as we continue to work to improve our network services. Speaking of which, you will see a maintenance window coming up shortly to address some points in this report and complete some upgrades.

Thank you for your trust and support in us as always, and myself and the team look forward to continuing to support your connectivity going forward.