Citywide, re: their outage a couple days earlier, throwing Eastlink under the bus:
The outage was caused by our last mile provider when their provisioning/billing software system started removing provisioning profiles of our customers (and the customers of a number of other ISPs) – executing a procedure that essentially disconnects a customer.
Over the next few hours, more and more customers lost their connections. Our technical teams immediately communicated with our last mile provider – initially they were reluctant and even refused to fix the problem – however, when we escalated it to their management team and still didn’t receive a response – so I called our last mile provider’s President on his cell phone – he indicated that this hadn’t been an intentional act and shortly after that - they started to correct the problem.
Although restoration started at around 3:00 PM, for some customers it was as late as 7:00 AM Tuesday morning before the underlying issue was fixed and 100% of the connections were once more operational – re-provisioning the connections was significantly more difficult/time-consuming than disconnecting them.
We feel that this problem was a very preventable issue that should never have happened in the first place.
We asked our last mile provider to post a public apology (that would have been the “right thing to do” in my opinion) – they declined.