As I’ve mentioned before, I’m a massive fan of Battlefield 3 on my PS3. One of the reasons for my recent upgrade to BT Infinity was improved bandwidth for playing this game.
So, days after installation, on Friday 5th October, I was hit by a problem. I was constantly finding myself disconnected from the EA servers part way into a game (particularly annoying because when this happens it counts it as a “lose” in your game statistics). I assumed it was an issue that evening and went to bed.
On Saturday afternoon I had a game without any problems. But then, Sunday evening and Monday evening I had the same disconnection problems – now worse, often I was unable to even start a game. And I wasn’t the only one.
Messages flooded into the Twitter account for EA who would only state that they were “investigating”. The assumption amongst people, most of whom appeared to have PS3s, was that this was caused by EA’s recent launch of FIFA 13 and Medal of Honor : Warfighter Beta.
However, on Tuesday the penny dropped after performing a search and finding a discussion on the BT forum – this was only affecting BT Infinity users. This in turn lead me to another forum thread where other gamers, not just BF3, were reporting problems, even with non-gaming websites such as Twitter. Those with far more knowledge on networking had performed some investigation and had found the problem.
In a nutshell, from between 7pm and midnight, connections to certain US sites (including EA game servers, Twitter, etc) were having massive packet loses – that is data was being lost during transmission, causing the whole transmission to slow as data had to be re-transmitted repeatedly. It was further narrowed down to the sites that were maintained for BT by a third party.
Here is an example network trace (thanks to MilanoChris)
People had been emailing, calling and Tweeting to BT but had no response. Those who spoke with them on the phone was presented with an off-shore Help Desk who had no idea what they were talking about and appeared to have no interest either.
Indeed, the BT Service Status page didn’t show any issues and this had now been going on for 5 days. Were BT not aware of problems with their own system?
I Tweeted @BTCare twice and had no response. I emailed them but received a generic response asking them to check the settings on my Home Hub. In the end, those on the forum decided to simply bombard BT with fault requests. After a number of hours of doing this, and the issue being raised by The Register, BT decided to respond. The problem appeared on their Service Status and they started replying to Tweets to say they were investigating.
Later in the day, one of their network engineers posted on the forum to state that whilst their third party were looking into the cause they had re-routed a lot of the traffic via another source…
We are aware of this issue and we have made some changes today, however the upstream has a significant congestion issue in their network (today we don’t even reach 60% of our available capacity to the LINX, which is the place where ISP’s in the UK exchange traffic).
We have changed our network to directly exchange traffic with rackspace (who host some EA sites) and we have re-routed a number of internet sites via other paths. We have raised this congestion issue with third party.
I explicitly asked the engineer to confirm that, because a temporary workaround has been put into place, the issue would not be closed until it was fully resolved – he confirmed this would be the case. That night I got into BF3 fine and had a great game.
Next day, more news – this time via ISPreview – but the Service Status has been removed. My assumption was that the issue had therefore been fixed and asked on the forum why we hadn’t been told this. An administrator said it hadn’t. Odd then that the open service status had therefore been removed (it wasn’t even listed under the resolved problems).
Over a week later and I’m not sure if the problem is resolved, but I’m guessing if it’s not then it’s on its way to being so. But a lot of people are left questioning the support provided by BT. We are all paying for a premium broadband service but we received anything but premium support. If BT were aware of this issue since the Friday (and you’d like to think they were) why did they not say anything? In fact they went out of their way to ignore queries via Twitter or to respond on the forum.
BT may have the best infrastructure and the best Broadband connection, but until they can match it with suitable level of support, they will never be best.