Appbox +Stream routing issues and compensation - Bytesized Hosting

Hi guys,

So it seems I picked a bad few days to go on Holiday. As soon as I left I heard the first signs of various routing issues. I want to use this time to explain exactly how things work on our end, what (probably) went wrong and what we are doing about it. There is a lot of information in this post so feel free to skip headers that are not interesting to you although I would recommend you take the time to read all of i.

How are the Bytesized servers connected to the internet?

When I first decided to buy my own hardware to host the Appboxes Feral kindly offered to be our connection to the internet. They already had some good peering and transit agreements and going this route would ensure our members would have fast default routes regardless of where they were. The alternative would be to get our own routers and try and get our own partnerships with the different ISPs which is hard when you have little traffic. Going with Feral gave us a good staring point.

What is (re-)routing

Routing works by packet forwarding, there is a start point and a destination point and we get from here to there by giving packets to the next node in the chain repeating this process until the packets arrived at the destination. Now along the line you will never know where the packet will go next, you only know where you, a node in the chain, needs to give the packets to next. The longer a packet is on road the slower it will get, usually. To get from Amsterdam to Melbourne a packet needs to travel a lot further then from Amsterdam to Paris, usually he farther a packet needs to travel the slower the connection will become.

Sadly that's not the only way things can slow down. The various nodes in the chain will have different capacity and the slowest connection will bottleneck the entire chain. For example let's say our server is on 10Gbit, the next router we use is also on 10Gbit, next we have a IX in Amsterdam that is 5Gbit and a hop in Paris that's doing 2Gbit the maximum our connection from here to Paris will do in that case in 2Gbit. However, what if Amsterdam is getting a lot of traffic at one point and the full 5Gbit is already used, in that case our connection might slow down completely to sub KB levels as it can't get through Amsterdam fast enough.

Now we can't decide how packets travel in the complete chain, what we can decide though is who we give our packet to first, and who we give it to first decides what the rest of the chain looks like. This is the essence of the rerouting feature. By changing the first node we change the entire chain, in most cases.

By handing it over to Cogent instead of Level3 we might bypassing the Amsterdam IX that was already tapped out, fixing our connectivity issues.

Why did routing issues pop-up now?

Feral has certain default routes that define who the first node in the chain is for certain IP ranges. The last few days they worked on large overhaul of these default lists to increase their internal throughput. That's why certain members who had no issues before are now routed through a different ISP that might be less optimal. Rerouting should fix things in those cases.

I rerouted but it is not working

There was an added problem where the rerouting page was up but the changes were not actually pushed to the core router responsible for the routing. Please try all the routes again and make sure you find the fastest one. For help with that see our previous news post. if after everything there is still no improvement then please submit the details asked in the linked news post in a ticket and we will put it through to Feral so they can look into it.

I had a crappy experience the last few days, I want compensation

I feel you, having a service you can't use to the fullest must be frustrating, but please understand that since we don't control the network we are at the whims of our service provider who in some cases is also dependent on other ISP to have their ducks in a row. We do our best on our end to communicate what is happening and what you can do to try and fix things. But in the end the internet is a fluid web of connections that nobody has full control over.

Even though it's out of our control we want to compensate members who feel wronged, that's why in a few minutes a new button will appear online on your Appbox +Stream box dashboard that will allow you to add a week of free service to your account. We will pay for that ourselves. Since there is no technical way for us to figure out who is actually suffering from this weeks routing issues and who is not we decided to enable this for all members on our +Stream plans regardless of whether they reported issues. I hope our trust in our members is not misplaced.

Since we can't be sure the routing issues are now solved you will have a full month to use this new feature until 4th of September.

Now in closing I want to ask everybody again to think before they create outraged tickets, the abuse me and the other support are getting from some of you is just insane. We do our best to give you the best experience out there every day and it's really demotivating to get called all kinds of things. We are humans just like you, please treat us like such.

tl;dr

Feral restructures their network which caused new routes for most members, these routes might not always be perfect, broken reroute feature gave the impression it did not change anything making things worse. You can get 7 free days by pressing a button if you had issues this week. Also; don't be a dick.