Our journey into TLS/SSL - Bytesized Hosting

tl;dr: Restart your (supported) apps to get some form of SSL access.

If there is one feature that we had the most problems with the last couple of years it's been TLS/SSL (SSL from now on to keep things simple). In theory it's very straightforward but when you need to run it on our scale with the little resources we have available it becomes really tricky. In this post I wanted to run you through our journey into SSL so you understand our thought process which led up to our current implementation. Be warned this post does get a bit technical at times.

What is SSL

Let's start with a brief intro into what SSL is and does and why you should care. SSL is a protocol that allows a client and a server to communicate with each other using encrypted messages. Because of this encryption nobody else but the client and the server know what is being talked about.

When you browse the web using normal unencrypted http all parties that handle your traffic, for instance from your computer to Google's servers, can in theory see your query. When you use http over SSL (https from now on) they can still see you are trying to reach Google's servers but not what kind of information you are asking for. This is especially important when transmitting sensitive data such as usernames and passwords.

There is a second benefit; with SSL you can verify that the server you are speaking with is actually the server you tried to reach. When you are on a network where you are not in control of the entire network stack, such as a work/university network or a public wifi point, it's possible that an attacker could 'man-in-the-middle' you by pretending to be the server you want to reach. SSL adds another layer of authentication that prevents attacks like these. Your browser will give a warning when a SSL certificate is actually not verified by an external party. You probably have seen messages like this when using so called 'self-signed certificates'. Self-signed certificates are an easy (and free) way of adding some level of security, however it's not perfect. Self-signed certificates still encrypt your connection so that the privacy of your connection is guaranteed, however it does not protect you from attackers that might try to hijack your connection by employing a 'man-in-the-middle' attack when you are using public networks.

Bytesized Apps and SSL

We offer a wide variety of apps and each of those apps offers their own SSL implementation. Almost all of them however use 'self-signed certificates', some of the apps can generate self-signed certificates for you, for others you might need to do it yourself. Luckily for a lot of the applications there is a simpler way around it with less configuration hassle. To explain how we need to step back for a minute and look at what we currently offer to reach your applications running on our servers.

All our Appboxes come with a 'pretty url' feature. It ensures you can always reach your supported application on a special url that's easy to remember such as http://username.servername.bysh.me/appname. The alternative is remembering the port your app is running on such as http://servername.bysh.me:PORT. We achieve this by having a web server running on the server that we feed new configuration every time you install a supported app. So when your browser requests http://jd.sacredheart.bysh.me/deluge you are actually speaking directly to the web server which in turn proxies the request directly to your application on it’s unique port. This has an extra benefit because almost all networks allow connections on port 80 even strict work networks this allows you to connect to your apps when the port your apps is running on is normally not allowed.

By making this web server we run listen on https to we could provide you with SSL access to your apps without having to configure and get a certificate for each individual application.

The problem(s)

The above is easier said than done. SSL certificates are unique and each domain requires their own certificate. Because we give out custom domains for each user this means that we would have to create a unique certificate for each user. Since each certificate costs money that would be a big problem. Luckily around the time we were working on this a new service popped-up called Let’s Encrypt! which promised free and automated SSL certificates. We quickly started coding to see if we could use this new service as a way around our initial problem.

Sadly this is when we ran into more problems. Let’s encrypt only allows 20 certificates to be issued per domain per week. The amount of churn we have combined with the amount of existing members means we would never ever get all the certificates issued. We found a way around this by means of a “SAN Certificate”. This is a specially crafted certificate that allows you to add 100 domains into one certificate. However this led us to the next problem. Just opening the SAN certificate would expose all 100 usernames in the certificate. We value privacy and exposing our members to the world seemed like a bad idea.

The next idea we had was just to create garbage domain names (http://7468697369736e6f74617265616c74657874.servername.bysh.me) and assign those randomly to each user. However that meant the domains were no longer rememberable and that you could still easily discover services running which we were not happy with.

The last option available to use was using wildcard domains. A wildcard domain allows you to use an asterisk in place of a name to allow it to work for multiple domains. For example the wildcard domain *.sacredheart.bysh.me will be valid for https://jd.sacredheart.bysh.me, https://turk.sacredheart.bysh.me, https://bob.sacredheart.bysh.me etc. It also has the added benefit of not leaking usernames since it’s valid for all possible options. There also was a clear downside: Let’s Encrypt doesn’t support wildcards. This means that for each server we need to pay for a wildcard certificate and we can’t roll it out in a programmatic fashion as the normal process of SSL certificates doesn’t have automation (in most cases). There is also a double wildcard domain *.*.bysh.me which in theory could work for all our servers with just one certificate but sadly the SSL spec is vague on the topic and therefore almost no browser considers certificates like these as valid.

Our solution

We decided to scratch all the Let’s encrypt code and implement SSL in two ways.

1. Setup self-signed certificates on all servers so that users can use encrypted communication for all supported apps.
2. Buy wildcard certificates for all our Appbox +Stream servers hosted by Bytesized.

When you visit your box’s dashboard now you will see three links for all apps that can be proxied through our web server

• Secure link. A link through https proxied through our web server.
• Normal link. A link through normal http proxied through our web server.
• Direct link. A link connecting directly to your app’s port. (some apps that are run by the web server directly won’t have this link)

Make sure you reboot your apps to ensure SSL supported is enabled for your box.

Future plans

We still think Let’s encrypt can be used and I have ideas for a ‘custom domain’ feature that allows you to bring your own domain name and let it run through our web server. Sadly I’m just one guy with too many things to code so it might be a while before I can bring this idea to fruition.