Let's talk about RAM baby 🎵 - Bytesized Hosting
Let's talk about RAM baby 🎵
It's been a while since our last news post but sadly it's time to have a discussion, about RAM.
When I started building the Appbox platform a couple of years ago we initially supported 5 apps, at that point memory limitations were not really an issue. The apps we supported had been in development for years which inturn meant they were fairly stable and optimised with memory in mind. However as time went on we added support for more and more applications, we now support over 35 applications. Some of them built well... others not so much.
When we allowed Mono applications (Radarr, Sonarr and Jackett) things started to look a bit bleaker, most of these apps used a lot of RAM some of it because of leaks but also because of Mono developers had less control over the allocation of said RAM. To overcome this we decided to invest into more RAM, filling up all the slots in our +Stream servers to a total of 256GB. We also helped out various applications by debugging their memory issues resulting in a lot of memory- leak fixes, especially for Radarr.
Things went smooth for a while, until the rise of Rclone and Plexdrive. These applications are built to cache and therefor use large amounts of RAM, especially when not configured properly. Even with the rise of these applications we were doing ok until Plex released the first 0.15.X version and thereby introduced memory leaks into the Plex Media Server process. Because of the leaks I've estimated Plex is now using around 3TB of memory on our servers in total. The average memory consumption has gone from 250MB to 1.2GB per Plex instance. Add to that a bug in the latest HWE Ubuntu kernel that makes memory reclaiming more expensive then it should be and we have a recipe for disaster.
All of this has resulted in less than stable servers the last few weeks. This is something that really hurts me because I pride myself on providing an awesome service, but I must admit that the last few weeks on the +Stream servers have not been. I've already reached out to Plex via various platforms but they have yet to acknowledge this is an issue they are working on fixing.
We need to discern two types of solutions; long term and short term. Let's start with the short term solutions first.
- To begin restarting Plex instances every morning at 09:00 CEST. This being the least impactful time for our userbase.
- Start doing weekly forced upgrades on all Plex server instances to see if the newer versions solve the memory issues.
- Start warning people using an insane amount of memory.
In the long term we are going to start limiting RAM usage per user. Nobody is to blame for this but when you have unlimited resource available to use then you have no reason to uninstall unused apps or optimize applications like Rclone to use less memory. There is simply no other way to ensure fair usage across the board as we increase our app library. I'm not setting any limits just yet but know that they are coming.
Lastly in the upcoming weeks we are going to upgrade the +Stream servers to the latest Ubuntu LTS release. There have been a couple of optimisations in the newer kernel releases that will help with our specific usage patterns and will make memory reclaiming snappier and therefor the servers happier.
For now our priority is making sure our servers are stable again by employing the short term solutions. Once that's done we will continue the discussion on RAM limitations per user.
Once again my apologies for the stability of some of the servers the past few weeks, things will improve and as always let me know your ideas and thoughts in the comments.
Update 1: You can now view your RAM usage. To get a feel for how much you are using. The baseline should be around 1.5GB.
Update 2: I've changed the restart frequency for Plex to every five days instead of every day. Please look for recent memory leak / memory usage issues on the Plex forums and let your voices be heard.
Update 3: Plex is now only restarted if your instance is leaking over 1GB of memory and that check is being done once every three days.
Please sign-up for an account to join the discussion.