Setting up Rclone and Crontab for automated cloud storage

Table of contents

1. About this guide
2. Directories
3. Setting up Rclone
   a. Requirements
   b. Drive remote setup
   c. Crypt remote setup
   d. Union remote setup
4. Setting up scripts
   a. Upload media
   b. Startup script
   c. Shutdown script
5. Setting up Crontab
6. Final steps
   a. Mounting
   b. Sonarr / Radarr
   c. Plex

About this guide

Precursor

This guide will take you through a specific setup. It's not the only one perfect setup that everyone should use, but it has proven to be easy to maintain and to run smoothly for a lot of people so far.

IMPORTANT NOTE: Please remember that while Bytesized Hosting does provide these tools for their members, they don't provide any official support for them. If something goes wrong, you're on your own. Please read this news post.

Updates

04/01/2022

Mergerfs is no longer used in this guide and has been replaced with an Rclone Union remote, which serves the same purpose but without the added complexity of Mergerfs. Thanks to "@Error Loading Username" on Discord for letting me know about it.

The Archived Mergerfs version of this page can be found here.

Rclone

Rclone is a tool that allows you to mount remote filesystems. This tool will be used for attaching Google Drive to your box. In this guide we will be using an Rclone crypt remote for remote file encryption, but that part is completely optional. We will be using a union remote to merge the remote and local directories into one virtual directory and control the flow of data between them. This will be used to:

• Expose a single access point to your media files for your apps (Sonarr, Plex, etc.) which effectively hides the difference
• Employ a local, temporary file storage that ensures a low amount of API hits and no stuck apps because of a frozen mount.
• Allow Plex to receive filesystem notifications so that it is immediately and automatically aware of new or changed media files.

We will not be using a cache remote.

Note: Rclone updates periodically, and the numbers change on the storage remote type list. As you use this guide, reference the name of the remote type you need since the number may not match later. ie Google Drive (NOT Google Cloud Storage), Encrypt, and Union.

Mergerfs

Mergerfs is no longer used as part of this setup, as it has been replaced with an Rclone Union remote.

The Archived Mergerfs version of this page can be found here.

Mergerfs is a tool that allows you to merge several other directories into one virtual directory and control the flow of data between them. This tool will be used to:

• Expose a single access point to your media files for your apps (Sonarr, Plex, etc.) which effectively hides the difference
• Employ a local, temporary file storage that ensures a low amount of API hits and no stuck apps because of a frozen mount.
• Allow Plex to receive filesystem notifications so that it is immediately and automatically aware of new or changed media files. This is not possible with just Rclone.

Media handling

While you are free to chose how to handle and rename your media, it is the author's recommendation that you use the built-in media handling and renaming in Sonarr/Radarr. Combined with a Union remote, this works seamlessly. This guide will be based on the assumption that you don't use an external tool such as Filebot or Olaris Rename.

Crontab

Crontab is a tool that allows you to run scripts automatically at a set interval. This tool will be used to automatically transfer files from the local temporary storage to the remote at an interval of your choice.

Directories

In the terminal, enter:

• mkdir -p ~/mnt/media_merge ~/media_tmp ~/scripts

Setting up Rclone

Requirements

For this part you will need to have already prepared access to the Drive API. You will need a Client ID and Secret. More info can be found here.

Rclone setup is done through the terminal. You can either connect through SSH or through VNC.

Drive remote setup

1. Open a terminal and type rclone config
2. Type n for new remote.
3. As name, type gdrive

4. As storage, type 13 for Google Drive.

   Note: You can use any storage solution you like, but this guide is based on Google Drive.

5. Enter your Client ID. DO NOT skip this part.

6. Enter your Client Secret. DO NOT skip this part.

7. As scope, type 1 for full access.

8. As root folder ID, leave blank.

9. For service account file, leave blank.

10. For edit advanced config, type n for no.

11. For use auto config, type n for no.

12. At this point you will be presented with a URL that you need to enter in a browser and grant access with your Google account. You will then be presented with a validation code you need to paste in the terminal.

    Note: If you are using GSuite, you will need to grant access with your GSuite account, not your personal Google account.

13. For configure as team drive, type n for no.

14. Verify that everything is OK and type y for yes.

Crypt remote setup

1. While still in the rclone config screen, type n for new remote.
2. As name, type gcrypt
3. As storage, type 10 for crypt.

4. As remote, type gdrive: plus the root path of your media on Google Drive. It is possible to just put your media in the root folder, but it is not recommended.

   Example: On Google Drive, I have a folder named rclone and I want to place my encrypted media inside that. I will enter gdrive:rclone

5. As filename encryption, type 1 for standard.

6. As directory name encryption, type 1 for true.

7. As pass, type g for generate.

8. As strength, type 128 for 128 bits.

9. You will be presented with the generated password. Save it, and DO NOT LOSE IT. Without this, you will be unable to decrypt your media in the future, should you need to.

10. As salt, type g for generate.

11. As strength, type 128 for 128 bits.

12. You will be presented with the generated salt. Save it, and DO NOT LOSE IT. Without this, you will be unable to decrypt your media in the future, should you need to.

13. Verify that everything is OK and type y for yes.

Union remote setup

1. While still in the rclone config screen, type n for new remote.
2. As name, type union
3. As storage, type 30 for Union. Check the presented list at the time.

4. As upstreams, type gcrypt::nc /home/HD/USERNAME/media_tmp

   Note: Replace HD and USERNAME with your own values found on the dashboard.

5. As action policy, leave blank for default

6. As create policy, leave blank for default

7. As search policy, leave blank for default

8. As cache time, leave blank for default

9. Verify that everything is OK and type y for yes.

10. You are now done setting up your remotes. Type q to quit Rclone config.

Setting up scripts

Upload media

This script will be run automatically at a set interval to transfer the media files from the temporary local directory to the remote.

In the terminal:

• Type nano ~/scripts/uploadmedia

• Paste in:

  #!/bin/bash
  
  echo "Moving local media files to remote..."
  screen -dmS uploadmedia /usr/bin/rclone move ~/media_tmp gcrypt: --delete-empty-src-dirs -L -v --stats 5s
  

• Press Ctrl + X

• Press Y for yes.

• Press Enter to confirm file name.

• Type chmod +x ~/scripts/uploadmedia to make the script executable.

Startup script

This script is responsible for starting your Rclone mount. It will also do so automatically on any server restarts initiated by staff, and when you click the Restart box button on the control panel.

For this script you will need a user agent string. This is just a random string of letters and numbers. You can just spam your keyboard for this if you want. It doesn't matter what it is, as long as you have one.

In the terminal:

• Type nano ~/.startup/gdrive

• Paste in:

  #!/bin/bash
  
  USER_AGENT=XXXXXXXXXXXXXXXXX
  
  export TMPDIR=$HOME/tmp
  PID_FILE=$HOME/.config/rclone/rclone.pid
  if [ -e $PID_FILE ]; then
      PID=`cat $PID_FILE`
      if ! kill -0 $PID > /dev/null 2>&1; then
          echo "Removing stale $PID_FILE"
          rm $PID_FILE
      fi
  fi
  
  /sbin/start-stop-daemon -S --pidfile $PID_FILE --make-pidfile -u $USER -d $HOME -b -a /usr/bin/rclone -- mount union: ~/mnt/media_merge --allow-other --user-agent="$USER_AGENT" --timeout 1h --vfs-cache-mode full
  

• Replace the X's at the top with your own values.

• Press Ctrl + X

• Press Y for yes.

• Press Enter to confirm file name.

• Type chmod +x ~/.startup/gdrive to make the script executable.

Shutdown script

This script is responsible for shutting down your Rclone mount if required.

In the terminal:

• Type in nano ~/.shutdown/gdrive

• Paste in:

  #!/bin/bash
  
  PID_FILE=$HOME/.config/rclone/rclone.pid
  /sbin/start-stop-daemon --pidfile $PID_FILE -u $USER -d $HOME -K -a /usr/bin/rclone
  
  if [ -e $PID_FILE ]; then
      PID=`cat $PID_FILE`
      if ! kill -0 $PID > /dev/null 2>&1; then
          echo "Removing stale $PID_FILE"
          rm $PID_FILE
      fi
  fi
  

• Press Ctrl + X

• Press Y for yes.

• Press Enter to confirm file name.

• Type chmod +x ~/.shutdown/gdrive to make the script executable.

Setting up Crontab

As mentioned earlier, Crontab lets you automatically run a script at a set interval. We've already created the uploadmedia script, so all we need to do now is to tell Crontab how often to run it.

In the terminal:

• Type crontab -e
• If prompted, select nano as a text editor.
• Below the long explanation in the crontab file, add a new line: 0 3 * * * ~/scripts/uploadmedia This will move media from the local temporary location to the remote every day at 3 AM, based on the server's time. If you want to run it at another time, read the explanation in the crontab file and edit the line you entered.
• Press Ctrl + X
• Press Y for yes.
• Press Enter to confirm file name.

Final steps

At this point everything is set up. All that's left to do is to run your startup script and and point your apps at the new virtual directory.

Because of this virtual Union directory managed by Rclone, your apps can't distinguish between media existing in the local temporary location, or on the remote.

Mounting

Note that this only has to be done once or in case of issues, as the mounts will auto-start on future box/server restarts.

In the terminal:

• Type ~/.startup/gdrive
• Type ps -ef and verify that rclone is in the list.

Sonarr / Radarr

Before this part you will need to make the TV Shows and Movies directories if they don't already exist.

In the terminal:

• Type mkdir ~/mnt/media_merge/TV\ Shows
• Type mkdir ~/mnt/media_merge/Movies

Root path

1. Go to add a new TV show.
2. In the root folder drop-down, select Add a new path

3. The browser does not work properly, so you have to type in your home directory followed by mnt/media_merge/TV Shows for Sonarr and mnt/media_merge/Movies for Radarr.

   Note: Your home directory can be found on the box control panel in the right side.

   Example: /home/hd27/myusername/mnt/media_merge/TV Shows

Plex

The final step is to point Plex at the virtual directory. At the moment Plex is not able to see it because we started the mounts while it was still running, so you will have to restart Plex from the control panel before proceeding.

• In Plex, go to Settings and Manage libraries
• Create or edit the TV Shows and Movies libraries.
• Point them at data/mnt/media_merge/TV Shows and data/mnt/media_merge/Movies respectively.

Extra

By default Plex will automatically remove the metadata for media where the files are missing. This is an issue when we're using a remote for two reasons:

1. If the mount for fails for some reason, Plex will clear out your entire library.
2. When Sonarr/Radarr performs a quality upgrade, Plex will clear out that show/movie.

In order to circumvent this, open the Plex web interface and go to Settings > Library. Disable the option Empty trash automatically after every scan and Plex will no longer remove missing media but rather just mark it as trash. When the mount is available again or the quality upgrade has finished, the media will automatically be unmarked as trash. The only downside to this is that when you intentionally remove media externally, you have to manually also remove it from your Plex library.