Setting Up Rclone, Merger Fs And Crontab For Automated Cloud Storage - Wiki wiki page

Setting up Rclone, Mergerfs and Crontab for automated cloud storage

About

Precursor

This guide will take you through a specific setup. It's not the only one perfect setup that everyone should use, but it has proven to be easy to maintain and to run smoothly for a lot of people so far.

IMPORTANT NOTE: Please remember that while Bytesized Hosting does provide these tools for their members, they don't provide any official support for them. If something goes wrong, you're on your own. Please read this news post.

Rclone

Rclone is a tool that allows you to mount remote filesystems. This tool will be used for attaching Google Drive to your box. In this guide we will be using an Rclone crypt remote for remote file encryption, but that part is completely optional. We will not be using a cache remote.

Mergerfs

Mergerfs is a tool that allows you to merge several other directories into one virtual directory and control the flow of data between them. This tool will be used to:

  • Expose a single access point to your media files for your apps (Sonarr, Plex, etc.) which effectively hides the difference
  • Employ a local, temporary file storage that ensures a low amount of API hits and no stuck apps because of a frozen mount.
  • Allow Plex to receive filesystem notifications so that it is immediately and automatically aware of new or changed media files. This is not possible with just Rclone.

Media handling

While you are free to chose how to handle and rename your media, it is the author's recommendation that you use the built-in media handling and renaming in Sonarr/Radarr. Combined with Mergerfs, this works seamlessly. This guide will be based on the assumption that you don't use an external tool such as Filebot or Olaris Rename.

Crontab

Crontab is a tool that allows you to run scripts automatically at a set interval. This tool will be used to automatically transfer files from the local temporary storage to the remote at an interval of your choice.

Setting up Rclone

Requirements

For this part you will need to have already prepared access to the Drive API. You will need a Client ID and Secret. More info can be found here.

Rclone setup is done through the terminal. You can either connect through SSH or through VNC.

Drive remote setup

  1. Open a terminal and type rclone config
  2. Type n for new remote.
  3. As name, type gdrive
  4. As storage, type 13 for Google Drive.

    Note: You can use any storage solution you like, but this guide is based on Google Drive.

  5. Enter your Client ID. DO NOT skip this part.

  6. Enter your Client Secret. DO NOT skip this part.

  7. As scope, type 1 for full access.

  8. As root folder ID, leave blank.

  9. For service account file, leave blank.

  10. For edit advanced config, type n for no.

  11. For use auto config, type n for no.

  12. At this point you will be presented with a URL that you need to enter in a browser and grant access with your Google account. You will then be presented with a validation code you need to paste in the terminal.

    Note: If you are using GSuite, you will need to grant access with your GSuite account, not your personal Google account.

  13. For configure as team drive, type n for no.

  14. Verify that everything is OK and type y for yes.

Crypt remote setup

  1. While still in the rclone config screen, type n for new remote.
  2. As name, type gcrypt
  3. As storage, type 10 for crypt.
  4. As remote, type gdrive: plus the root path of your media on Google Drive. It is possible to just put your media in the root folder, but it is not recommended.

    Example: On Google Drive, I have a folder named rclone and I want to place my encrypted media inside that. I will enter gdrive:rclone

  5. As filename encryption, type 1 for standard.

  6. As directory name encryption, type 1 for true.

  7. As pass, type g for generate.

  8. As strength, type 128 for 128 bits.

  9. You will be presented with the generated password. Save it, and DO NOT LOSE IT. Without this, you will be unable to decrypt your media in the future, should you need to.

  10. As salt, type g for generate.

  11. As strength, type 128 for 128 bits.

  12. You will be presented with the generated salt. Save it, and DO NOT LOSE IT. Without this, you will be unable to decrypt your media in the future, should you need to.

  13. Verify that everything is OK and type y for yes.

  14. You are now done setting up your remotes. Type q to quit Rclone config.

Setting up directories and scripts

Directories

In the terminal, enter:

  • mkdir ~/mnt
  • mkdir ~/mnt/gdrive
  • mkdir ~/mnt/media_merge
  • mkdir ~/media_tmp
  • mkdir ~/scripts
  • mkdir ~/.config/mergerfs

Scripts

Uploadmedia

This script will be run automatically at a set interval to transfer the media files from the temporary local directory to the remote.

In the terminal:

  • Type nano ~/scripts/uploadmedia
  • Paste in:

    #!/bin/bash
    
    echo "Moving local media files to remote..."
    screen -dmS uploadmedia /usr/local/bin/rclone move ~/media_tmp gcrypt: --delete-empty-src-dirs -v --stats 5s
    
  • Press Ctrl + X

  • Press Y for yes.

  • Press Enter to confirm file name.

  • Type chmod +x ~/scripts/uploadmedia to make the script executable.

Startup script

This script is responsible for starting your Rclone and Mergerfs mounts. It will also do so automatically on any server restarts initiated by staff, and when you click the Restart box button on the control panel.

For this script you will need some information beforehand:

  • User ID and Group ID: Enter id in a terminal to get this information. They are signified by uid and gid respectively.
  • A user agent string. This is just a random string of letters and numbers. You can just spam your keyboard for this if you want. It doesn't matter what it is, as long as you have one.

In the terminal:

  • Type nano ~/.startup/gdrive
  • Paste in:

    #!/bin/bash
    
    USER_ID=XXXXX
    GROUP_ID=XXXXX
    USER_AGENT=XXXXXXXXXXXXXXXXX
    
    export TMPDIR=$HOME/tmp
    PID_FILE=$HOME/.config/rclone/rclone.pid
    if [ -e $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if ! kill -0 $PID > /dev/null 2>&1; then
            echo "Removing stale $PID_FILE"
            rm $PID_FILE
        fi
    fi
    
    /sbin/start-stop-daemon -S --pidfile $PID_FILE --make-pidfile -u $USER -d $HOME -b -a /usr/local/bin/rclone -- mount gcrypt: ~/mnt/gdrive --allow-other --user-agent="$USER_AGENT" --timeout 1h --uid $USER_ID --gid $GROUP_ID --vfs-cache-mode minimal
    
    PID_FILE=$HOME/.config/mergerfs/mergerfs.pid
    if [ -e $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if ! kill -0 $PID > /dev/null 2>&1; then
            echo "Removing stale $PID_FILE"
            rm $PID_FILE
        fi
    fi
    
    /sbin/start-stop-daemon -S --pidfile $PID_FILE --make-pidfile -u $USER -d $HOME -b -a /usr/bin/mergerfs -- -f -o defaults,sync_read,auto_cache,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff $HOME/media_tmp:$HOME/mnt/gdrive $HOME/mnt/media_merge
    
  • Replace the X's at the top with your own values.

  • Press Ctrl + X

  • Press Y for yes.

  • Press Enter to confirm file name.

  • Type chmod +x ~/.startup/gdrive to make the script executable.

Shutdown script

This script is responsible for shutting down your Rclone and Mergerfs mounts if required.

In the terminal:

  • Type in nano ~/.shutdown/gdrive
  • Paste in:

    #!/bin/bash
    
    PID_FILE=$HOME/.config/rclone/rclone.pid
    /sbin/start-stop-daemon --pidfile $PID_FILE -u $USER -d $HOME -K -a /usr/local/bin/rclone
    
    if [ -e $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if ! kill -0 $PID > /dev/null 2>&1; then
            echo "Removing stale $PID_FILE"
            rm $PID_FILE
        fi
    fi
    
    PID_FILE=$HOME/.config/mergerfs/mergerfs.pid
    /sbin/start-stop-daemon --pidfile $PID_FILE -u $USER -d $HOME -K -a /usr/bin/mergerfs
    
    if [ -e $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if ! kill -0 $PID > /dev/null 2>&1; then
            echo "Removing stale $PID_FILE"
            rm $PID_FILE
        fi
    fi
    
  • Press Ctrl + X

  • Press Y for yes.

  • Press Enter to confirm file name.

  • Type chmod +x ~/.shutdown/gdrive to make the script executable.

Setting up Crontab

As mentioned earlier, Crontab lets you automatically run a script at a set interval. We've already created the uploadmedia script, so all we need to do now is to tell Crontab how often to run it.

In the terminal:

  • Type crontab -e
  • If prompted, select nano as a text editor.
  • Below the long explanation in the crontab file, add a new line: 0 3 * * * ~/scripts/uploadmedia This will move media from the local temporary location to the remote every day at 3 am, based on the server's time. If you want to run it at another time, read the explanation in the crontab file and edit the line you entered.
  • Press Ctrl + X
  • Press Y for yes.
  • Press Enter to confirm file name.

Manual mount and setting up apps

At this point everything is set up. All that's left to do is to run your startup script and and point your apps at the new virtual directory.

Because of this virtual directory managed by Mergerfs, your apps can't distinguish between media existing in the local temporary location, or on the remote.

Mergerfs

Mounting

Note that this only has to be done once or in case of issues, as the mounts will auto-start on future box/server restarts.

In the terminal:

  • Type ~/.startup/gdrive
  • Type ps -ef and verify that rclone and mergerfs are both in the list.

Sonarr / Radarr

Before this part you will need to make the TV Shows and Movies directories if they don't already exist.

In the terminal:

  • Type mkdir ~/mnt/media_merge/TV\ Shows
  • Type mkdir ~/mnt/media_merge/Movies

Root path

  1. Go to add a new TV show.
  2. In the root folder drop-down, select Add a new path
  3. The browser does not work properly, so you have to type in your home directory followed by mnt/media_merge/TV Shows for Sonarr and mnt/media_merge/Movies for Radarr.

    Note: Your home directory can be found on the box control panel in the right side.

    Example: /home/hd27/myusername/mnt/media_merge/TV Shows

Plex

The final step is to point Plex at the virtual directory. At the moment Plex is not able to see it because we started the mounts while it was still running, so you will have to restart Plex from the control panel before proceeding.

  • In Plex, go to Settings and Manage libraries
  • Create or edit the TV Shows and Movies libraries.
  • Point them at data/mnt/media_merge/TV Shows and data/mnt/media_merge/Movies respectively.

Extra

By default Plex will automatically remove the metadata for media where the files are missing. This is an issue when we're using a remote for two reasons:

  1. If the mount for fails for some reason, Plex will clear out your entire library.
  2. When Sonarr/Radarr performs a quality upgrade, Plex will clear out that show/movie.

In order to circumvent this, open the Plex web interface and go to Settings > Library. Disable the option Empty trash automatically after every scan and Plex will no longer remove missing media but rather just mark it as trash. When the mount is available again or the quality upgrade has finished, the media will automatically be unmarked as trash. The only downside to this is that when you intentinally remove media externally, you have to manually also remove it from your Plex library.


Last Author Contributors Versions Last update
Alchemist Maran, emilnymann96 8 Tue, 21 Jul 2020 09:21:11 +0200