Rclone

According to the official website:

“Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors’ web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat.”

With this tool we can automatically move all files in a specific folder that is located locally on the robot to Google Drive, as well as easily add other custom functionalities for our use-cases.

Additionally, Rclone supports a wide range of providers, which allows us to easily switch between providers without having to adjust the functionality of the file upload itself, making this a future-proof choice.

Configuration

If not present yet, first install rclone locally on the computer:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Rclone also needs to be configured locally, for which this guide can be followed.

The configuration requires getting a token from Google drive. If this token has not been created yet, follow this guide to set up a client_id. It is important to also publish the app, otherwise the access token needs to be refreshed every 7 days. Find the client_id back at:

Google Cloud > APIs & Services > select the project that contains the client ID at the top of the page > Credentials > select the client under OAuth 2.0 Client IDs > Clients > Additional information (> Client secrets)

Possibly a message appears that downloading the client secrets is no longer available. In case of a lost secret, a new one needs to be created.

When Rclone has been correctly configured, $HOME/.config/rclone/rclone.conf should exist with at least the following contents:

[<rclone_configured_name>]
type = drive
client_id = <client_id>
client_secret = <client_secret>
scope = drive
token = {"access_token":"...","token_type":"...","refresh_token":"...","expiry":"..."}
team_drive =

Note: it is not necessary to follow all of these steps for every new device. Only rclone config needs to be run on the new device, copy the existing client_id and client_secret, and generate a new token on a computer that has access to a browser.

Docker Setup

To activate the Rclone file synchronisation task, one requires to activate the corresponding docker container. The rclone_filesync docker container in our repository contains this functionality. This container expects a rclone.env file in the home directory of the host device with the following content:

# Filesync configuration
RCLONE_READY_DIR=<data_ready_directory_path>
RCLONE_REMOTE=<rclone_configured_name:unique_robot_name>
RCLONE_LOG_LEVEL=<log_level>
RCLONE_FILE_AGE_SECONDS=<integer>
RESCAN_INTERVAL_SECONDS=<integer>

# Container timezone
TZ=UTC

Create the file if it doesn’t exist yet and fill the required variables. Next, one can start the container:

cd alliander_robotics/rclone
docker compose up -d

To inspect the logs, run the following command in a terminal:

docker compose logs -f rclone_filesync