Sync with Rclone

One command copy/sync between local and remote directories using rclone. Made for simplicity and efficiency by excluding unnecessary files.

Version: 4.1 (2026-03-19) Repository: https://github.com/IBL-bioinfo/sync-with-rclone

Quick Start

1. Copy sync-with-rclone.sh and sync-with-rclone.config to your target folder
2. Edit sync-with-rclone.config to set your remote location and preferences
3. chmod u+x sync-with-rclone.sh
4. Run: ./sync-with-rclone.sh pull or ./sync-with-rclone.sh push

Usage

./sync-with-rclone.sh pull|push [options] [additional rclone parameters]

Options

• -h, --help — Show help message and exit
• --sync — Use rclone sync instead of default copy operation (WARNING: deletes files in destination not present in source)
• --exclude "[pattern]" — Exclude files matching pattern (can be used multiple times)
• --include "[pattern]" — Include files matching pattern (can be used multiple times), will automatically exclude all other files
• -y — Use with caution: skip confirmation prompts and run non-interactively

Examples

# Simple pull
./sync-with-rclone.sh pull

# Push with dry-run preview
./sync-with-rclone.sh push --dry-run

# Sync with automatic confirmation
./sync-with-rclone.sh push --sync -y

# Push only source files
./sync-with-rclone.sh push --include "src/**" --dry-run

# Push with custom rclone timeout
./sync-with-rclone.sh push --timeout 120m

Configuration

All settings are stored in sync-with-rclone.config:

• REMOTE_NAME — Name of your rclone remote, you must have this remote configured in rclone by running rclone config and following the prompts (you probably named your config e.g. rd for Research Drive)
• REMOTE_PATH — Path on the remote (e.g., /path/to/backup)
• ALLOW_PULL — Set to false to disable pull operations, useful for write-only backup scripts
• ALLOW_PUSH — Set to false to disable push operations, useful for read-only restore scripts
• RECORD_GIT_COMMIT — Set to true to record git commit info when pushing
• exclude — Array of additional patterns to exclude from config
• global_exclude — Array of patterns to always exclude, do not change unless you know what you're doing (includes common "no need to backup" files like *.tmp, etc.)

Features

• Built-in ignore list — Automatically excludes common "no need to backup" files
• Git support — Records commit hash and status for git repositories (without syncing .git directories)
• Flexible filtering — Include/exclude patterns support via command-line or config
• Sync with safety — Sync operation shows preview of deletions and asks for confirmation
• Non-interactive mode — Use -y flag to skip prompts or --dry-run for preview without changes

Git Repository Tracking

Git itself is a version control system. The small files git generated in the .git directory are not necessary to backup, and can be easily regenerated by cloning the repository again. Most importantly, as creating files are time consuming on the remote, the small files in .git directory will greatly slow down the rclone progress, especially for large repositories. This is why the script excludes the .git directory by default. However, it can be useful to keep track of the git commit hash and status at the time of backup for reference.

⚠️ WARNING: Local git repositories (without a remote) will lose their version tracking after pushing to remote. Only the latest commit hash and status will be recorded in the text file. If you want to keep the full git history, configure a remote (GitHub, GitLab, etc.) and push to it. That remote is your actual version tracking—the backup script only records the current state for reference.

Thus, this script does not sync the .git directory at all, but when RECORD_GIT_COMMIT=true (default) and pushing, it records the current git commit hash, remote URLs, and status in a [REPOSITORY_DIR]_git_repository.txt file in the same directory as the repository. This allows you to keep track of the repository's state at the time of backup. This includes:

• The current commit information of the repository (commit hash, date, author, message)
• The remote URLs configured for the repository
• The current status of the repository (modified, added, deleted, untracked files)

Example content of the generated git repository information file:

============ git repository information ============
Date: Sun Jul 19 12:16:26 CEST 2025
Commit Hash: 3688o888id6fb...
Remote(s):
origin: [email protected]:user/repo.git
backup: [email protected]:group/repo.git
Git Status:
Working directory clean
============ git repository information ============
Date: Sun Jul 20 14:09:18 CEST 2025
Commit Hash: 5uuuuuuuuuuuu...
Remote(s):
origin: [email protected]:user/repo.git
backup: [email protected]:group/repo.git
Git Status:
 M file-modified.txt
A  file-added.txt
?? file-untracked.txt
 D file-deleted.txt

RECENT UPDATES (3.0 to 4.1)

Version 4.1 (2026-03-19): Cleaner Dry-Run No-Change Output

This release improves clarity when a dry-run finds no file-level changes:

1. CLEAR NO-CHANGE MESSAGE

   • Added explicit output: "Nothing will change." when dry-run has no pending changes
   • Avoids confusion caused by only seeing rclone summary/check counters

2. EARLY EXIT WHEN NO CHANGES

   • Real operation now stops before confirmation/execution if dry-run pre-check reports no changes
   • Prevents unnecessary prompts and no-op runs

Version 4.0 (2026-03-10): Improved Safety for Sync Operations

This release prioritizes safety and usability when performing sync operations to prevent accidental data loss:

1. SAFER DEFAULT BEHAVIOR

   • Default operation is now "copy" (safer) instead of fixing it in the config file
   • The OPERATION variable is no longer set in the config file

2. COMMAND-LINE SYNC FLAG

   • Added --sync flag to switch to sync operation at runtime
   • Clear warnings in help output about destructive behavior
   • Allows users to override the safe default (copy) only when intentional

3. DELETION PREVIEW AND CONFIRMATION

   • For sync operations, script performs dry-run first
   • Shows user which files will be permanently deleted
   • Requires explicit confirmation before proceeding
   • Non-interactive mode (-y) waits 5 seconds before auto-confirming

4. IMPROVED DOCUMENTATION

   • Updated version numbers in script and help output
   • Clarified usage patterns and behavior in help text
   • Better explanations of safe vs dangerous operations