Add root remote capability to combine backend

[Anthchirp]

The associated forum post URL from https://forum.rclone.org

What is your current rclone version (output from rclone version)?

rclone v1.70.0-DEV

• os/version: Microsoft Windows 11 Pro 24H2 24H2 (64 bit)
• os/kernel: 10.0.26100.3194 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.22.1
• go/linking: static
• go/tags: none

What problem are you are trying to solve?

I want to use kopia to backup large datasets to cloud storage. Kopia can use object stores and rclone as backend.

In Kopia the first character of the stored file name is used to indicate the object type:
bulk data starts with a p, metadata with a q, indices with an x, etc.

I am a cheapskate: I want to use tiering to reduce the cost as much as possible. Bulk data I would like to drop into a place that is unbounded yet affordable, say the Google Cloud Storage archive tier, and metadata and indices I want to keep somewhere where storage size is limited but random access is free, say Onedrive.

Rclone's combine works on paths, not file prefixes, but luckily kopia supports sharding, which can be configured so that the first character of the file name is turned into a directory path.
However, kopia also wants to store a sharding configuration file .shards, alongside some other stuff into the root path, and there may be other unanticipated prefixes, such as n, and maybe more prefixes in the future.
So I can't just use rclone combine with 26 remotes a-z - and that wouldn't be a particularly nice solution anyway.

I looked into union, but that doesn't fit my use case - I want guaranteed behaviour based on the top level path alone.

How do you think rclone should be changed to solve that?

Copying the example from the docs:

$ rclone tree s3:imagesbucket
/
├── image1.jpg
└── image2.jpg

$ rclone tree drive:important/files
/
├── file1.txt
└── file2.txt

Using combine with upstreams = images=s3:imagesbucket files=drive:important/files gives:

$ rclone tree combined:
/
├── files
│   ├── file1.txt
│   └── file2.txt
└── images
    ├── image1.jpg
    └── image2.jpg

My proposal is to allow adding in one upstream remote at the root level,
upstreams = images=s3:imagesbucket =drive:important/files to give:

$ rclone tree combined_with_root:
/
├── file1.txt
├── file2.txt
└── images
    ├── image1.jpg
    └── image2.jpg

This should be relatively easy to achieve.
It is consistent with the current configuration spec, which explicitly forbids = with empty directory name.
If may also help resolve comments about current combine's root behaviour.

This would introduce a new edge case when the root remote contains a file/directory that has a name conflict with another upstream remote. The resolution would be that the upstream remote wins, which is consistent with mounting another filesystem on top of an existing one, and would result in the overlaid file/directory being hidden.

I had a look at the existing combine implementation and was able to get working List and file reading support for this, so I now wanted to gauge whether there is any interest in having this feature, before I dive deeper into the other operations.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.