Deglobalise rclone config

Hi,
I'm using rclone as a library but so far I had to prevent concurrent access (rclone config updates).
I was wondering if there is a plan to make rclone usable with various configs in parallel.

Thanks!

Deglobalise rclone config

At the moment having global config is limiting rclone's ability to run different processes concurrently.

This is not noticeable in the command line, but when using rclone via the API or as a library it is very noticeable.

Globals which need fixing

Rclone global config is registered by rc.AddOption and these can be seen using the rclone API with

rclone rc --loopback options/get

The top level general config blocks are these - these are used for all rclone commands more or less.

filter - filters in use for listing
main - configures the general non-backend flags
log - configures --log-file/format/syslog
rc - configures remote control server (and rclone rcd)

These config blocks are used only by the relevant commands

dlna - configures rclone serve dlna
ftp - configures rclone serve ftp
http - configures rclone serve http
mount - configures rclone mount
sftp - configures rclone serve sftp
vfs - configures vfs flags for rclone mount/serve

High level plan

Create a system to attach the current configuration to a context.Context.

Wherever global config is used, fetch it from the context, so instead of reading config like this

fs.Config.DryRun

Read it with

fs.GetConfig(ctx).DryRun

Where fs.GetConfig will read the ctx looking for a key for the main config. If one isn't found then it will use the (now private) default config.

This will need functions to create the new config contexts - maybe with a merge non-default values into the new config. This will enable use of rclone as a library with non-global config. So something like

myConfig := fs.NewConfig()
myConfig.DryRun = true // set custom config here
newCtx := fs.SetConfig(ctx)

This will need a similar Set/Get function for each config block, however only the global config is widely spread througout the rclone code so there may be a neater way of doing it. This needs a bit of thought.

As part of this plan the now global config variables should be made package private so they can't be read from different packages.

Potential difficulties

Context everwhere

Rclone was started (at the time of go 1.0!) well before the context.Context system was invented. There has been some effort threading context through rclone, but there are likely more places that it needs to go. So for this change to be successful it will be necessary to thread context through more of rclone.

This is by and large a mechanical process and with the right editor support it doesn't take too long.

It will generate a lot of changes though!

See: #3257 for background.

Config everywhere

fs.Config is used in 91 files in rclone so doing the change to use a context driven configuration will generate a lot of changes. Again these are mostly mechanical in nature and probably can be scripted.

filter.Active is used in 13 files.

Extension work

Use command line flag strings to create the context

So instead of the above, you'd write something like

newCtx := config.FromFlags(ctx, `--dry-run --bwlimit 10M --exclude "*.jpg"`)

This would enable use from the API which would make the API much more usable

These flags could then go in the backend config say as a flags section for each backend.

fs: Add context to NewFs #3257 #4685

test: Add context to mockfs.NewFs #3257 #4685

fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685

fs: add context.Context to fs.GetModifyWindow #3257 #4685

accounting: add context.Context #3257 #4685

config: add context.Context #3257 #4685

fs: Add context to NewFs #3257 #4685

test: Add context to mockfs.NewFs #3257 #4685

fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685

fs: add context.Context to fs.GetModifyWindow #3257 #4685

accounting: add context.Context #3257 #4685

@ncw

@ncw Thanks for your guidance, I've mostly managed to rebase our fork onto 1.55.0. I have two questions that have come up:

The Fs constructors now take a context and often seem to store that context internally. The fundamental sync operations (like syncCopyMove) also take a context. Does the Fs pluck the current filter from the context with which it was constructed, or from the context passed into syncCopyMove?
It looks like there is now an explicit call needed to load the config file, as my test suite is failing with many errors like this: internal error: no config file system found. Did you call configfile.LoadConfig(ctx)?
Right now our application fakes a config file by assigning to the ConfigFileGet and ConfigFileSet function pointers. Is there a call I can make to satisfy the above error without actually loading anything from the file system?

Thanks again for all of your excellent work on this library!

Thanks for your guidance, I've mostly managed to rebase our fork onto 1.55.0. I have two questions that have come up:

The Fs constructors now take a context and often seem to store that context internally. The fundamental sync operations (like syncCopyMove) also take a context. Does the Fs pluck the current filter from the context with which it was constructed, or from the context passed into syncCopyMove?

The Fs only uses the filter for things like --low-level-retries. Filters are used from the one passed into syncCopyMove. They can of course be the same one if you want.

It looks like there is now an explicit call needed to load the config file, as my test suite is failing with many errors like this: internal error: no config file system found. Did you call ?
Right now our application fakes a config file by assigning to the ConfigFileGet and ConfigFileSet function pointers. Is there a call I can make to satisfy the above error without actually loading anything from the file system?

You can do several things...

The easiest thing for you would be to call SetConfigPath with an empty string indicating no config file, then call configfile.LoadConfig(ctx) as well as overriding ConfigFileGet and ConfigFileSet as you do at the moment

rclone/fs/config/config.go

Line 317 in 08a2df5

func SetConfigPath(path string) (err error) {

Or you can implement the richer Storage interface

rclone/fs/config/config.go

Line 68 in 08a2df5

type Storage interface {

And then call SetStorage to set it.

rclone/fs/config/config.go

Line 333 in 08a2df5

func SetData(newData Storage) {

You don't need to implement all the methods - you can just copy the skeleton here

rclone/fs/config/default_storage.go

Line 3 in 08a2df5

// Default config.Storage which panics with a useful error when used

And implement GetValue and SetValue should be enough unless you are going to call rclone config

Awesome, thank you! I ended up going the "implement Storage" route and that is working beautifully. I ended up implementing the full Storage interface because it was straightforward to do in-memory; I could push it upstream as a PR that may be useful to future users embedding rclone as a library, though the implementation was so straightforward it might not be worth it.

Great!

If you'd like to send the in memory Storage that would be great. Quite possibly that should be the default and it is up to users to switch it out to something else.

fs: Add context to NewFs rclone#3257 rclone#4685

test: Add context to mockfs.NewFs rclone#3257 rclone#4685

fs: Add context to fs.Features.Fill & fs.Features.Mask rclone#3257 rc…

fs: add context.Context to fs.GetModifyWindow rclone#3257 rclone#4685

accounting: add context.Context rclone#3257 rclone#4685

config: add context.Context rclone#3257 rclone#4685

fs: deglobalise the config rclone#4685

fshttp: read config from ctx not passed in ConfigInfo rclone#4685

rc: add context to flag Reload function rclone#4685

filter: deglobalise to put filter config into the context rclone#4685