Unify url handling with filesources

[nuwang]

This PR unifies URL handling with filesources. Closes: #14658

Features

• Enables Galaxy to handle urls through a filesource (at present, http(s), base64, ftp and drs are preconfigured, with s3 being admin configurable)
• Enables all url sources to have an injectable user_context. For example, admins can inject credentials for protected DRS servers as in this example

  - type: drs
    id: mydrsserver
    doc: Test drs repository filesource
    http_headers:
      Authorization: |-
        #import base64
        Basic ${base64.b64encode(str.encode(user.preferences['mydrsserver|username'] + ":" + user.preferences['mydrsserver|password'])).decode()}

• Enables admins to configure the specificity of the match. For example, admins can inject different credentials for specific sites, so that passwordless access to those sites are possible: example

  - type: http
    id: test1
    doc: A specific http url handler
    url_regex: "^https?://www.usegalaxy.org/"
    http_headers:
      Authorization: "Bearer ${user.preferences['oidc|bearer_token']}"

• Enables various DRS access methods to be handled by existing filesources (e.g. http and s3 access methods are now routed to filesources: example
• In future, injecting OIDC single-sign-on tokens should be possible by utilizing the work done in: OIDC tokens #15300

Backward compatibility

• No changes to the client are necessary.
• Filesources that inherit from BaseFileSource generally do not require any changes. (*except)
• Except that an additional kwargs parameter has been introduced to some method signatures as a means of future expansion and for supporting chained DRS resolution
• If a plugin directly implements the FileSource interface, it must now implement the score_url_match method to indicate its ability to handle a particular url, get_browsable method to indicate that it supports the SupportsBrowsing interface and the to_relative_path method to convert a gxfile url to a relative path. BaseFileSource handles these cases so that most FIleSources do not need any changes.
• The get_scheme method has been removed from the interface definition as it is no longer essential, but remains in the BaseFileSource.
• The s3.py filesource has been removed, as the s3fs.py filesource does the same thing. The reasons for removing s3.py include its dependencies being less maintained (last updated in 2019) and having less config documentation, in contrast to s3fs.py

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jmchilton]

I was somewhat skeptical but this is very nice - wonderful job!

The **kwargs getting passed everywhere is very typical Galaxy and feels at home in this codebase and is a thing I've done with my own plugin frameworks in the past. I think we should perhaps try to get away from that though by adding a bit more structure. It looks like there is a limited number of known options with fixed types for the plugins we've defined - could we capture all of those in a TypedDict and then use typing_extensions.Unpack to annotate all those kwargs.

• https://peps.python.org/pep-0692/#using-unpack-with-types-other-than-typeddict
• Allow using TypedDict for more precise typing of **kwds python/mypy#13471

It looks like TypedDict can specify they don't capture the totality of options (https://mypy.readthedocs.io/en/stable/more_types.html#totality), which would allow downstream plugins to do their own thing as long as they don't conflict with existing plugins.

If you're uninterested in working through all the details - would you be willing to at least defined the dictionaries of possible options for me and annotate what interfaces accept which ones with a comment or something? I could probably work through the possibilities but you're much closer to the code now. Even if we don't figure out how to use the experimental mypy features - I think that would serve as good documentation for the module.

[jmchilton]

So cool!

[nuwang]

I think we should perhaps try to get away from that though by adding a bit more structure. It looks like there is a limited number of known options with fixed types for the plugins we've defined - could we capture all of those in a TypedDict and then use typing_extensions.Unpack to annotate all those kwargs.

This is a very good point and I fully agree with the runaway nature of a kwargs dict - it's hard to stop it from devolving into an incomprehensible bag of data or know who's depending or modifying which bits. I added it because I didn't feel I had many options - it was either a matter of polluting the interface with weird properties, introducing another object similar to a UserContext, or tacking it onto the UserContext itself. Ultimately, I just chose the easy way out with kwargs. I didn't know about TypedDicts so I'm happy to look into that and give it a go, and am also ears to other approaches to handling this.

[nuwang]

@jmchilton I've made the changes that you've proposed. The TypedDict with Unpack works well, and flagged a number of relevant issues, even though it's still experimental and required a special switch to enable. I eventually switched the kwargs over to a FileSourceOptions object. I thought it made for a cleaner design - too many Unpack/TypedDicts are potentially confusing I think. However, I did use TypedDicts with Unpack for all the constructor arguments, and the whole thing seems to work quite nicely together. Thanks for the suggestion and let me know whether this looks better.

[jmchilton]

Hats off to you @nuwang - this is wonderful!