Experimental Features in Stable Packages

[dyladan]

This issue is to discuss the possibilities for exposing experimental features in stable packages. Examples for experimental features are:

1. New APIs such as logs/events
2. Extensions to existing stable APIs like context/tracing/metrics
3. New SDK features such as file-based configuration
4. New behavior for existing APIs

For a concrete example of an extension to a stable API, see open-telemetry/opentelemetry-js-api#123. This PR adds 2 new methods attach and detach to the context API and context manager interface. Currently, we are required to either add this to the stable API, or release an entirely separate experimental context API and context manager package.

package.json#exports

This option allows a package author to specify different entry points, for example @opentelemetry/api/experimental. It is currently used to export our SugaredTracer. It provides an explicit experimental package import path which signals to users that there may be breaking changes in minor versions. One major advantage of something like this is that it would allow us to also export experimental versions of existing APIs because they are an entirely separate export.

Problems

Complexity

This is just a complex option. For the example given above of adding new context API methods, the context API needs to be re-exported from the ./experimental export in a way that is a drop-in replacement, doesn't affect the existing export, and is obvious to the user. This makes the code a bit cumbersome. For an example, see #4669

Typescript Support

Typescript support is not great. Support was only added in 4.7 (we support 4.4 currently) and requires the user to enable Node16 module resolution. This in turn requires the user to set module: Node16, which emits ESM and not commonjs. This is fine for an experimental package, but if we want to be able to use these experimental exports in our SDK, we would also need to enable these features, which would be breaking for commonjs users.

A workaround for this is to not consume the /experimental namespace in our SDK packages. We would simply redeclare any types. Given the above example, we would have to declare an ExperimentalContextManager type in the AsyncHooksContextManager package and rely on typescript to recognize the compatible types.

@experimental JSDoc Tags

This tag marks an entity as experimental. It is a documentation-only solution, which means it is not enforced by tooling and may not be noticed by some users depending on their tooling. One major advantage of this solution is that it is very simple.

Problems

Overriding Methods

This requires the addition of a new experimental version of any API that needs experimental functionality. As an example, if we were going to experimentally change the behavior of the startSpan API, we would need to add a new method like experimentalStartSpan or similar because we can't override the existing API.

[pichlermarc]

A workaround for this is to not consume the /experimental namespace in our SDK packages. We would simply redeclare any types. Given the above example, we would have to declare an ExperimentalContextManager type in the AsyncHooksContextManager package and rely on typescript to recognize the compatible types.

@dyladan and I talked about this offline yesterday:

We have to re-declare types from the API in the consuming packages anyway if the API is used as a peer dependency like >=1.0.0 <1.9.0. As users can use the package with an older version of the API, it might happen that they don't contain the types API types we're using which breaks users (see #4260).

To implement something like this we can go a similar way as proposed by #4622.

[pichlermarc]

Overriding Methods

This requires the addition of a new experimental version of any API that needs experimental functionality. As an example, if we were going to experimentally change the behavior of the startSpan API, we would need to add a new method like experimentalStartSpan or similar because we can't override the existing API.

I think this may also be solvable by the same wrap()-approach. If wrap() is used we use the new behavior on the returned object in place of the old one.

Would you mind providing an example on which change would result in us having to change the API in such a way?