Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the runtime #7054

Merged
merged 15 commits into from
Jun 13, 2024
Merged

Refactor the runtime #7054

merged 15 commits into from
Jun 13, 2024

Conversation

ardatan
Copy link
Owner

@ardatan ardatan commented Jun 3, 2024

This PR contains refactoring, and it doesn't have features for end-user mostly

New Method for Cleaning-up Resources
Transports are the components that help us to execute the queries against the subgraph. Each subgraph has their transport definition on top like @transport(subgraph: "MyRest", location: "http://myrest.com", kind: "rest") or @transport(subgraph: "MyGQL", location: "http://mygql.com/graphql", kind: "graphql-http"). So Mesh takes the kind from this definition for the subgraph, then loads the transport to get the executor for that specific subgraph.

But some transports need a cleanup phase. For example, Neo4J and MySQL transports need to close the connection to the upstream when the gateway is terminated. This PR implements Explicit Resource Management a.k.a. Disposable s for them. So Mesh creates a DisposableStack that collects all the Disposable objects then call their dispose callbacks by respecting their asyncness

Same goes to the plugins... Some plugins need to have a cleanup phase. Let's say we have a tracing plugin that starts a long-running connection to a database, then this needs to be gracefully closed after the gateway is terminated.

Also we have some other components such as Cache storages which might need the same thing such as Redis.
So this PR basically implements Disposable pattern for those components that need to be cleaned up in a certain way.

Previously, Mesh was using PubSub shared across the components, and emitting destroy event when the gateway needs to be terminated. Right not the definition is other way around. Instead of taking the entire PubSub to the components, components provide their own disposal callbacks.


New Abstraction for Unified Graph Execution

GraphQL Mesh currently uses Schema Stitching, and schema stitching has its own way of generating the query planner, and creating the unified schema.
But this will be changed in the future since GraphQL Mesh will use a new query planner that takes the unified schema, and gives the executor instead of providing an executable schema as in Stitching. And also this query planner doesn't need the subgraphs extracted as in Stitching.
So this PR seperates the Stitching specific handling of unified graph. So later on, we can replace the unified graph execution part with any solution. This basically decouples Mesh from Stitching. All of the Stitching logic lives under fusion-runtime package while the actual Mesh layer stays under serve-runtime.


Use ESM instead of CommonJS

By following the direction in JavaScript ecosystem, Mesh CLI packages serve-cli and compose-cli now will use ESM instead of CommonJS with this PR. So it can benefit from the advantages of ESM instead of good-old CommonJS.


Send File Upload Streams directly in REST Transport

Previously REST transport was consuming the given File stream inside the executor then send it to the upstream. But now it pipes the stream directly to the upstream which will improve the performance, and it also simplifies the code.


Simplify the hooks handling code and MaybePromise handling on runtime

Due to the limitations in JavaScript, if a function is able to return a promise or the value directly, it is not good to use async/await because if it is a sync value, this impacts the performance in a bad way. In that case, we need to have the following code for MaybePromises.

const res$ = someFn(...someArgs);
if (isPromise(res$)) {
 return res$.then(res => handleRes(res));
}
return handleRes(res);

And apparently this looks ugly. @EmrysMyrddin recently implemented mapMaybePromise that simplifies this into the following;

return mapMaybePromise(
  someFn(...someArgs),
  res => handleRes(res), // You don't even need a seperate `handleRes` since it is not repeated :) 
);

This PR replaces all extra isPromise usages as in the first version of the code above with mapMaybePromise which simplified the code, and improved the eligibility.


More silent logs during tests

This PR disables logs for most of the unit tests, so unit tests will be more silent and faster(log blocks the thread) .

@ardatan ardatan marked this pull request as draft June 3, 2024 15:24
@ardatan ardatan force-pushed the transforms-v1 branch 2 times, most recently from 281ecac to fb36311 Compare June 3, 2024 15:34
@ardatan ardatan force-pushed the refactor-runtime branch from 444a022 to 61e3392 Compare June 3, 2024 15:35
@ardatan ardatan marked this pull request as ready for review June 4, 2024 09:56
Copy link
Contributor

github-actions bot commented Jun 4, 2024

💻 Website Preview

The latest changes are available as preview in: https://0c31e638.graphql-mesh.pages.dev

Copy link
Contributor

github-actions bot commented Jun 4, 2024

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package Version Info
@graphql-mesh/cache-cfw-kv 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/cache-file 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/cache-localforage 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/cache-redis 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/compose-cli 0.4.0-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/fusion-composition 0.0.6-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/fusion-runtime 0.3.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/apollo-link 0.99.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/cli 0.90.11-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/config 0.100.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/graphql 0.98.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/grpc 0.99.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/json-schema 0.100.12-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/mongoose 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/mysql 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/neo4j 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/odata 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/openapi 0.100.13-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/postgraphile 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/raml 0.100.12-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/soap 0.99.4-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/supergraph 0.4.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/thrift 0.99.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/tuql 0.99.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/http 0.99.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/merger-bare 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/merger-federation 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/merger-stitching 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/runtime 0.99.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/store 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-cache 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-encapsulate 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-extend 0.98.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-federation 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-filter-schema 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-hive 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-hoist-field 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-naming-convention 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-prefix 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-prune 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-rate-limit 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-rename 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-replace-field 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-resolvers-composition 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-transfer-schema 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transform-type-merging 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/types 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/urql-exchange 0.99.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/utils 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/json-schema 0.100.12-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/mysql 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/neo4j 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/openapi 0.100.13-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/raml 0.100.12-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/soap 0.99.4-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/sqlite 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@omnigraph/thrift 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-deduplicate-request 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-hive 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-http-cache 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-http-details-extensions 0.98.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-live-query 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-mock 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-newrelic 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-operation-field-permissions 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-operation-headers 0.1.11-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-prometheus 0.100.6-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-rate-limit 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-response-cache 0.98.9-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-snapshot 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/plugin-statsd 0.98.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/serve-cli 0.5.0-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/serve-runtime 0.3.11-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-common 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-http 0.1.14-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-mysql 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-neo4j 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-rest 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-soap 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-sqlite 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎
@graphql-mesh/transport-thrift 0.2.8-alpha-20240613150227-daf2b86082f4b0c16ee25500999c2cffb2e0d286 npm ↗︎ unpkg ↗︎

@ardatan ardatan marked this pull request as draft June 5, 2024 12:52
@ardatan ardatan marked this pull request as ready for review June 11, 2024 05:21
@ardatan ardatan force-pushed the refactor-runtime branch from d3fb457 to 9b9cf9d Compare June 12, 2024 13:11
Copy link
Collaborator

@enisdenjo enisdenjo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't reviewed in detail, but I approve the general direction.

I especially like the disposable approach and the improved (less data) in the configs.

Base automatically changed from transforms-v1 to master June 13, 2024 14:52
@ardatan ardatan force-pushed the refactor-runtime branch from 494a30f to daf2b86 Compare June 13, 2024 14:56
@ardatan ardatan merged commit 88d6232 into master Jun 13, 2024
21 of 23 checks passed
@ardatan ardatan deleted the refactor-runtime branch June 13, 2024 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants