Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise metalayer snapshots #4925

Merged
merged 3 commits into from
Jan 5, 2024
Merged

Optimise metalayer snapshots #4925

merged 3 commits into from
Jan 5, 2024

Conversation

neilalexander
Copy link
Member

@neilalexander neilalexander commented Jan 4, 2024

This PR optimises the marshalling of metalayer snapshots with a few techniques:

  • Simplifies the custom marshalers for a number of types that appear often in the snapshotted config, as these were using reflection unnecessarily
  • Simplifies the custom unmarshallers for the same types, as these were string-building more than necessary
  • Attempts to reduce the number of reallocations that might happen in metaSnapshot() by preallocating slices

Signed-off-by: Neil Twigg neil@nats.io

Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. I left a comment on whether we should remove client info from snapshots for streams and consumers. We don't need it at that point and probably should not store it.

How are performance results looking?

Config: sa.Config,
Group: sa.Group,
Sync: sa.Sync,
Client: sa.Client,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need client here for snapshots? Both for streams and consumers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do, because there are quite a few places that stream assignments check for the service account and other fields once rehydrated from the snapshot.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct we need account, but not much else I believe.

Signed-off-by: Neil Twigg <neil@nats.io>
@neilalexander neilalexander marked this pull request as ready for review January 5, 2024 14:46
@neilalexander neilalexander requested a review from a team as a code owner January 5, 2024 14:46
@derekcollison
Copy link
Member

Let me know if you want a review now for merge.

@neilalexander
Copy link
Member Author

Think this is fine for review to merge for now, looking into the ClientInfo separately.

server/consumer.go Outdated Show resolved Hide resolved
server/store.go Outdated Show resolved Hide resolved
server/store.go Outdated Show resolved Hide resolved
Signed-off-by: Neil Twigg <neil@nats.io>
Copy link
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@derekcollison derekcollison merged commit fb79814 into main Jan 5, 2024
4 checks passed
@derekcollison derekcollison deleted the neil/goccymeta branch January 5, 2024 18:53
wallyqs added a commit that referenced this pull request Jan 9, 2024
Cherry-pick the following PRs into the 2.10.8 release branch:

* #4918 
* #4923
* #4925
* #4926
* #4927
* #4929 
* #4933
* #4835

Signed-off-by: Neil Twigg <neil@nats.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants