Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

channelz: ensure SubChannel is always registered with parent #7094

Conversation

daniel-weisse
Copy link

@daniel-weisse daniel-weisse commented Apr 4, 2024

channelz.SubChannel may be missing parent information if created using channelz.RegisterSubChannel.
This is because channelz.RegisterSubChannel does not take a channelz.Channel as argument, but just the ID of this channel.
The function itself then tries to load the Channel from a package scoped channelMap.
This map does not contain any entries if channelz data collection is turned off, resulting in the SubChannel being created without a parent.
Which in turn results in the SubChannel's string representation being logged as <nil> SubChannel #1234 (1234 being some ID of the SubChannel).

This PR fixes this by restoring the behavior of v1.62.2, which instead of passing just the ID of a SubChannel's parent, passed a reference to the parent Channel, returning an error if it is nil:

func RegisterSubChannel(c Channel, pid *Identifier, ref string) (*Identifier, error) {
if pid == nil {
return nil, errors.New("a SubChannel's parent id cannot be nil")
}
id := IDGen.genID()
if !IsOn() {
return newIdentifer(RefSubChannel, id, pid), nil
}

RELEASE NOTES: none

Copy link

codecov bot commented Apr 4, 2024

Codecov Report

Merging #7094 (3694b2e) into master (6fbcd8a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 3694b2e differs from pull request most recent head 5f17e6e. Consider uploading reports for the commit 5f17e6e to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #7094   +/-   ##
=======================================
  Coverage   81.07%   81.08%           
=======================================
  Files         345      345           
  Lines       33927    33928    +1     
=======================================
+ Hits        27508    27512    +4     
+ Misses       5241     5235    -6     
- Partials     1178     1181    +3     
Files Coverage Δ
internal/channelz/subchannel.go 94.73% <100.00%> (+0.29%) ⬆️

... and 25 files with indirect coverage changes

@dfawley
Copy link
Member

dfawley commented Apr 4, 2024

@daniel-weisse do you know if this is a regression with the latest release? If so we may prefer to restore the previous behavior instead.

@daniel-weisse
Copy link
Author

daniel-weisse commented Apr 5, 2024

The Channel and SubChannel types were introduced in v1.63.
v1.62 used the Identifier struct, which served the same purpose.

From a brief look at the v1.62.2 code, it looks like parent information was always omitted from its string representation if there was no parent:

func newIdentifer(typ RefChannelType, id int64, pid *Identifier) *Identifier {
str := fmt.Sprintf("%s #%d", typ, id)
if pid != nil {
str = fmt.Sprintf("%s %s", pid, str)
}
return &Identifier{typ: typ, id: id, str: str, pid: pid}
}

func (id *Identifier) String() string {
return id.str
}

Edit: Took another look at the code.
I think the actual regression is in how the SubChannel resource is created in v1.63.
In v1.62.2, RegisterSubChannel throws an error if no parent is provided. It then creates the (SubChannel) Identifier by passing the parent ID, even if channelz is not turned on:

func RegisterListenSocket(s Socket, pid *Identifier, ref string) (*Identifier, error) {
if pid == nil {
return nil, errors.New("a ListenSocket's parent id cannot be 0")
}
id := IDGen.genID()
if !IsOn() {
return newIdentifer(RefListenSocket, id, pid), nil
}

In v1.63 however, the RegisterSubChannel performs no checks for the channels parent since the parent is no longer passed as an argument, but held in a packaged scoped channelMap.
When the SubChannel is created with channelz being turned off, no parent information is passed to the subchannel:

func RegisterSubChannel(pid int64, ref string) *SubChannel {
id := IDGen.genID()
if !IsOn() {
return &SubChannel{ID: id}
}

I think the correct way would be passing the parent information here, like it is done later on in the function.
This doesn't work however, since the parent Channel was never added to the channelMap db in the first place, because channelz is turned off.

One fix would be adjusting the signature of the RegisterSubChannel function to take the SubChannel's parent *Channel as an argument, instead of just its pid.

So we would go from this:

func RegisterSubChannel(pid int64, ref string) *SubChannel {

to this:

func RegisterSubChannel(parent *Channel, ref string) *SubChannel {

The function then no longer needs to load the parent from the channel map

@dfawley
Copy link
Member

dfawley commented Apr 5, 2024

One fix would be adjusting the signature of the RegisterSubChannel function to take the SubChannel's parent *Channel as an argument, instead of just its pid.

This sounds good. Subchannels should always have parents, so I was skeptical about the initial change proposal -- we should ideally be logging both the channel+subchannel IDs in all the subchannel events.

@daniel-weisse daniel-weisse changed the title channelz: omit SubChannel.parent in string representation if parent is nil channelz: ensure SubChannel is always registered with parent Apr 8, 2024
@daniel-weisse daniel-weisse force-pushed the channelz-subchannel-string-check branch 2 times, most recently from 45e7acf to c328773 Compare April 8, 2024 07:34
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
@daniel-weisse daniel-weisse force-pushed the channelz-subchannel-string-check branch from c328773 to 5f17e6e Compare April 8, 2024 07:36
@daniel-weisse
Copy link
Author

Updated the PR to restore the RegisterSubChannel function to the behavior of v1.62.2.
This should ensure SubChannels are always created with a valid parent, so this information is never logged as nil

Comment on lines +148 to +150
if pid == nil {
return nil, errors.New("a SubChannel's parent id cannot be nil")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not return an error here since it complicates the code so much, and it should just be an invariant of the code like any other dereference. Just panic and we'll make sure RegisterSubChannel is always called properly.

Actually, this change is pretty important to get done quickly, so I created #7101 and we will be doing a patch release with it ASAP. Thank you for the fix!

@dfawley
Copy link
Member

dfawley commented Apr 8, 2024

As mentioned in the review comment, we'll be continuing this in #7101. Sorry for that, but it needs to get done very quickly. Thank you!

@dfawley dfawley closed this Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants