Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimise filestore writeFullState for high subject cardinality #5004

Merged
merged 2 commits into from
Jan 26, 2024
Merged

Optimise filestore writeFullState for high subject cardinality #5004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7ed08f8
In `writeFullState`, use `Match` instead of `Iter`, don't allocate 4K…
neilalexander Jan 26, 2024
c86bdcf
Add test to measure time taken on `writeFullState` (skipped by default)
neilalexander Jan 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions server/filestore.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7329,11 +7329,12 @@ func (fs *fileStore) writeFullState() error {
binary.MaxVarintLen64 + 8 + 8 // last index + record checksum + full state checksum

// Do 4k on stack if possible.
var raw [4 * 1024]byte
const ssz = 4 * 1024
var buf []byte

if sz <= cap(raw) {
buf, sz = raw[0:2:cap(raw)], cap(raw)
if sz <= ssz {
var _buf [ssz]byte
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it would make runtime work since you are betting it does not grow stack unless on conditional but it would then need to escape the block properly. Being only 4k I think its fine on main stack, I like the make branch.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can put it back to being unconditional, but I did check it with escape analysis and the escape paths are the same (either via .Seal() or a realloc with append), regardless of whether it's inside the conditional block or not. Just seems like a little way to save the allocation from happening at all if we know from the get-go that we're over 4K.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok if you checked.. Hopefully makes sense though that the runtime seems like it would need to do more work to not decrease stack leaving that block since access to that escapes the block but not the func(). Stack growth is free unless it splits the stack being very big.

buf, sz = _buf[0:2:ssz], ssz
} else {
buf = make([]byte, hdrLen, sz)
}
Expand All @@ -7349,15 +7350,14 @@ func (fs *fileStore) writeFullState() error {
// Do per subject information map if applicable.
buf = binary.AppendUvarint(buf, uint64(numSubjects))
if numSubjects > 0 {
fs.psim.Iter(func(subj []byte, psi *psi) bool {
fs.psim.Match([]byte(fwcs), func(subj []byte, psi *psi) {
buf = binary.AppendUvarint(buf, uint64(len(subj)))
buf = append(buf, subj...)
buf = binary.AppendUvarint(buf, psi.total)
buf = binary.AppendUvarint(buf, uint64(psi.fblk))
if psi.total > 1 {
buf = binary.AppendUvarint(buf, uint64(psi.lblk))
}
return true
})
}

Expand Down
23 changes: 23 additions & 0 deletions server/filestore_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6535,6 +6535,29 @@ func TestFileStoreOptimizeFirstLoadNextMsgWithSequenceZero(t *testing.T) {
require_Equal(t, fs.cacheLoads(), 1)
}

func TestFileStoreWriteFullStateHighSubjectCardinality(t *testing.T) {
t.Skip()

sd := t.TempDir()
fs, err := newFileStore(
FileStoreConfig{StoreDir: sd, BlockSize: 4096},
StreamConfig{Name: "zzz", Subjects: []string{"foo.*"}, Storage: FileStorage})
require_NoError(t, err)
defer fs.Stop()

msg := []byte{1, 2, 3}

for i := 0; i < 1_000_000; i++ {
subj := fmt.Sprintf("subj_%d", i)
_, _, err := fs.StoreMsg(subj, nil, msg)
require_NoError(t, err)
}

start := time.Now()
require_NoError(t, fs.writeFullState())
t.Logf("Took %s to writeFullState", time.Since(start))
}

///////////////////////////////////////////////////////////////////////////
// Benchmarks
///////////////////////////////////////////////////////////////////////////
Expand Down