Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use size-based batch limiting #172

Open
nblumhardt opened this issue Mar 7, 2022 · 0 comments
Open

Use size-based batch limiting #172

nblumhardt opened this issue Mar 7, 2022 · 0 comments

Comments

@nblumhardt
Copy link
Member

On the server-side, Seq imposes two size limits on incoming events:

  • The event size limit, which applies to the JSON-encoded byte size of individual events, and
  • The batch size limit, which applies to the total size (including newlines) of the CLEF-encoded POST body.

On the client side, this sink currently uses an event size limit, but batches are limited by count (batchPostingLimit) rather than byte size. This makes it difficult to reliably align the client and server settings, leading to potential batch loss when oversize batches are discarded, or poor batch utilization if the batch size has to be set to an excessively conservative low value.

A future version of this sink should replace batchPostingLimit with a size-based control.

The main caveat that will complicate this is the dependency on PeriodicBatchingSink, which implements the current count-based logic. The dependency on PeriodicBatchingSink will likely need to be broken and the functionality replaced in order to make the proposed change here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant