Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Durable sink should batch based on event size and count #12

Open
jamie94bc opened this issue Feb 24, 2016 · 1 comment
Open

Durable sink should batch based on event size and count #12

jamie94bc opened this issue Feb 24, 2016 · 1 comment

Comments

@jamie94bc
Copy link

We're logging lots of different sized events - some tiny, some huge. We ran into an issue with Seq refusing to accept a large batch of events which occurred because lots of large events were submitted in a single batch.

Rather than reducing the batch count, it would be great to have the the option to also batch by request size. This option would allow many smaller events to be sent at once (say 1000 per batch) and larger events to be sent in smaller batches.

Event with the changes in dev branch to drop large events, it's possible individual events may not exceed _eventBodyLimitBytes.Value but when batched will exceed's Seq's Raw Event Payload Limit.

By batching based on request size, the likelihood of events being dropped would be much lower.

@nblumhardt
Copy link
Member

Thanks for the suggestion, definitely more opportunities to improve here 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants