Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature(streaming): support streaming logs with websocket #444

Open
EandrewJones opened this issue May 7, 2024 · 0 comments
Open

Feature(streaming): support streaming logs with websocket #444

EandrewJones opened this issue May 7, 2024 · 0 comments

Comments

@EandrewJones
Copy link
Contributor

Problem

Currently, UserALE bundles logs into a buffer and flushes the buffer on a regular interval or in response to certain event triggers before POSTing them to the REST API url. This works well in most circumstances, but adds unnecessary network overhead. Also if requests are dropped we have no back up mechanism.

Solution

Add support for websocket connections if the backend REST API supports them. Upon first connection to the back-end, we would check to see if web sockets are supported, if so, we would upgrade the connection.

This achieves two things:

  • Minimizes network overhead
  • Adds redundancy

We would still store logs in a client-side memory buffer and flush them over the socket. However, the socket reduces network overhead. And if the connection ever gets dropped before the buffer has been flushed, then we can concurrently work to re-establish the connect while continuing to store event logs in the buffer. Upon reconnect, you flush all logs to relieve back-pressue. In this way, no logs are ever dropped. As an escape hatch, if the buffer exceeds a certain memory threshold, then you can treat it like a FIFO queue and start dumping old logs to avoid a memory leak.

@EandrewJones EandrewJones changed the title Features(streaming): support streaming logs with webscoket Feature(streaming): support streaming logs with webscoket May 7, 2024
@EandrewJones EandrewJones changed the title Feature(streaming): support streaming logs with webscoket Feature(streaming): support streaming logs with websocket May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant