Fail-over to a second server if the first is unreachable

[nblumhardt]

If the attempt to send events to the primary Seq server fails, the sink could be configured to try a secondary Seq server, or more conveniently, a Seq Forwarder that will persist the events and transmit them back to the primary server once it's available.

For example:

Log.Logger = new LoggerConfiguration()
    .WriteTo.Seq(...
        secondaryServerUrl: "https://some-seq-forwarder",
        secondaryApiKey: "789078907890789")
    .CreateLogger();

Some questions:

• Would this happen immediately, or would it be triggered after some number of failed retries?
• An alternative might be to accept an array of server/API key pairs, however this won't be as friendly to use from app settings. Is more than one secondary server necessary?
• Assuming that 4xx errors would not fail-over - certainly this would make sense for 400, but what about 404, 401?
• Could the sink somehow signal that it was in failed-over mode, e.g. by adding an event or property, or would a prolonged fail-over be detected by other means (perhaps by attaching a property through the secondary forwarder?)

[bbrandt]

I think this is a great idea. In the case of a primary going to Seq and a secondary going to a Seq forwarder, it may be useful to define the benefit. Opposing view may say that, “I have to setup the Seq forwarder on the local host either way, why try to bypass it?” Disclaimer: I am not currently using Seq in any deployed scenarios, so may not be the best one to take an opinion from. My experience is with prior experimentation with POC’s. * I believe a configurable retry count would be useful, but not required. * I think a primary and secondary is best for keeping configuration simple. * Can’t comment. * I think having a property indicating that fail-over has occurred would be valuable. I would add the property wherever it’s cheapest to implement, whether client or forwarder I do not know. I seem to recall lower levels of abstraction possibly being more costly to manipulate the event, but my recollection could be hazy.

[nblumhardt]

Thanks for the thoughts Ben!

For completeness we should probably also consider that adding a generic failoverTo: capability to all Serilog sinks would be better overall for the ecosystem, though considerably more design/implementation/co-ordination work.

Log.Logger = new LoggerConfiguration()
    .WriteTo.Seq(...,
        failoverTo: w => w.Seq(..., ))
    .CreateLogger();

The API is obviously just one possible sketch. Batching makes this pretty tricky, but not impossible. Adding @serilog/reviewers-core in case this sounds worthwhile to explore further.

(It would be nice to start extracting more of the common concerns from the Seq/Splunk/Elasticsearch sinks, which all tend to evolve these capabilities in parallel....)

[tsimbalar]

Just some more thoughts ...

Maybe a more "general" way would be to wrap it all , something like this :

Log.Logger = new LoggerConfiguration()
    .WriteTo.Try(w => w.Seq(primary, ), failoverOptions)
          .ThenTry(w => w.Seq(secondary, ), failoverOptions)
          .FallbackTo(w => w.File(secondary, ))
    .CreateLogger();

The idea being that the w in Try(w => w), ThenTry(w => w) is a special kind of SinkConfiguration that specific sinks could light up when they support fallback ... (i.e. they provide a way to detect that the destination is currently unavailable).

And then there's the question of AuditTo ... should it also support failover and what would that look like ?

[mivano]

Would be nice if this can be generalized to (also) support durable mode. So to support sinks that now have this as part of their code base, but actually have similar code (and issues).

Log.Logger = new LoggerConfiguration()
    .WriteTo.Durable(w => w.Seq(primary, ), durableOptions)
          .ThenTry(w => w.Seq(secondary, ), failoverOptions)
          .FallbackTo(w => w.File(secondary, ))
    .CreateLogger();

The durable option wraps the actual sink and, based on the options, preserves the logevents and tries to push them to the inner sink. The sink, however, needs to indicate that it could not deliver the items. Maybe using a specific exception, although that might be a bit heavy, or by inheriting a special BufferedPeriodicSink?

Any thoughts? The implementation in the ES sink is tricky. Duplication of code, different ways of sending the data etc. Can fix it in the sink itself, but a more common pattern might be nicer?

[mivano]

@nblumhardt also in reference to this issue (serilog/serilog#668) and what I m trying to fix in the durable sink for the ES sink. Although the functionality is great, the implementation differs from the real source. A more generic solution would be nice. Curious about your thoughts.

[nblumhardt]

Another might be introducing Task IBatchedLogEventSink.Emit(LogEvent[] events) somewhere, and having the sinks implement that. Failures could be propagated per batch via the task's ContinueWith(), thus supporting the chaining from one sink to the next.

[mivano]

Interesting idea. As long as the Durable functionality is capable of temporary buffering LogEvents and writing to/reading from file, they can be passed to the next Emit function.

That is also one of the issues of the ES buffer; the input for ES is JSON as that is how it is serialized in the buffer when written to disk, but it misses the context to build the right index name and reuse the real sink logic as it does not pass in the LogEvents again. By providing a durable wrapper you can potentially make all sinks durable without code duplication.

[kevingates]

Any movement on this? Just curious?

[nblumhardt]

Hi @kevingates!

IBatchedLogEventSink now exists in Serilog.Sinks.PeriodicBatching, and I think we're happy enough with the shape of it that it could move to Serilog if needed:

https://github.com/serilog/serilog-sinks-periodicbatching/blob/dev/src/Serilog.Sinks.PeriodicBatching/Sinks/PeriodicBatching/IBatchedLogEventSink.cs

There's no official FallbackSink out there that I know of, but I suspect that one could now be hacked together as a proof-of-concept for sinks that already support IBatchedLogEventSink (like the Seq one, email, and several others) using LoggerSinkConfiguration.Wrap(), which provides a sneaky way to get at the ILogEventSink behind friendly WriteTo configuration methods, and a dose of good-old private reflection to pick up PeriodicBatchingSink's _batchedLogEventSink field:

https://github.com/serilog/serilog-sinks-periodicbatching/blob/dev/src/Serilog.Sinks.PeriodicBatching/Sinks/PeriodicBatching/PeriodicBatchingSink.cs#L45

A PoC that demonstrated the API and mechanics (either as a gist or example nupkg) would help zoom in on any potential issues, illustrate the API possibilities, and might be enough for us to open up some minimal extension points to make this all possible (either via Serilog.Sinks.PeriodicBatching, or with some core APIs).

HTH!