VReplication: Improve handling of vplayer stalls

[mattlord]

Description

Observations:

Logs:

• vstreamer.go:1022] stream (at source tablet) error @ 056f7cce-9679-11ee-93de-7a43718bd292:1-88,05b63ad0-9679-11ee-8848-b6dd624d298a:1-339542,4e956b44-99ec-11ee-b993-76445c58387d:1-6192085,4e980878-99ec-11ee-9055-cae1ac6f6e27:1-17946122,b912bee5-99ec-11ee-8938-46a387219895:1-10463847: EOF (errno 2013) (sqlstate HY000)
• dbclient.go:125] error in stream 50, will retry after 5s: vttablet: rpc error: code = Unknown desc = stream (at source tablet) error @ 056f7cce-9679-11ee-93de-7a43718bd292:1-88,05b63ad0-9679-11ee-8848-b6dd624d298a:1-339542,4e956b44-99ec-11ee-b993-76445c58387d:1-6196344,4e980878-99ec-11ee-9055-cae1ac6f6e27:1-17946122,b912bee5-99ec-11ee-8938-46a387219895:1-10463847: unexpected EOF
• binlog_connection.go:164] connection closed during binlog stream (possibly intentional): unexpected EOF

MySQL connection:

• The filtered DB user that VReplication uses (vplayer in this case) on the target is executing inserts/updates/deletes.

VReplication / OnlineDDL:

• We're regularly going into the error state and then retrying.
• But we're not making any progress, we're stuck on the same GTID position.
• In the end, we saw that this was because each query we ended up executing on the target (by the vplayer) — from the replicated binlog events — was doing a table scan and it was a large table. This was an OnlineDDL bug resulting from how we handle when the charset + collation is changed on PK columns — but that is beside the point here as it's a general scenario that can occur in VReplication.

Thesis:

• The vplayer is stuck trying to process the last batch read from the relay log. It cannot complete/commit that batch quickly as each table WRITE is doing a scan (each one taking ~ 3-5 seconds in case seen) and thus it cannot complete the batch of contents last read from the relay log in a typically reasonable amount of time.
• This means that the goroutine writing the next batch to the relay log ends up blocked as we hit the configured max relay log items / bytes on the target.
• This in turn means that we eventually cannot read more events from the open com_binlog_dump_gtid connection on the source tablet as that's what is then sent to the vplayer on the target tablet (via the VStream RPC call that the target made to the source) to write to the relay log.
• When in this state we're not sending MySQL replication protocol heartbeats so mysqld ends up closing the binlog dump connection when no more IO is done by the time we hit slave_net_timeout. When this error ends the VStream RPC from the target to the source the workflow goes into the error state (to be retried 5 seconds later by default).
• So we are stuck in this loop indefinitely and not making any progress.

This is a valid scenario that can be encountered. The real problem here is that it was very much NOT obvious what was happening and where the real problem lied. Ideally we should provide information that allows the user to zero in on the actual problem as described above: the queries being executed via the replication stream are doing table scans and we'll need to address that somehow: new indexes, workflow adjustments (in the seen case we needed to adjust the workflow binlog source filter that OnlineDDL created), source and/or target tablet config adjustments (increasing slave_net_timeout on the source e.g.), etc.

Proposed improvements:

• ???

Related Issue(s)

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

Attention: Patch coverage is 69.23077% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 68.45%. Comparing base (1de3daa) to head (bee30d4).

Files	Patch %	Lines
...vt/vttablet/tabletmanager/vreplication/relaylog.go	66.66%	2 Missing ⚠️
.../vt/vttablet/tabletmanager/vreplication/vplayer.go	71.42%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #15797   +/-   ##
=======================================
  Coverage   68.45%   68.45%           
=======================================
  Files        1558     1558           
  Lines      195928   195940   +12     
=======================================
+ Hits       134128   134140   +12     
  Misses      61800    61800           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.