Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Only create workflow streams on relevant shards while creating a MoveTables workflow for multi-tenant migrations #15748

Open
rohit-nayak-ps opened this issue Apr 18, 2024 · 0 comments

Comments

@rohit-nayak-ps
Copy link
Contributor

rohit-nayak-ps commented Apr 18, 2024

Feature Description

A common case while migrating to a sharded Vitess cluster, from a multi-schema multi-tenant architecture, is that a tenant will be scoped to a single Vitess shard. This is because most tenants will be small and also because sharding a tenant could cause cross-shard queries (and consequently a significant regression in performance).

Currently Vitess allows a shard subset to be specified for all other MoveTables subcommands, other than Create, for optimizing the shard-by-shard migration use case (ref: #9987). It was not needed for shard-by-shard migrations, because the source shards imply what the target shards will be.

However for multi-tenant migration, the target shards will be decided by the target keyspace's VSchema/Vindex. For the scenario above, the current logic will result in workflow streams getting created on all target shards, but only one shard will actually receive data. For a cluster with a large number of shards and multiple concurrent tenant migrations this will be very wasteful and hugely reduce vreplication bandwidth.

It will be ideal if MoveTables is able to allow specifying the target shards on which VReplication should run, or figure it out based on the VSchema.

@rohit-nayak-ps rohit-nayak-ps added Type: Feature Request Needs Triage This issue needs to be correctly labelled and triaged labels Apr 18, 2024
@rohit-nayak-ps rohit-nayak-ps self-assigned this Apr 18, 2024
@rohit-nayak-ps rohit-nayak-ps added Component: VReplication and removed Needs Triage This issue needs to be correctly labelled and triaged labels Apr 18, 2024
@rohit-nayak-ps rohit-nayak-ps changed the title Feature Request: Only create workflow streams on relevent shards while creating a MoveTables workflow for multi-tenant migrations Feature Request: Only create workflow streams on relevant shards while creating a MoveTables workflow for multi-tenant migrations Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant