MaxConcurrentActivityExecutionSize is not honored when default value for TaskListActivitiesPerSecond is used.

[petergardfjall]

Describe the bug
We recently discovered that Cadence does not honor MaxConcurrentActivityExecutionSize when the default value for TaskListActivitiesPerSecond is used. Notably it seemed that whenever we started a workflow, the worker would start about twice the amount of activities that we had configured. Not good, since we are trying to protect a downstream resource from being oversubscribed.

After some debugging I think I found the offending code in internal_worker.go#L1025 and the creation of a "locally dispatched activity worker" (originally introduced in #1029).

There is a workaround in setting TaskListActivitiesPerSecond to anything but the default, since that prevents the creation of the "extra" activity worker. Still this is surprising, incorrect and undocumented behavior.

To Reproduce
Is the issue reproducible?

• Yes

Steps to reproduce the behavior:

• Start one worker with MaxConcurrentActivityExecutionSize: 1 and make sure TaskListActivitiesPerSecond is kept at its default value (100000.0).
  • Let the worker handle a simple workflow with one activity lasting at least a few seconds.
• Have a client start two such workflows in quick succession.
• Watch both workflow activities being concurrently executed despite MaxConcurrentActivityExecutionSize being 1.

Expected behavior
I expected the first activity to start and execute until completion before the second activity was allowed to start (given the worker setting: MaxConcurrentActivityExecutionSize: 1).