Better distribution of users and fix distributed hatch rate

Fixes #1618
Fixes #896
Fixes #1635
There's quite a lot of changes in this PR, but it was required to solve these two issues.
Changes

New logic for computing the distribution of users (locust/distribution.py).
New logic for dispatching the users (locust/dispatch.py). The master is now responsible to compute the distribution of users and then dispatching a portion of this distribution to each worker. The spawn rate is also respected in distributed mode.
Possible breaking change: signature of spawning_complete event now takes user_class_occurrences instead of user_count.
Behaviour change: users are no longer stopped at a rate of spawn_rate. See the docstring of locust.dispatch.dispatch_users for the rationale behind this decision. The correct way to have a ramp down is to use the load test shape feature.

Demo
Example of a real distributed load test shape with 5 workers and a stop timeout of 30s:

Load test shape profile for this run:

duration
users
spawn rate

2m
5
1

2m
10
1

2m
15
1

2m
20
1

5m
25
1

2m
20
1

2m
15
1

2m
10
1

2m
1
1

5m
25
5

2m
1
1

5m
25
5

2m
1
1

The ramp-up of each stage takes around 4-5s which correlates with the spawn rate of 1. Also, we can see that the users are gracefully stopped (or killed after the stop timeout of 30s). That's why the ramp-down have the small staircases instead of one single drop.
Other
I hope I did not break anything. All the tests are passing, but while running some real load tests (such as the one above), it seems like the users are not always properly stopped. I added an integration test in test_runners.TestMasterWorkerRunners.test_distributed_shape_with_stop_timeout and I can't seem to reproduce the issue, so I don't think  this problem is caused by my changes. Let me know if you think it might be because of my changes. Otherwise, I will open a new issue with some idea I have that might help with this.

One of the tests is flaky, I'll look into it tomorrow

Phew. That's a lot of changes and a lot of new code. Is there any way you could simplify it a little bit? I dont feel I can review it in its current state.
Does it really need to be recursive?
Is there really any benefit to using the "postfix" style for-loops? Imho, they make things harder to read.
I dont like the changes to input handling. Having two ways of handling input makes no sense. Try to stick as closely as possible to the current solution to avoid changing something that really isnt related to the distribution of users.

I reworked the distribution code. It is now simpler (in my opinion) and no longer uses the recursive pattern.
I also restored the original input_listener and only modified the anonymous functions given in the key_to_func dictionary.

I need to perform more testing on my side. When I run some real scenarios with the web ui, the chart for the spawned users shows weird patterns (e.g. users decreasing before going up again).

build is still failing (intermittently?)

To be frank, I'm still a little skeptical to such a big change and I dont think I'll have time to review it any time soon. Maybe @heyman has some input?

build is still failing (intermittently?)

The new integration test TestMasterWorkerRunners.test_distributed_shape_with_stop_timeout is still flaky. I'll make it robust so that it passes every time. It's a bit tricky with the timing and async nature of the code.

To be frank, I'm still a little skeptical to such a big change and I dont think I'll have time to review it any time soon.

I don't know what to tell you, the previous implementation had flaws when running in distributed mode as well as with the load test shape, the latter misbehaving in both modes. The problem are well explained in the two issues this PR is trying to solve.
The two years old Issue #896 is labelled as low prio, but I personally think this is a bug. Say you have 100 workers and want to hatch 1000 users at a rate of 10, with the previous implementation, you'd have an effective rate of 1000/s ten times in a row which does not make any sense and can screw up your load test. Now, imagine with even more workers...it's clear the implementation was not scalable and flawed.
In my opinion and after a considerable amount of thinking and studying the runners codebase, the most reliable, and probably most precise way to handle this is to have the master manage the state of users and their dispatch as this PR does.
Regarding the alternative of having the master send the user count along with a delay so that the worker waits before spawning; I don't think this would scale nor work very well because of the unpredictable nature of the network having variable latencies and response times. Furthermore, each worker might take a unpredictable amount of time to process each received spawn messages (because it is busy doing another task or else). Given these aspects, it would be very difficult for the master to compute this delay accurately.
I think a good place to start is by looking at the tests in test_dispatch.py and test_distribution.py so it is easier to understand the new behaviours.
I think the most challenging changes to make sense of are in the runners module. So, I'll try to summarize the changes here:

Before, Runner.spawn_users() and Runner.stop_users() would take a relative amount of users to spawn and stop. These two functions were responsible to compute the number of users in each class to start/stop (using Runner.weight_users()). Now, Runner.spawn_users() and Runner.stop_users() are given a dictionary with exactly the number of each user class to start or stop.
Before, the Runner.start() function would simply check if stopping or spawning users is necessary and then call either Runner.spawn_users() or Runner.stop_users(). Now, Runner.start() has become the "brain" and it is responsible to compute the user distribution and dispatch (with a delay in between) the users to start and stop. With the new code, a single call to Runner.start() will start and stop users depending on the user class. This explains the added complexity needed to handle starting and stopping users at the same time.
Before, the worker runner was directly using Runner.start(). However, this is no longer appropriate since the worker is now "brainless"  because it receives the pre-computed occurrences for each user class without any spawn rate. Its job is simply to altered its state to match the desired state as quickly as possible. For that matter, a new WorkerRunner.start_worker() has been added to clearly differentiate with the Runner.start() method which is never used by the worker class.
The Runner.weight() has been replaced by the weight() function in the distribution.py module. I chose to factor out this function in its own module in order to not add more code in the runners.py module which is already quite large. I also find it's easier to test a standalone weight() function because it does not require to setup the Environment and WorkerRunner objects which are unnecessary to test the behaviour.

Another note is that Runner.spawn_users() and Runner.stop_users() are not intended to be used outside the runner classes. The proper way to control the number of running users and spawn rate is to use Runner.start() since it will take into account the current state of the running users and issue the proper calls to Runner.spawn_users() and Runner.stop_users(). That is why the input_listener needed to be updated to no longer use Runner.spawn_users() and Runner.stop_users().

I agree that this is an area that needs fixes, it is just that I dont have time to review it. I have spent way too much time on the project in the last couple of months anyway :)
Low prio is just my way to mark tickets as non-critical (and while this is definitely something that we want to fix, it is not critical)
If @heyman, or maybe even someone of the other main contributors (doesnt have to be someone with merge permissions) does a first sweep of feedback I can give it a last look and help with the actual clicking of the merge-button :)
Maybe @max-rocket-internet is interested in this kind of feature?

Maybe @max-rocket-internet is interested in this kind of feature?

Not really. As I understand this PR is about the distribution of user classes, not locust users, right? In all our load tests I think only 1 or 2 a using more than a single HttpUser. We just use task decorator to split functionality by ratio of requests.

The correct way to have a ramp down is to use the load test shape feature.

By this logic you could say the same about ramping up.

Not really.

I see that you showed some interests in solving the distributed hatch rate issue https://github.com/locustio/locust/issues/896 which is addressed by this PR.

As I understand this PR is about the distribution of user classes, not locust users, right?

It also addresses the issues of fair dispatch of the users in distributed mode.

We just use task decorator to split functionality by ratio of requests.

I think this works well for most of the use cases. Especially when each task is more or less "equal" in terms of runtime. However, I found that when dealing with unbalanced tasks, i.e. some tasks take only few seconds or less whereas others can take several minutes, this way of doing was not working very well. The problem is that, eventually, a large number of users get "stuck" on these long-running tasks even if they have a lower weight compared to the short-running tasks. By separating in different user classes, it is possible to get much more control over how frequent each task is executed.

By this logic you could say the same about ramping up.

Absolutely, and I think people should do so. As I see it, the spawn rate is mainly there as a convenience. For instance, it's useful when one only wishes to ramp up linearly to a certain steady-state number of users.

I fixed the test that was flaky.

I'm planning to take a look at this tomorrow morning after I've had some sleep.

Not sure when/if we can expect @heyman to get back on this. I would like to suggest a simpler approach using a deck of cards algorithm:
At startup, locust generates a deck of cards (Users), based on their weights, placed in a deterministic order, with each new card minimizing the "distance" to the ideal distribution, and the deck size large enough to give it the actual ideal distribution.
For example, if there is a User1 with weight 2 and a User2 with weight 3 the deck would be something like [User2, User1, User2, User1, User2, User2].
When locust starts ramping up (either because it entered a new phase in a LoadShape or at start), it extends the deck if needed (using the same distribution as the first time it was populated), and then starts launching users in order.
In order for distribution of users to be even when running distributed, we can use the ID of the worker (once https://github.com/locustio/locust/issues/1601 is implemented) to offset where to start.
Two things I haven't detailed here are ramp down and how to print ramp up rates, but I think that can be solved.

Not sure when/if we can expect @heyman to get back on this. I would like to suggest a simpler approach using a deck of cards algorithm:
At startup, locust generates a deck of cards (Users), based on their weights, placed in a deterministic order, with each new card minimizing the "distance" to the ideal distribution, and the deck size large enough to give it the actual ideal distribution.
For example, if there is a User1 with weight 2 and a User2 with weight 3 the deck would be something like [User2, User1, User2, User1, User2, User2].
When locust starts ramping up (either because it entered a new phase in a LoadShape or at start), it extends the deck if needed (using the same distribution as the first time it was populated), and then starts launching users in order.
In order for distribution of users to be even when running distributed, we can use the ID of the worker (once https://github.com/locustio/locust/issues/1601 is implemented) to offset where to start.
Two things I haven't detailed here are ramp down and how to print ramp up rates, but I think that can be solved.

Actually, when I think about it: Another option might be to just ignore this problem alltogether (which most other tools do) and set the number of users of each User type directly, rather than controlling the total number of users. But that would mean a bigger change to locust.

Actually, when I think about it: Another option might be to just ignore this problem alltogether (which most other tools do) and set the number of users of each User type directly, rather than controlling the total number of users. But that would mean a bigger change to locust.

I'm sorry in advance for my bluntness, but I find it very sad that my proposed solution which I thoroughly tested and used in my load tests is being discarded like that without proper attention given to it. I have the impression that all the justifications and explanations I provided in my previous comments were not really read. Instead, other solutions that do not even have a working PoC are being proposed whereas mine is implemented and working solving some long stating issues with locust.
If someone else can come up with a working solution to all the problems this PR is trying to solve, I'll be really happy. Until then, locust is truly unusable in the distributed mode with multiple users.
I get that I made quite a lot of changes and if that's what bothering the maintainers, I can look into reverting some of the changes that were not directly related to the PR. However, I doubt that this will really help because they are a small portion of the overall changes.
I'm sorry for my tone and my goal is not to attack anyone. I'm just sad and surprised that this working solution is just being discarded. I'm open to criticism and suggestions, but suggesting a completely different solution seems like reaching quite a lot. Every suggestions that were made in November were incorporated.
My final question is: what's wrong with this current PR? What can I do to make it easier for the maintainers and reviewers? Would a zoom/team/etc call help clarifying some technical questions?

I'm sorry in advance for my bluntness, but I find it very sad that my proposed solution which I thoroughly tested and used in my load tests is being discarded like that without proper attention given to it. I have the impression that all the justifications and explanations I provided in my previous comments were not really read. Instead, other solutions that do not even have a working PoC are being proposed whereas mine is implemented and working solving some long stating issues with locust.
If someone else can come up with a working solution to all the problems this PR is trying to solve, I'll be really happy. Until then, locust is truly unusable in the distributed mode with multiple users.
I get that I made quite a lot of changes and if that's what bothering the maintainers, I can look into reverting some of the changes that were not directly related to the PR. However, I doubt that this will really help because they are a small portion of the overall changes.
I'm sorry for my tone and my goal is not to attack anyone. I'm just sad and surprised that this working solution is just being discarded. I'm open to criticism and suggestions, but suggesting a completely different solution seems like reaching quite a lot. Every suggestions that were made in November were incorporated.
My final question is: what's wrong with this current PR? What can I do to make it easier for the maintainers and reviewers? Would a zoom/team/etc call help clarifying some technical questions?

Hi! I totally understand your "bluntness", and I apologize for the delay.
I feel even your final solution is just too big a change at this point. While the tests are working and I understand that it works well for you, there is just too many ways this could break in some edge case, at scale or when workers appear/disappear.
Parts of your solution are really good (the calculation of distance between ideal and current distribution for instance), but moving control from worker to master is a very large change and risks making Locust more fragile (this is my main issue with this PR)
I get that you really want this improvement, but we have to be restrictive in accepting large changes, because as core contributors, we are on the hook for maintaining everything.
At the moment, adding/removing workers during a run skews the user distribution in a way that I would definitely call a bug (especially if you add users one at time, using keyboard input), but for most cases the current behaviour is not massively broken and is possible to work around.
I'm sorry if my attempts to give feedback early on led you to invest more time in what ultimately does not (in my opinion) fit well with the project. I did express my doubts a couple of times, but maybe I should have been even more clear.
A smaller PR along the lines of what I suggested in my last comment would be very welcome, but I totally understand if you dont want to put in even more effort...

Hi! I totally understand your "bluntness", and I apologize for the delay.
I feel even your final solution is just too big a change at this point. While the tests are working and I understand that it works well for you, there is just too many ways this could break in some edge case, at scale or when workers appear/disappear.
Parts of your solution are really good (the calculation of distance between ideal and current distribution for instance), but moving control from worker to master is a very large change and risks making Locust more fragile (this is my main issue with this PR)
I get that you really want this improvement, but we have to be restrictive in accepting large changes, because as core contributors, we are on the hook for maintaining everything.
At the moment, adding/removing workers during a run skews the user distribution in a way that I would definitely call a bug (especially if you add users one at time, using keyboard input), but for most cases the current behaviour is not massively broken and is possible to work around.
I'm sorry if my attempts to give feedback early on led you to invest more time in what ultimately does not (in my opinion) fit well with the project. I did express my doubts a couple of times, but maybe I should have been even more clear.
A smaller PR along the lines of what I suggested in my last comment would be very welcome, but I totally understand if you dont want to put in even more effort...

Sorry for not providing more feedback on this yet. I'm still planning to do so, but have been swamped with work since before christmas.

Sorry for not providing more feedback on this yet. I'm still planning to do so, but have been swamped with work since before christmas.

because as core contributors, we are on the hook for maintaining everything.

I understand.

Sorry for not providing more feedback on this yet. I'm still planning to do so, but have been swamped with work since before christmas.

I also understand.

that it works well for you

That's why I hoped some other power users of locust could test the branch's code with their use cases.

at scale or when workers appear/disappear.

Those two cases are technically covered.

However, I don't really see how these problems can be reliably solved by not having the master have authority on the distribution and dispatch of users. How can the global state and behaviour of a distributed load test can be reliably controlled if no single node takes decisions? The fundamental problem IMO is precisely the fact that the master has currently no authority on the scheduling (scheduling = distribution + dispatch) of the users in distributed mode.
As I see it, locust should follow the same architecture and principles as Kubernetes: you have the control plane which decides the desired state of the cluster (this is the master in locust). The control plane sends instructions to each kubelet (the workers in locust) which is then responsible to make sure its stuff running on its node matches the instructions sent by the control plane. In turns, each kubelet reports its state to the control plane and the control loop goes on. Given the success and the scale at which Kubernetes can operate, I'd say that this architecture works well.
Locust works very well in these cases:

Standalone mode
Distributed mode with few user classes having similar weights and not too small spawn rate. Note that the current code never respects the spawn rate.

I guess that's why most of the use cases are well-behaved with the current code because they fit in the above list.

because as core contributors, we are on the hook for maintaining everything.

I understand.

Sorry for not providing more feedback on this yet. I'm still planning to do so, but have been swamped with work since before christmas.

I also understand.

that it works well for you

That's why I hoped some other power users of locust could test the branch's code with their use cases.

at scale or when workers appear/disappear.

Those two cases are technically covered.

However, I don't really see how these problems can be reliably solved by not having the master have authority on the distribution and dispatch of users. How can the global state and behaviour of a distributed load test can be reliably controlled if no single node takes decisions? The fundamental problem IMO is precisely the fact that the master has currently no authority on the scheduling (scheduling = distribution + dispatch) of the users in distributed mode.
As I see it, locust should follow the same architecture and principles as Kubernetes: you have the control plane which decides the desired state of the cluster (this is the master in locust). The control plane sends instructions to each kubelet (the workers in locust) which is then responsible to make sure its stuff running on its node matches the instructions sent by the control plane. In turns, each kubelet reports its state to the control plane and the control loop goes on. Given the success and the scale at which Kubernetes can operate, I'd say that this architecture works well.
Locust works very well in these cases:

Standalone mode
Distributed mode with few user classes having similar weights and not too small spawn rate. Note that the current code never respects the spawn rate.

I guess that's why most of the use cases are well-behaved with the current code because they fit in the above list.

@heyman Any chance you'll have time to look at this?
There seems to be some breaking changes though. When registering this event https://docs.locust.io/en/stable/api.html#locust.event.Events.spawning_complete I get this error (havent really dug into it)
[2021-02-10 20:15:00,060] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-02-10 20:15:00,061] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 1 still running
[2021-02-10 20:15:00,061] lafp-mac-JG5J.int.svenskaspel.se/DEBUG/locust.runners: Updating state to 'running', old state was 'spawning'
[2021-02-10 20:15:00,061] lafp-mac-JG5J.int.svenskaspel.se/ERROR/root: Uncaught exception in event handler:
Traceback (most recent call last):
  File "/Users/lafp/git/locust/locust/event.py", line 40, in fire
    handler(**kwargs)
TypeError: spawning_complete() missing 1 required positional argument: 'user_count'

@cyberw, is this event covered in the test suite? I'll look into it

@cyberw, is this event covered in the test suite? I'll look into it

I'm guessing it is not covered. It happened when I ran locust-plugins TimescaleListener.

Ok, I know the cause. In the PR description, I mentioned:

Possible breaking change: signature of spawning_complete event now takes user_class_occurrences instead of user_count.

It's in fact a breaking change and not just a possible one. I can find a way to not change the function's signature or have some kind a logic to convert user_count to user_class_occurrences.

Ah, that explains it. I wouldnt spend too much time on it right now. Lets wait for heyman to give his view.

@cyberw, I reverted the previous signature for the spawning_complete event. You can retry with my branch (git+https://github.com/mboutet/locust@94094a8adbb95c3fc827d5f3cd6d2a2620bdcb43) if you want.

@heyman  / @cyberw  - we are suffering from this issue during our executions and one of my guys pointed me to this issue.
I would agree with @mboutet statement that lack of control coming from master is the thing that should be addressed. There is no way to dynamically add workers or starting a small number of VU with a big number of workers and different weights.
What was the way in the past to introduce breaking changes? Were there any volunteers that were using "beta" or anything like that?
Is there any way to ask people on Slack for "testing it" in live situation?

hi @domik82 ! Right now the issue is that I dont like the change (not totally against it, but it is kind of a big change, and I dont have any more time to spend on it).
I would welcome a smaller PR, along the lines of what I suggested.
@heyman is another option, if you manage to convince him, then I'm ok with it :)
Of course, there is nothing stopping you from running this branch and trying it out. If you do and find no issues then that would increase the chance of convincing me it is a good change.

Of course, there is nothing stopping you from running this branch and trying it out. If you do and find no issues then that would increase the chance of convincing me it is a good change.

We will try to test it on our code base - have you ever thought about official BETA releases? It would be way easier to validate if instead of building the code on our own we just could pull in a named BETA and then provide comments.
Currently, for security reasons, our Linux boxes don't have connectivity to GITHub so we need to do workarounds to be able to deploy such built solution.

What was the way in the past to introduce breaking changes?

@domik82, as far as I know, I did not introduced any voluntary breaking changes (the only discovered one I introduced, I fixed it the other day). So unless some users are relying on internal logic, I don't think this will cause any problem. If breaking changes are discovered, I'll surely look into them.  I'm also regularly rebasing on top of the master in order to validate against the latest locust.
Also, I'm glad that you agree with the fact that the master should have full authority. As you said, I believe this would open the door to great features as well as fix the long standing issues.

I would welcome a smaller PR, along the lines of what I suggested.

@cyberw, as I said many times, it won't work. Or at least, I fail to see how it would fix the issues. I suggest you implement your idea (or a PoC at least) as I did for mine. If your solution fix the issues, I'll gladly throw my PR in the garbage. I only wish for Locust to work as intended in more complex scenarios.
Believe me, I would have liked this PR to be tiny. Unfortunately, the distributed logic was flawed and needed a complete rewrite. Sometimes, code needs to be refactored and rewrote in order to move forward.
I tested my branch multiple times in distributed mode for the last few months and all the issues are gone.

Ok, since you have put this much work into it, it deserves some effort from my part. I tested it out a little (running distributed with 4 workers) but immediately ran into a couple of issues.

Interrupting master during ramp up seems to consistently give me this (that should just stop the test in an orderly fashion and display the final report)

[2021-03-02 22:23:40,393] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Sending spawn jobs of 10 users and 0.25 spawn rate to 4 ready clients
^CKeyboardInterrupt
2021-03-02T21:23:49Z
Traceback (most recent call last):
  File "/usr/local/bin/locust", line 33, in <module>
    sys.exit(load_entry_point('locust', 'console_scripts', 'locust')())
  File "/Users/lafp/git/locust/locust/main.py", line 331, in main
    runner.start(options.num_users, options.spawn_rate)
  File "/Users/lafp/git/locust/locust/runners.py", line 598, in start
    for dispatched_users in users_dispatcher:
  File "/Users/lafp/git/locust/locust/dispatch.py", line 116, in dispatch_users
    gevent.sleep(max(0.0, wait_between_dispatch - delta))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gevent/hub.py", line 166, in sleep
    hub.wait(t)
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
KeyboardInterrupt

Distribution is incorrect if Users dont have an explicit weight. I had User1 with weight no weight specified and User2 with weight 7. Starting 8 users consistently gave me 5 User1 and 3 User2 (!)

Logging is a little too verbose as well, and maybe it could be made to look nicer (no braces with unclear meaning)
[2021-03-02 22:59:37,661] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser1": 1} ({"DemoUser1": 1, "DemoUser2": 0} already running)...
[2021-03-02 22:59:37,662] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser1 spawned
[2021-03-02 22:59:37,662] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 2 still running
[2021-03-02 22:59:38,662] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser1": 1} ({"DemoUser1": 2, "DemoUser2": 0} already running)...
[2021-03-02 22:59:38,663] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser1 spawned
[2021-03-02 22:59:38,663] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 3 still running
[2021-03-02 22:59:39,664] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser1": 1} ({"DemoUser1": 3, "DemoUser2": 0} already running)...
[2021-03-02 22:59:39,664] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser1 spawned
[2021-03-02 22:59:39,664] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 4 still running
[2021-03-02 22:59:40,665] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser1": 1} ({"DemoUser1": 4, "DemoUser2": 0} already running)...
[2021-03-02 22:59:40,666] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser1 spawned
[2021-03-02 22:59:40,666] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 5 still running
[2021-03-02 22:59:41,665] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser1": 5, "DemoUser2": 0} already running)...
[2021-03-02 22:59:41,666] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-02 22:59:41,666] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 6 still running
[2021-03-02 22:59:42,668] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser1": 5, "DemoUser2": 1} already running)...
[2021-03-02 22:59:42,668] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-02 22:59:42,668] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 7 still running
[2021-03-02 22:59:43,669] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser1": 5, "DemoUser2": 2} already running)...
[2021-03-02 22:59:43,670] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-02 22:59:43,670] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 8 still running

If these issues get fixed and you and other people run it for a while, then I can have another look.

I tried running it in scenario with 7 user classes, 10 expected users and 5 hosts, but all the created users were belonging to the same user class

@cyberw I will look into it and get back to you.
@stanislawskwark, that's surprising. Normally, you should have got at least 1 user for each class with the 3 remaining users distributed amongst the user class with the highest weight. Can you share more information so I can debug?

@mboutet from what I observed this happens with small amount of users on multiple hosts
I tried to run following locustfile on 5 hosts and 5 users.
`from locust import HttpUser, task
host = "http://github.com"
class UserA(HttpUser):
host = host
@task
def x(self):
    self.client.request("GET", "/", name="user A")

class UserB(HttpUser):
host = host
@task
def x(self):
    self.client.request("GET", "/", name="user B")

class UserC(HttpUser):
host = host
@task
def x(self):
    self.client.request("GET", "/", name="user C")

class UserD(HttpUser):
host = host
@task
def x(self):
    self.client.request("GET", "/", name="user D")

class UserE(HttpUser):
host = host
@task
def x(self):
    self.client.request("GET", "/", name="user E")`

it happens when I want to create small amount of users on multiple workers, for higher numbers of users it works fine, on one machine it's also ok

@cyberw I tested twice the case of sending an interrupt to the master (ctrl+c) with 4 workers while it is spawning and couldn't reproduce the traceback you had. Here are the logs for one of my test. The scenario is using 4 workers, 3 user classes (weights: 4, 5, 1), and load test shape with the following profile:
load_profile:
  - duration: 1m
    users: 3
    spawn_rate: 0.5
  - duration: 45m
    users: 30
    spawn_rate: 0.5

I sent the interrupt just after the second spawning to 30 users started. Also note that my scenario includes some long-running tasks (>5min) which explains why all the users are not stopped within the stop timeout threshold. Can you share what kind of scenario you were running so I can reproduce the issue?
As for the worker logs (what you showed were one of the worker logs, right?), what would be nicer in your opinion? What I did was adapt the previous log message to now show the number of each user class being spawned, stopped, etc. Because a log should be a single line, I chose to serialize the user distribution to json. The previous log was only indicating Spawning xx users (yy users already running). What I can do is indicate both, e.g:
Spawning additional 5 users, 1 user already running (spawning: {"DemoUser1" 2, "DemoUser2": 3}, already running: {"DemoUser1": 1, "DemoUser2": 0})

Or I could separate into two logs: the previous log message with INFO level and the new one with DEBUG level. When the level is DEBUG, I could inhibit the INFO level log to reduce duplication. Personally, I'd prefer the combined version.

I spoke in private on Slack with @stanislawskwark and it seems that their test environnement might somehow still be running the 1.4.3 version of locust thus producing the incorrect behaviour. I ran the test with the given locustfile (7 user classes), 20 workers, and 10 expected users and here are the results:

I then increased the user count to 100 with a spawn rate of 1:

As expected, all the user classes are correctly distributed, the global spawn rate is respected and the users are well balanced across workers.

Here's a minimum reproduction of the error (I'm using locust-swarm which uses SSH port forwarding between master and worker, but I dont think it makes any difference, and I've stripped away all the fluff provided by locust-swarm so it is just plain commands, which you should be able to run yourself). Maybe you can even run everything locally...
I've stripped away everything from the test plan:
from locust import events, task, HttpUser
from locust.user.wait_time import constant

class DemoUser(HttpUser):
    host = f"https://www.google.com"
    wait_time = constant(2)

    @task
    def t(self):
        self.client.get("/")

I interrupt master after a short while. Normally there should be one or two report tables printed by this time as well:
locust --master --master-bind-port 6000 --master-bind-host=<IP_ADDRESS> --expect-workers 1 --headless -f d2.py --exit-code-on-error 0 --web-host=<IP_ADDRESS> -u 10
[2021-03-03 22:19:38,640] my-machine/INFO/root: Waiting for workers to be ready, 0 of 1 connected
[2021-03-03 22:19:38,802] my-machine/INFO/locust.runners: Client 'my-remote-machine_d061d3f096974715ac50dc22f87cf3ec' reported as ready. Currently 1 clients ready to swarm.
[2021-03-03 22:19:39,641] my-machine/INFO/locust.runners: Sending spawn jobs of 10 users and 1.00 spawn rate to 1 ready clients
^CKeyboardInterrupt
2021-03-03T21:19:47Z
Traceback (most recent call last):
  File "/usr/local/bin/locust", line 33, in <module>
    sys.exit(load_entry_point('locust', 'console_scripts', 'locust')())
  File "/Users/lafp/git/locust/locust/main.py", line 331, in main
    runner.start(options.num_users, options.spawn_rate)
  File "/Users/lafp/git/locust/locust/runners.py", line 598, in start
    for dispatched_users in users_dispatcher:
  File "/Users/lafp/git/locust/locust/dispatch.py", line 116, in dispatch_users
    gevent.sleep(max(0.0, wait_between_dispatch - delta))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/gevent/hub.py", line 166, in sleep
    hub.wait(t)
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
KeyboardInterrupt```

Worker:

ssh -q -R 6000:localhost:6000 -R 6001:localhost:6001 my-remote-machine 'LOCUST_PORT="6000" locust --worker --master-port 6000 --headless -f d2.py'
[2021-03-03 22:19:34,014] my-remote-machine/INFO/locust.main: Starting Locust 1.4.3
[2021-03-03 22:19:39,687] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 0} already running)...
[2021-03-03 22:19:39,688] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:39,688] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 1 still running
[2021-03-03 22:19:40,690] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 1} already running)...
[2021-03-03 22:19:40,690] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:40,690] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 2 still running
[2021-03-03 22:19:41,694] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 2} already running)...
[2021-03-03 22:19:41,694] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:41,694] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 3 still running
[2021-03-03 22:19:42,699] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 3} already running)...
[2021-03-03 22:19:42,699] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:42,699] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 4 still running
[2021-03-03 22:19:43,699] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 4} already running)...
[2021-03-03 22:19:43,699] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:43,699] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 5 still running
[2021-03-03 22:19:44,703] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 5} already running)...
[2021-03-03 22:19:44,703] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:44,703] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 6 still running
[2021-03-03 22:19:45,706] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 6} already running)...
[2021-03-03 22:19:45,707] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:45,707] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 7 still running
[2021-03-03 22:19:46,712] my-remote-machine/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 7} already running)...
[2021-03-03 22:19:46,712] my-remote-machine/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-03 22:19:46,712] my-remote-machine/INFO/locust.runners: 0 Users have been stopped, 8 still running
^C```
If I actually let it finish spawning it starts printing reports, and shut down works as normal.
locust --master --master-bind-port 6000 --master-bind-host=<IP_ADDRESS> --expect-workers 1 --headless -f d2.py --exit-code-on-error 0 --web-host=<IP_ADDRESS> -u 10
[2021-03-03 22:37:01,434] my-machine/INFO/root: Waiting for workers to be ready, 0 of 1 connected
[2021-03-03 22:37:01,552] my-machine/INFO/locust.runners: Client 'my-remote-machine_11a039bed0cf48eaaa723862365ecbb7' reported as ready. Currently 1 clients ready to swarm.
[2021-03-03 22:37:02,437] my-machine/INFO/locust.runners: Sending spawn jobs of 10 users and 1.00 spawn rate to 1 ready clients
[2021-03-03 22:37:11,463] my-machine/INFO/locust.main: No run time limit set, use CTRL+C to interrupt.
[2021-03-03 22:37:11,464] my-machine/INFO/locust.main: Starting Locust 1.4.3
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                             16  16(100.00%)  |       6       3      12       6  |    1.00    1.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        16  16(100.00%)  |       6       3      12       6  |    1.00    1.00

 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                             30  30(100.00%)  |       6       3      12       6  |    1.78    1.78
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        30  30(100.00%)  |       6       3      12       6  |    1.78    1.78

KeyboardInterrupt
2021-03-03T21:37:13Z
[2021-03-03 22:37:13,872] my-machine/INFO/locust.main: Running teardowns...

@cyberw, can you try again with the latest commit I pushed?
From my understanding, it can happen whenever a KeyboardInterrupt is received while waiting in a gevent.sleep. So it could also happen in the shape_worker in the gevent.sleep(1).

I confirm it works fine now for me and it solves the issues with distribution of users I had on 1.4.3

That seems to have solved the crash, but I'm still not getting any reports printed during ramp up. Also there seems to be no logging saying ramp up has finished?
[2021-03-05 08:48:24,441] lafp-mac-JG5J.int.svenskaspel.se/INFO/root: Waiting for workers to be ready, 0 of 1 connected
[2021-03-05 08:48:25,014] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Client 'perftest03.test.svenskaspel.se_88982844eba14957870dd19de976c44c' reported as ready. Currently 1 clients ready to swarm.
[2021-03-05 08:48:25,442] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Sending spawn jobs of 10 users and 1.00 spawn rate to 1 ready clients
[2021-03-05 08:48:34,475] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.main: No run time limit set, use CTRL+C to interrupt.
[2021-03-05 08:48:34,475] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.main: Starting Locust 1.4.3
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                             25  25(100.00%)  |       6       3      15       6  |    1.71    1.71
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        25  25(100.00%)  |       6       3      15       6  |    1.71    1.71

Here's the output from running standalone locust. Note how I'm getting reports much earlier than in distributed.
[2021-03-05 08:52:53,872] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.main: No run time limit set, use CTRL+C to interrupt.
[2021-03-05 08:52:53,872] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.main: Starting Locust 1.4.3
[2021-03-05 08:52:53,872] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 0} already running)...
[2021-03-05 08:52:53,873] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-05 08:52:53,873] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 1 still running
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                         0     0(0.00%)  |       0       0       0       0  |    0.00    0.00

[2021-03-05 08:52:54,875] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 1} already running)...
[2021-03-05 08:52:54,876] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-05 08:52:54,876] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 2 still running
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              2   2(100.00%)  |      46      41      51      41  |    0.00    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                         2   2(100.00%)  |      46      41      51      41  |    0.00    0.00

[2021-03-05 08:52:55,878] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 2} already running)...
[2021-03-05 08:52:55,879] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-05 08:52:55,879] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 3 still running
[2021-03-05 08:52:56,883] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 3} already running)...
[2021-03-05 08:52:56,883] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-05 08:52:56,883] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 4 still running
 Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              6   6(100.00%)  |      32      15      51      33  |    1.00    1.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                         6   6(100.00%)  |      32      15      51      33  |    1.00    1.00

And these two lines are spammy:
[2021-03-05 08:52:53,873] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-05 08:52:53,873] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 1 still running

Users of one type seem to be spawned before the second type:
[2021-03-06 00:00:57,525] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 0, "DemoUser2": 0} already running)...
[2021-03-06 00:00:57,525] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-06 00:00:57,526] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 1 still running
[2021-03-06 00:00:58,528] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 1, "DemoUser2": 0} already running)...
[2021-03-06 00:00:58,528] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-06 00:00:58,528] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 2 still running
[2021-03-06 00:00:59,530] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 2, "DemoUser2": 0} already running)...
[2021-03-06 00:00:59,531] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-06 00:00:59,531] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 3 still running
[2021-03-06 00:01:00,532] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 3, "DemoUser2": 0} already running)...
[2021-03-06 00:01:00,532] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-06 00:01:00,533] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 4 still running
[2021-03-06 00:01:01,534] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser": 1} ({"DemoUser": 4, "DemoUser2": 0} already running)...
[2021-03-06 00:01:01,534] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser spawned
[2021-03-06 00:01:01,534] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 5 still running
[2021-03-06 00:01:02,539] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser": 5, "DemoUser2": 0} already running)...
[2021-03-06 00:01:02,539] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-06 00:01:02,540] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 6 still running
[2021-03-06 00:01:03,540] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser": 5, "DemoUser2": 1} already running)...
[2021-03-06 00:01:03,541] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-06 00:01:03,541] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 7 still running
[2021-03-06 00:01:04,544] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser": 5, "DemoUser2": 2} already running)...
[2021-03-06 00:01:04,545] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-06 00:01:04,545] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 8 still running
[2021-03-06 00:01:05,545] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser": 5, "DemoUser2": 3} already running)...
[2021-03-06 00:01:05,545] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned
[2021-03-06 00:01:05,545] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: 0 Users have been stopped, 9 still running
[2021-03-06 00:01:06,546] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: Spawning additional {"DemoUser2": 1} ({"DemoUser": 5, "DemoUser2": 4} already running)...
[2021-03-06 00:01:06,547] lafp-mac-JG5J.int.svenskaspel.se/INFO/locust.runners: All users of class DemoUser2 spawned

Users of one type seem to be spawned before the second type:

This is the expected behaviour during a ramp up. During ramp up, to respect the global spawn rate, users are dispatched class-by-class. The end distribution of users amongst worker is balanced (equal number of users ±1 for each worker).
The distribution and dispatch code is thoroughly tested in test_dispatch.py and test_distribution.py.
Regarding your previous comments, I will address them.

Users of one type seem to be spawned before the second type:

This is the expected behaviour during a ramp up. During ramp up, to respect the global spawn rate, users are dispatched class-by-class. The end distribution of users amongst workers is balanced (equal number of users ±1 for each worker).

The distribution needs to be (reasonably) balanced during ramp up as well, so it needs to be fixed before we can merge.

@mboutet - do you think you will be able to work on this PR to improve the distribution? We use plenty of different user classes with weights and the distribution class by class also won't fit our usage pattern.
At the same time, the current hatching algorithm (1/s = 1 VU per worker/s) means that the more workers we have the faster we spawn leading to some unwanted behaviors.

@domik82, no I don't have time to work on that.
I did try to balance the user classes during spawning, but it turns out that it is more difficult than it seems. Namely, it was difficult to handle all the cases such as a worker already having some unbalanced users running on it.
I'm not saying that it is impossible to spawn each user class in a round-robin fashion (as opposed to all users of UserClassA, then all users of UserClassB, etc.). It's more that the current implementation seemed satisfying and already far less erratic than the previous locust (i.e. master branch), so I decided it was enough.
Beside, I think that if you really wish to closely control the number of users over time, LoadTestShape is the way to go. If you use a LoadTestShape to slowly spawn the users over a relatively long period of time, the spawning will be balanced.
I'll probably look into this in the future, but it won't be done in the short term because I'm already busy. I'll periodically merge the master in this branch in order to keep it fresh.
@domik82, if you're willing to contribute to this PR and modify the dispatch logic to balance the user classes during spawning, you can contact me and we can work on this together. I'm on the locust's Slack workspace under the same handle as on GitHub.

I prototyped some changes to dispatch the user class in a balanced fashion so that during ramp-up, the users also approximately respect the final distribution at the end of the ramp up. I'll have to update my tests. Don't expect this change to be ready in the short term as I will work on this when I have some downtime.

@domik82, I refactored the dispatch logic in order for the user classes to be balanced during a ramp-up.
You can observe the behaviour in this test for instance: locust/test/test_dispatch.py#L508. It'd be great if you could validate that it works with your use cases.
Note: I have no idea why this particular test is failing in the pipeline, all tests are passing locally using make test.

@domik82, I refactored the dispatch logic in order for the user classes to be balanced during a ramp-up.

@mboutet we did some testing yesterday and it looks ok from the distribution perspective.
We need to do some more tests as data we sent do influx was gone and we don't know if this is something in our code or in your branch/master.

I checked again and I can confirm what @domik82 says - I'm quite sure it's caused by this code though. I ran the same code on 1.4.3 and then and this branch. on 1.4.3 everything worked as expected, but on this branch I can't see the data in Grafana. To me it looks like behavior of locust events somehow changed.

Which event hook(s) in particular are you suspecting?

From: stanislawskwark @.>
Sent: Wednesday, April 7, 2021 7:11:10 AM
To: locustio/locust @.>
Cc: Maxence Boutet @.>; Mention @.>
Subject: Re: [locustio/locust] Better distribution of users and fix distributed hatch rate (#1621)
CAUTION: This email originated from outside of Explorance. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I checked again and I can confirm what @domik82https://github.com/domik82 says - I'm quite sure it's caused by this code though. I ran the same code on 1.4.3 and then and this branch. on 1.4.3 everything worked as expected, but on this branch I can't see the data in Grafana. To me it looks like behavior of locust events somehow changed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/locustio/locust/pull/1621#issuecomment-814824328, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMPI63EBLDKOVFYFZHTA7Y3THQ4U5ANCNFSM4TONJ7SA.

checked it and it's missing "user_count" in "on_worker_report" event

checked it and it's missing "user_count" in "on_worker_report" event

Thanks @stanislawskwark! I will fix it today.

problem I mentioned before is fixed :) works fine for me

@mboutet During load tests last week, we've found that there is a performance issue while you have for example 6 users classes in test and you want to distribute 2k users. https://github.com/mboutet/locust/blob/users-distribution/locust/distribution.py#L81 creates 27M combinations which are processed for 60 minutes. I've checked flame graph for this case and most of the time was spent in https://github.com/mboutet/locust/blob/users-distribution/locust/distribution.py#L84.

@dannyfreak, indeed this is not good. I'll look into it. Thanks for the investigation.
Can you give me a little more details on your configuration?

Are you using load test shape?
What's the spawn rate?

@dannyfreak, What's weird is that I have a unit test for such a case which computes the distribution for<PHONE_NUMBER>783 total users (just an arbitrarily large number) and 15 user classes.

Ok, I can reproduce cases where the number of combinations explodes! I'll fix this. Thanks a lot @dannyfreak  for catching this!

@dannyfreak, I published a version with a potential fix https://pypi.org/project/locust-mboutet/<IP_ADDRESS>/. Can you try it and let me know it fixes the performance issue?

@dannyfreak, I published a version with a potential fix https://pypi.org/project/locust-mboutet/<IP_ADDRESS>/. Can you try it and let me know it fixes the performance issue?

Cool, I'll check it later on our use case and will let you know.

@mboutet I spawned 2000 users with 6 user classes and got:

line 79, in _find_ideal_users_to_add_or_remove
assert number_of_users_to_add_or_remove <= len(user_classes), number_of_users_to_add_or_remove
AssertionError: 77

Ok, sorry about that, I added this assertion because I wasn't sure if this case could happen. I'll look into it and get back to you.

@dannyfreak, is there any way you could share your locust file or give more info on your configuration? What are the weights assigned to each of the user class?

@dannyfreak, is there any way you could share your locust file or give more info on your configuration? What are the weights assigned to each of the user class?

It works :) Assertion worked properly, I had an issue with my code.

@dannyfreak, glad it worked! For how long have you been using the locust from this PR?

@mboutet We've been using for couple of months from time to time for testing the user distribution. Last week we agreed to test it more thorough, because it looks like it suits our needs for proper user distribution. I will do more testing this week if everything will work fine, we would be very happy if this change could be merged to master.

The fact that this broke together with the changes in #1769 (imo) shows that it is kind of fragile.
The PR fixes some important issues (respecting user distribution with large number of workers and/or load shapes) but is a change.
The other issues that it solves (distribution over workers being added/removed during a run) are (in my opinion) niche, and if you ignore them a much simpler solution should be possible (perhaps in line with what I suggested in https://github.com/locustio/locust/pull/1621#issuecomment-761832693)
You can of course continue testing it and improving the code if you want, but know that it will not be accepted into master. I just want to reiterate this so that people don't waste their time.

@cyberw
The fact that you think it broke because of the change in https://github.com/locustio/locust/pull/1769 is just wrong and false.
The test that is failing is not captured in the current state of the master branch, since no such test exists in master.
And for your information, the problem is also present for the master branch. It becomes evident, when you add these tests to  test_runners.py:

Click to expand
class TestMasterWorkerRunners2(LocustTestCase):
    @staticmethod
    def _dump_nodes_states(master) -> str:
        return json.dumps(
            {
                "master.state": master.state,
                "workers_seen_from_master": [{"state": client.state} for client in master.clients.all],
            },
            indent="  ",
        )

    def test_distributed_shape1(self):
        class TestUser1(User):
            def on_start(self):
                sleep(1.5)

            @task
            def my_task(self):
                sleep(15)

            def on_stop(self):
                sleep(1.5)

        class TestUser2(User):
            def on_start(self):
                sleep(0.5)

            @task
            def my_task(self):
                sleep(5)

            def on_stop(self):
                sleep(0.5)

        class TestUser3(User):
            def on_start(self):
                sleep(1)

            @task
            def my_task(self):
                sleep(6)

            def on_stop(self):
                sleep(1)

        class TestUser4(User):
            def on_start(self):
                sleep(0.1)

            @task
            def my_task(self):
                sleep(45)

            def on_stop(self):
                sleep(0.1)

        class TestUser5(User):
            def on_start(self):
                sleep(0.1)

            @task
            def my_task(self):
                sleep(20)

            def on_stop(self):
                sleep(0.1)

        class TestUser6(User):
            def on_start(self):
                sleep(0.8)

            @task
            def my_task(self):
                sleep(60)

            def on_stop(self):
                sleep(0.5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 2:
                    return 9, 9
                elif run_time < 10:
                    return 100, 30
                elif run_time < 20:
                    return 60, 10
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            test_shape = TestShape()
            master_env = Environment(
                user_classes=[TestUser1, TestUser2, TestUser3, TestUser4, TestUser5, TestUser6],
                shape_class=test_shape,
                stop_timeout=15,
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(60):
                worker_env = Environment(
                    user_classes=[TestUser1, TestUser2, TestUser3, TestUser4, TestUser5, TestUser6]
                )
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(time.time() - ts <= 60, self._dump_nodes_states(master))
                sleep()

    def test_distributed_shape2(self):
        class TestUser1(User):
            def on_start(self):
                sleep(1.5)

            @task
            def my_task(self):
                sleep(15)

            def on_stop(self):
                sleep(1.5)

        class TestUser2(User):
            def on_start(self):
                sleep(0.5)

            @task
            def my_task(self):
                sleep(5)

            def on_stop(self):
                sleep(0.5)

        class TestUser3(User):
            def on_start(self):
                sleep(1)

            @task
            def my_task(self):
                sleep(6)

            def on_stop(self):
                sleep(1)

        class TestUser4(User):
            def on_start(self):
                sleep(0.1)

            @task
            def my_task(self):
                sleep(45)

            def on_stop(self):
                sleep(0.1)

        class TestUser5(User):
            def on_start(self):
                sleep(0.1)

            @task
            def my_task(self):
                sleep(20)

            def on_stop(self):
                sleep(0.1)

        class TestUser6(User):
            def on_start(self):
                sleep(0.8)

            @task
            def my_task(self):
                sleep(60)

            def on_stop(self):
                sleep(0.5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 10:
                    return 5, 3
                elif run_time < 20:
                    return 10, 3
                elif run_time < 30:
                    return 15, 3
                elif run_time < 60:
                    return 5, 3
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            test_shape = TestShape()
            master_env = Environment(
                user_classes=[TestUser1, TestUser2, TestUser3, TestUser4, TestUser5, TestUser6],
                shape_class=test_shape,
                stop_timeout=15,
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(5):
                worker_env = Environment(
                    user_classes=[TestUser1, TestUser2, TestUser3, TestUser4, TestUser5, TestUser6]
                )
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(
                    time.time() - ts <= 90,
                    self._dump_nodes_states(master),
                )
                sleep()

    def test_distributed_shape3(self):
        class TestUser1(User):
            def on_start(self):
                sleep(0.5)

            @task
            def my_task(self):
                sleep(60)

        class TestUser2(User):
            def on_start(self):
                sleep(0.5)

            @task
            def my_task(self):
                sleep(15)

        class TestUser3(User):
            def on_start(self):
                sleep(0.5)

            @task
            def my_task(self):
                sleep(5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 10:
                    return 5, 3
                elif run_time < 20:
                    return 10, 3
                elif run_time < 30:
                    return 15, 3
                elif run_time < 60:
                    return 5, 3
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            master_env = Environment(
                user_classes=[TestUser1, TestUser2, TestUser3], shape_class=TestShape(), stop_timeout=20
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(2):
                worker_env = Environment(user_classes=[TestUser1, TestUser2, TestUser3])
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            # After 60s, the test should stop. So if at 120s the master is still not
            # in the "stopped" state, then something is wrong.
            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(
                    time.time() - ts <= 120,
                    self._dump_nodes_states(master),
                )
                sleep()

    def test_distributed_shape4(self):
        class TestUser6(User):
            def on_start(self):
                sleep(0.8)

            @task
            def my_task(self):
                sleep(60)

            def on_stop(self):
                sleep(0.5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 10:
                    return 5, 3
                elif run_time < 20:
                    return 10, 3
                elif run_time < 30:
                    return 15, 3
                elif run_time < 60:
                    return 5, 3
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            test_shape = TestShape()
            master_env = Environment(
                user_classes=[TestUser6],
                shape_class=test_shape,
                stop_timeout=15,
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(5):
                worker_env = Environment(user_classes=[TestUser6])
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(
                    time.time() - ts <= 120,
                    self._dump_nodes_states(master),
                )
                sleep()

    def test_distributed_shape5(self):
        class TestUser6(User):
            def on_start(self):
                sleep(0.8)

            @task
            def my_task(self):
                sleep(60)

            def on_stop(self):
                sleep(0.5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 10:
                    return 5, 3
                elif run_time < 20:
                    return 10, 3
                elif run_time < 30:
                    return 15, 3
                elif run_time < 60:
                    return 5, 3
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            test_shape = TestShape()
            master_env = Environment(
                user_classes=[TestUser6],
                shape_class=test_shape,
                stop_timeout=1,
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(5):
                worker_env = Environment(user_classes=[TestUser6])
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(
                    time.time() - ts <= 120,
                    self._dump_nodes_states(master),
                )
                sleep()

    def test_distributed_shape6(self):
        class TestUser6(User):
            def on_start(self):
                sleep(0.8)

            @task
            def my_task(self):
                sleep(60)

            def on_stop(self):
                sleep(0.5)

        class TestShape(LoadTestShape):
            def tick(self):
                run_time = self.get_run_time()
                if run_time < 10:
                    return 5, 3
                elif run_time < 20:
                    return 10, 3
                elif run_time < 30:
                    return 15, 3
                elif run_time < 60:
                    return 5, 3
                else:
                    return None

        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
            test_shape = TestShape()
            master_env = Environment(
                user_classes=[TestUser6],
                shape_class=test_shape,
                stop_timeout=1,
            )
            master_env.shape_class.reset_time()
            master = master_env.create_master_runner("*", 0)

            workers = []
            for i in range(10):
                worker_env = Environment(user_classes=[TestUser6])
                worker = worker_env.create_worker_runner("<IP_ADDRESS>", master.server.port)
                workers.append(worker)

            # Give workers time to connect
            sleep(0.1)

            self.assertEqual("ready", master.state)
            self.assertEqual(len(workers), len(master.clients.ready))

            # Start a shape test
            master.start_shape()

            ts = time.time()
            while master.state != STATE_STOPPED:
                self.assertTrue(
                    time.time() - ts <= 120,
                    self._dump_nodes_states(master),
                )
                sleep()

Some of these tests can run for more than 2 minutes because of the nature of the scenarios. At least, when I run these tests on my computer, they fail most of the time.
Also, the logic in locust.runners.WorkerRunner.worker and locust.runners.MasterRunner.client_listener, which is where part of the problem is, has mostly been left untouched.
I found that the problem happens when a combination of one or more of the following conditions are met:

There are multiple workers, and the ratio of total_user_count/workers_count is not too high, i.e. each worker does not run a lot of users.
There are users with tasks taking more than just a few seconds, e.g. a task taking 30s.
There are multiple user classes with tasks taking different amount of time.
Stop timeout is set to a non-zero value. From what I can observe, a higher stop timeout will make the problem happen more frequently.

The chain of event is as follows:

Master sends stop to all workers.
Each worker goes into this logic:

elif msg.type == "stop":
    self.stop()
    self.client.send(Message("client_stopped", None, self.client_id))
    self.client.send(Message("client_ready", None, self.client_id))
    self.worker_state = STATE_INIT

When master receives the client_stopped it removes the worker from its clients dict.
When master receives the client_ready almost immediately after the client_stopped it adds back the worker to its clients dict.

When there is no stop timeout and there are few workers, then the new check_stopped condition will have the opportunity to transition the master to a stopped state because the client_ready will not be received until the next iteration of the locust.runners.MasterRunner.client_listener loop. However, if there's a stop timeout combined with some long-running tasks in the workers, then the probability that the master receives all the client_stopped from all workers before receiving any client_ready is greatly lowered.
If the above logic is modified to be:
elif msg.type == "stop":
    self.stop()
    self.client.send(Message("client_stopped", None, self.client_id))
    # +2 is just a small buffer to account for the random network latencies and/or other
    # random delays inherent to distributed systems.
    gevent.sleep((self.environment.stop_timeout or 0) + 2)
    self.client.send(Message("client_ready", None, self.client_id))
    self.worker_state = STATE_INIT

Then all workers should be able to send their client_stopped before any of the workers sends the client_ready message. That way, the master is able to transition to the stopped state. You can run the tests I provided to see that they now pass.
Next time you plan on making such a comment, take a little time to understand what's happening beforehand.

The other issues that it solves (distribution over workers being added/removed during a run) are (in my opinion) niche, and if you ignore them

Your opinion is wrong, and you are simply too close minded. The only thing you see is your use case. It's not because your
use cases are simpler that every other use cases are wrong. Others have signaled their desires of having these long-standing bugs (YES these are bugs!). Your refusal of even acknowledging that these are major bugs will simply drive away people that have real and more complex use cases. It's such a shame because Locust is great otherwise.

a much simpler solution should be possible (perhaps in line with what I suggested in #1621 (comment))

Again, you are wrong and blind to the fact that the master is the one that needs to maintain a consistent state. The fact that you blamed me for the problem introduced in https://github.com/locustio/locust/pull/1769 without attempting to analyse the problem itself leads me to believe you haven't taken the time to analyze that the solution you are proposing is nonsense.

After working with Locust for almost year I would say that Locust code is quite fragile and has plenty of bugs.
The fact is:
-users are not spawning with proper distribution
-users are sometimes not spawning at all
-we have way more users we would expect with higher amount of workers

sometimes we have to kill process before it would be fully stopped as stopping takes forever
while stopping VU are kind of hanging still creating the load while shouldn't
workers sometimes are not connecting to master
when worker disconnects the missing VU were not distributed to remaining nodes

The above code is the only one existing that tries to solve some of those problems. There are no other active users trying to solve it.
My team uses Locust every single day. When we gather people from reliability / dev /ops etc to assist us with observing the platform under load and we have trouble to run scenario not only we waste time we make fools of us.
I'm also not convinced with giving workers the decision what to do. If master controls everything it gives opportunity to change code further and introduce new features.
I'm sure there are other bugs that can show up. It's a complex problem and while I'm not sitting in the Locust code I doubt solution indeed can be that simple as described.
Me and my team are highly interested in simply solving the issue(s). The fact that there is only one active maintainer that can arbitrary approve/disapprove the solution makes me think if this was right choice of the tool.
The other choice is to live with those issues as no one fixes them which in our case is not possible as our retro shows that stability of running the bigger load is a concern.
I'm not really sure why we don't won't to make it work release and revert back if it would cause any major issue.

Hi!
I am sorry if I misjudged the problem in that other issue. I did try running the test you mentioned on master and that worked for me.
I respect your effort on this PR and wish you all the best. Maybe you are right and I am wrong, but I am not convinced and I have no time to debate it further.
Having been on the other end of this discussion (getting what seemed to me a reasonable and useful PR rejected by some know-it-all) I know this is incredibly frustrating. But as we cannot agree on a common solution there is no point in arguing further.

-users are not spawning with proper distribution
-users are sometimes not spawning at all

I believe these things could be solved along the lines I described with much smaller code changes. Has someone made an attempt to do it?

-we have way more users we would expect with higher amount of workers

That is very strange and worthy of investigation. Having used locust for ~5 years (although never with more than maybe 20-30 workers) I have not seen this issue.

sometimes we have to kill process before it would be fully stopped as stopping takes forever
while stopping VU are kind of hanging still creating the load while shouldn't
workers sometimes are not connecting to master

These are strange issues indeed. I may have encountered some of them myself a few times and they are really annoying. But is it likely that this PR would fix that?

when worker disconnects the missing VU were not distributed to remaining nodes

I consider such a feature unimportant as a test with disconnecting workers is invalid, and having higher than expected number of users on the remaining nodes is likely to create an extremely hard-to-debug-scenario.

Maybe I rushed my opinion.
If @domik82, and @dannyfreak run this for a while and find rampup to be completely linear & deterministic (in the normal case when workers dont disappear during the test) then I will have a look again later.
Perhaps @heyman will weigh in as well.

I did try running the test you mentioned on master and that worked for me.

I find that very surprising because I can assure you that everytime I run these tests on the master branch, they fail unless I apply the small change I proposed. I just switched back to the master branch (088a98bf8ff4035a0de3becc8cd4e887d618af53), added the tests and ran make test and can still observed the failures.

I believe these things could be solved along the lines I described with much smaller code changes. Has someone made an attempt to do it?

If this is so simple, then why don't you do it? If your hypothetical solution is so simple, then it should not take too much time to implement. Thus, your argument of you having no time to at least come up with a PoC is invalid. I think you grossly underestimate the overall difficulty of the problem. And you live in a lie thinking that this can be easily and reliably solved with a "small & simple" solution which does not involve the master as the authority.
As @domik82 mentioned, having the master controlling everything opens up the door for new and interesting features.

I consider such a feature unimportant as a test with randomly disconnecting workers is invalid

Again, only seeing your own interests and discarding all of the others which as @domik82 stated is very disturbing given you are the only active maintainer that chooses the solutions to accept or reject. This makes the way locust is managed very autocratic to say the least.
Also, "a test with randomly disconnecting workers" is far from being invalid. Say you run your load tests on k8s where pods are by definition ephemeral and can come and go at will. This right there is a valid case of "randomly disconnecting workers". If you run your load tests on spot VMs to save costs and one of the VMs is evicted, then it would be great to have the users rebalance onto the remaining nodes. Sure it's not ideal as each node will have more cpu/mem contention, but at least the whole test is not invalidated which is especially a good thing when running large scale multi-hours tests. Besides, a new node could be added and reconnect to the master in 10-15min or even less if scripted.

find rampup to be completely linear & deterministic (in the normal case when workers dont disappear during the test) as well as the code being stable of course, then I will have a look again later.

It's like you expect my changes to be 100% perfect and 100% bug free when locust from master is far from it. You literally have double standards. I'd say that my changes are already an improvement over "vanilla" locust. Besides, I addressed all the concrete bugs you and the others reported quite promptly. I'll be happy to address further bugs because it also benefits myself. So I literally have no advantages in ignoring bugs.

@domik82, @stanislawskwark, @dannyfreak I just published the latest version of this PR if you want to try it out: pip install locust-mboutet==<IP_ADDRESS>. It (theoraticaly) fixes the master node staying in the "stopping" state at the end of a test under certain conditions.

@cyberw - me and @dannyfreak are working within same team. Sadly for us @stanislawskwark left our team.
In total I have 12 people in dedicated performance team - that is a lot of testing going on in daily basis and we are seeing quite a lot of strange issues sometimes that "re-run" will fix.
Due to the fact that we are running HTPP load along with asynchronous websocket load we have to use way more CPU resources to handle that web socket traffic compared to "simple" HTTP load.
Currently we are already running tests with 150 workers as a single job and as we run different load scenarios next to each other we utilise 220 workers in total having multiple master nodes. That number grows every month as we push limits of the platform and fix perf issues in it.
That is a lot to spawn and a lot of things might go wrong.
So when you say we should stop the test if a single node disconnects I would say - if CPU can handle it (and we are trying to aim at 75% so we won't cross 85) - we should do absolutely everything to continue so re run is not needed.
We push so big load that queues need sometimes 30 min to drain. Once we start we shouldn't simply stop. This is our use case.
We have smaller daily regression scenarios running for 3h on 2 different environments / micro benchmark running 24/7 - usually those don't cause other problem than CPU.
Almost every week we run capacity tests. This is where we need to make sure that everything is stable. This is why you will see more engagement but I don't see the point of fixing issues that are already almost solved. Head of my department will not be happy about it.
At the moment we are occupied also with issue where we think that pacig is consuming way more CPU compared to test run without it.
We have our own "hacked" features that potentially we could share like transaction grouping.
We use shapes as json files with different type of shapes.
Community won't see those if we will have to spent time in reinventing the weel like figuring out a better fix while the problem seems to be quite complex and @mboutet spent already significant amount of time working on it .
I bet our complex scenarios will either run successfully or fail just like the issue Danny found.
Our engagement will be there for longer but I really need a wise decision making if with so little support we can afford rejecting PR nearly done.

@mboutet we have holidays in Poland now but I can assure you we will test it next week (it's asigment for @dannyfreak)
This issue is hurting us now so badly that we really need urgent fix for this.

@mboutet we have holidays in Poland now

Have some rest 😀 I'll be waiting for your feedback.

@mboutet we have holidays in Poland now

Have some rest 😀 I'll be waiting for your feedback.

Well seeing rejection comment is quite concerning having in mind that just 2 days ago I spent couple of hours with my team talking about what is most important "next" thing and Locust stability was mentioned multiple times.
It goes both with "Vanilla" issues as well our own code to make it better.
I simply hoped that your could would make our life simpler.

@cyberw - me and @dannyfreak are working within same team.

Cool!

Sadly for us @stanislawskwark left our team.

Thats too bad!

In total I have 12 people in dedicated performance team - that is a lot of testing going on in daily basis and we are seeing quite a lot of strange issues sometimes that "re-run" will fix.

Wow, cool stuff!

Due to the fact that we are running HTPP load along with asynchronous websocket load we have to use way more CPU resources to handle that web socket traffic compared to "simple" HTTP load.

Interesting. I have done a fair bit of websocket load and not encountered such issues (although perhaps not with high amount of traffic, just many many subscribers). I'll be doing some more websocket tests soon btw. Are you using an approach similar to SocketIOUser in locust-plugins or some other method? Sounds a little like maybe you're using something that isn't gevent-friendly (otherwise there should be no problem running somewhere around 1000 Users per worker, I dont see why websocket load should be super-heavy, unless maybe the client implementation is bad)

Currently we are already running tests with 150 workers as a single job and as we run different load scenarios next to each other we utilise 220 workers in total having multiple master nodes. That number grows every month as we push limits of the platform and fix perf issues in it.
That is a lot to spawn and a lot of things might go wrong.
So when you say we should stop the test if a single node disconnects I would say - if CPU can handle it (and we are trying to aim at 75% so we won't cross 85) - we should do absolutely everything to continue so re run is not needed.

I see. I guess there is a small sweet spot where some nodes may disconnect and the test result might still be valid if the load is rebalanced, but I think it is not the common case. Especially with such high cpu usage to start with you are going to get wildly varying results if nodes start disconnecting.
Anyways, I see the point I just dont think it is a major one. I dont think we should shut down by default if a worker disconnects, I was just saying it could be an alternative solution for load suddenly going "missing".
(in your particular case you should probably probably increase timeouts and/or increase worker count further so that you dont get disconnects. getting "some" disconnects is not "normal")

We have our own "hacked" features that potentially we could share like transaction grouping.
We use shapes as json files with different type of shapes.

Please do! Sharing is caring (and makes me care more about your needs ;)
Some features may fit better in locust-plugins (there's actually an implementation of transaction grouping there already)

I just did a short test and I still see incorrect distribution during ramp up. For sure, this can be fixed but the PR is not not "nearly done". I'm starting to warm up to the idea though (particularly when I think back on when I was on the other end :)

Using <IP_ADDRESS>, 4 workers, ramping up to 500
class DemoUser1(RestUser):
    weight = 1

    @task
    def t(self):
        self.client.get("/", name="1")
        time.sleep(500)

class DemoUser2(RestUser):
    weight = 2

    @task
    def t(self):
        self.client.get("/", name="2")
        time.sleep(500)

@cyberw I'm really glad to hear it.
Imagine the surprise when with way bigger distribution one of the critical user types ( ~5% out of 100) - is not triggered at all during entire scenario. It has to have small weight because it have big impact (it even has a simple prediction algorytm to know how to do things).
This was the point in time when I started to think I don't want any more workarounds. Because there is one - next Jenkins job, next master.
Simple output based on perf results we produce goes to CTO level. I just have to have way more confidence that everything goes as planned and we don't need to do so many double checks.

@cyberw I'm really glad to hear it.
Imagine the surprise when with way bigger distribution one of the critical user types ( ~5% out of 100) - is not triggered at all during entire scenario. It has to have small weight because it have big impact (it even has a simple prediction algorytm to know how to do things).

Yes, I've been in a similar situation myself, but the only case where this is a known issue is when running more workers than number of users divided by the fraction of total weight of your "rare" user, or when using LoadShapes (which makes the same rounding at each step, making the problem worse). Usually that can be worked around and if you have more issues than that then maybe you have some other problem with your setup...

Simple output based on perf results we produce goes to CTO level. I just have to have way more confidence that everything goes as planned and we don't need to do so many double checks.

You're assuming the proposed solution to be better than the old, much more tested one. I think it is just as likely that it will just have new and different problems (maybe even bigger ones, as the solution is more complex). We all want to make locust better, there is no argument there. Also, I get that this is important to you, but please stop with the "power talk". Your CTO doesnt pay my bills, so he gets no special consideration from me.
But anyways, back on topic: If the issues I have mentioned (comments in the code + the weird user balance during ramp up) get solved I will take another look.

@cyberw all I want to say is that myself and my company are quite serious about using Locust and it's not a "random" tool we picked.
At this stage I'm simply not in position to change the tool and start from scratch.
At the moment there is no other approach - code is broken and @mboutet is the only one willing to fix it. At this stage he also has the knowledge where to look for bug fix. My team will simply struggle at the begging and by amount of changes it looks like at least 2-4 weeks of work.

@cyberw all I want to say is that myself and my company are quite serious about using Locust and it's not a "random" tool we picked.

👍

At the moment there is no other approach - code is broken and @mboutet is the only one willing to fix it. At this stage he also has the knowledge where to look for bug fix. My team will simply struggle at the begging and by amount of changes it looks like at least 2-4 weeks of work.

Let see where it goes. (not sure exactly what you mean by "struggle at the begging")
You have at least changed my mind from "this is way to big, it is just not happening" to "if it can be done really well then maybe, because I'm not up for doing it myself in the way I think it should be done".

@cyberw, yes I see what you're saying. The users are kind of balanced during a ramp up, but they do not indeed follow the final distribution. I will look to fix this as it would be better to have the correct distribution at all time. I say "kind of balanced" because prior to https://github.com/locustio/locust/pull/1621/commits/6bb4b4158b7f4e4c3fc8e5c2b2583bee582cb1d6, it was even worse as all users of DemoUser1 were first spawned, then all of users of DemoUser2 were spawned, etc. Right now, the behaviour is that all users will be equally spawned.  When the user count of a given class reaches its target, then the dispatcher continues with the other classes, and so on. Anyway, I reiterate that I agree this is not ideal and will work to improve it.
In an earlier comment you were saying to demonstrate the linearity of a ramp-up. So I ran your scenario with varying amount of workers:

Click to expand
4 workers:

10 workers:

30 workers:

100 workers:

150 workers:

Of course, it's not a 100% perfect straight line but for all practical purposes it is a linear. There's no solution that will ever give us a perfectly straight line as it depends on the randomness of the network stack, cpu contention on the workers and master, and the fact that the plot is based on what is reported to the master by each worker. Also, I ran these tests on my computer that was already running other unrelated compute intensive stuff.

I did try running the test you mentioned on master and that worked for me.

I know that for some reasons you don't believe me, but I'm 99.9999% sure that the master also has the stopping state problem. I provision the 0.0001% in case I'm wrong 😛... I re-ran the test I provided numerous times and could reproduce the issue systematically both on my branch and on the master.
I still believe this comment of yours from yesterday was made in bad faith just to undermine what this PR is attempting to fix. As far as I know, this problem introduced by https://github.com/locustio/locust/pull/1769 is not at all related to this PR and also happens on the master. I just happened to have a test that is not in the master and could reproduce the issue. Have you had such a test in the master, you would have seen the regression from https://github.com/locustio/locust/pull/1769 right away. We could argue that this is a rare or corner case situation for most users, but it still is a regression/bug nonetheless.
What is bothering me from the beginning is your general attitude not just here but in some other issues as well. It's often the "I don't consider this a bug", "If it does this and that, then the test should be considered invalid". I really don't understand this attitude of yours. It really is not contributing in improving locust in the long term. What if I said that I believe the less-than-ideal distribution during ramp-up is a non-issue because it doesn't apply to my use cases? Of course I can't say that because I have the short end of the stick and you are the only one that can decide of the outcomes. For the record, my use cases are not really impacted by the behaviour, but if I can improve it and it can benefit others, then I'm happy to work on it.
I'm not bothered at all by this PR still not being merged as I know that it's not "perfect" and can still benefit testing and improvement as the problems it attempts to solve are complex. However, I can't alone cover all the use cases as the test suite that was already present doesn't cover them all. When instead people take a little time to test theses changes as you just did, things move forward and help improve the changes.
Another thing you are always claiming is that the changes will be hard to debug because of the added complexity. We can all agree that the new logic adds complexity, but the problem is really complex to begin with. I'm 99.9999% sure this can't be solve with a "quick & simple" solution even though you don't believe me. Vanilla locust is simpler because it was cutting corners for the aspects this PR tackles. So, my answer to "hard to debug" argument is that from the beginning I implemented an extensive test suite for the distribution and dispatch logics which is where most of the complexity is added:

locust/test/test_dispatch.py
locust/test/test_distribution.py

These tests are there to validate the behaviour as precisely as possible and catch possible regressions or bugs that may arise. If cases are not covered or wrong (as for the distribution during ramp-up), then I'll simply add or modify tests in consequence. I'm under the impression that you think the vanilla locust is easy to debug. Let me assure you that this is not the case. By now, I've spent hundredS of hours on this part of the codebase (vanilla & my branch). Finding what was causing the incorrect behaviour was not an easy task when I started working on this.
So thanks again for taking the time to test and report the distribution problem, I will come back with a solution and modified tests for locust/test/test_dispatch.py.

@cyberw, I managed to improve the dispatch so that the distribution is kept according to the weights during a ramp-up.
See the logs below:

Click to expand
[2021-06-03 14:33:35,069] mboutet-MBP/INFO/locust.runners: Client 'mboutet-MBP_90e1bb4c6b144dc7bfbb904b53f6bf26' reported as ready. Currently 1 clients ready to swarm.
[2021-06-03 14:33:35,069] mboutet-MBP/INFO/locust.runners: Client 'mboutet-MBP_62fff779a0a24ccb90f4cbbf00bf2545' reported as ready. Currently 2 clients ready to swarm.
[2021-06-03 14:33:35,069] mboutet-MBP/INFO/locust.runners: Client 'mboutet-MBP_f1c1001550bf4386a0afb6b7f837ba65' reported as ready. Currently 3 clients ready to swarm.
[2021-06-03 14:33:35,071] mboutet-MBP/INFO/locust.runners: Client 'mboutet-MBP_0ecbbb3a924e4f419d2c14cd0ff135db' reported as ready.

I'll make sure all the current tests pass and will push the update probably in the upcoming days.

@cyberw I was thinking about your solution that worker would still do the spawning. If I understood correctly workers still wouldn't be aware of each other.
With that case it's still not solving issue that was reported way back in the past and I think it's still there.
Let's say that we want to run capacity scenario using transaction A and B. 'A' could be "login" - B "add stuff to basket".
You know A has way smaller capacity compared to B. Therefore we push A once  'on start'  and B as a task.
We plan to use 100 workers as we plan cover capacity of B and we would like to do slow rump up not to overload A.
If each worker doesn't know about each other - in this case let's assume max value for A is 50 transaction per sec. You can set 1 but as each worker would do it so you get 100 logins per sec not desired 1.
I think there is no way to define smaller number unless you will use shape and this was the reason we started to use it more frequently.
If master controls the spawn rate defining 1 per sec should mean exactly that - 1 VU across entire cluster of workers and 99 workers should be idle per design.
Now I don't know if that behavior was fixed somewhere if not I would say that this PR should fix it (if it's not doing it already).

Hi everyone! First of all, let me say I am sorry for my poor attitude. This PR caught me "by surprise" and when I felt my initial feedback and suggestions were ignored I got even more frustrated. The reason I made that somewhat dramatic comment was that I felt we needed to make a decision, because the PR was just lingering.
In general, with a mature (not to say it is free of bugs or anything) OS project like Locust, changes need to be kept small, therefore I would have preferred a smaller change, even if that would leave a few niche requirements unsolved (they really seem niche to me, but I guess not to everyone). I guess one of the things that makes load generation tools a little peculiar is that they are used in so different ways. I use Locust in a very controlled way and would never accept slaves disappearing for instance, whereas others might use it for higher throughput or low level tests where a completed test has value even if there were some technical failures in the load generation side (I work almost exclusively with Locust's own metrics, but if you also use server side metrics then I guess a small amount of "flakiness" on the load generation side becomes more acceptable)
I was a little lazy and did not spend enough time looking at the code, and when I found some serious issues in the beginning I made the (incorrect) assumption that it was not good. What makes it even harder for me is that I did not write the original logic, so I feel a little unqualified to review its replacement (I'll still do it, but I'm just saying :)
Also, remember that I am just one guy and there are lots of users of locust with questions/contributions, so when you write comments that are super long I probably will not have time to read it all.

I still believe this comment of yours from yesterday was made in bad faith just to undermine what this PR is attempting to fix. As far as I know, this problem introduced by #1769 is not at all related to this PR and also happens on the master. I just happened to have a test that is not in the master and could reproduce the issue. Have you had such a test in the master, you would have seen the regression from #1769 right away. We could argue that this is a rare or corner case situation for most users, but it still is a regression/bug nonetheless.

I did not pretend or make up my test results, but I realize now that I must have made some error, because I did it again and can now see the error (I had to make some modifications to get the test to not crash, which I did not do last time, so I definitely messed up that time). My apologies. I did not fake my results to get some excuse to close the PR, but I'll admit that my sloppy analysis was probably impacted by my wish to just move on.
Phew. Ok, should we get back to actually talking about the code? :) I'll try to put in a few hours of testing and code reviewing this weekend and will get back to you some time next week.

@cyberw thanks for that comment and I'm glad we have you on board. Now that we have cleared the air I have one suggestion.
Person in my team (@dannyfreak) have actual "full time" story to work on this PR and I figure out if in our use case is behaving properly.
Thanks to that work we already found issues in our own code so there is double benefit on our side.
As I said earlier - we have long weekend here in Poland but we definitely we will have more results next week.
If that manual test runs would be a pass our plan for next phase was to switch temporarily to @mboutet version of Locust that is released and let it run on our daily builds so we would cought any random issues.
Now if you have spare time probably running your cases will be beneficial. But my feeling is your time could be limited by the work my coworkers will do.

Now if you have spare time probably running your cases will be beneficial. But my feeling is your time could be limited by the work my coworkers will do.

Cool. I'll hang back for a bit and maybe focus on reviewing the code itself then.

Hi, @mboutet I've started testing latest version. There is some strange behaviour of workers during our tests. They don't start in parallel but sequentially.
This is our  master branch without your changes:

This is locust-mboutet <IP_ADDRESS>

This is another kind of test from our master branch without your changes:

This is locust-mboutet <IP_ADDRESS>

Hi @dannyfreak,
This is expected and behaves "as designed".
As you know, the master branch just sends the number of users to spawn along with a spawn rate to all workers. This spawn rate is the desired spawn rate divided by the number of workers. Indeed, this results in all the workers spawning the users at the same time. However, this leads to the known issues of incorrect user distributions and incorrect global spawn rate, e.g. spawning 500 users with a spawn rate of 1/s to 100 workers results in huge 100 users staircase-like bumps versus a linear increase to 500 users.
The new algorithm dispatches users to one worker at a time until the worker is at its target user count, then goes to the next worker and so on. You can observe this behaviour in locust/test/test_dispatch.py#L1977. Keep in mind that the total user count running on a given worker NEVER exceeds total_users / number_of_workers which is what we want.
I did try at some point to implement the dispatch logic so that users would be dispatched in a round-robin fashion to workers. However, this added more complexity, and I decided against implementing it (for now). It did not add much value anyway because the final number of users running on a given worker at the end of a spawn cycle will be exactly the same whether the users are dispatch as they currently are or if they are dispatch in a round-robin way. I'm not against improving this logic eventually, but I would do it in a separate PR after this PR is merged.
I hope my explanation is clear enough. If not, let me know.
Also, I don't know how many user classes and workers you are using in the above tests, but beware that in the master version, the number of each user class running on each worker might be wrong. Thus, it is possible that with the master branch, less cpu-intensive users are spawned than what you actually expect.

Also, remember that I am just one guy and there are lots of users of locust with questions/contributions, so when you write comments that are super long I probably will not have time to read it all.

I completely understand and I think you're doing a fantastic job of replying to all the daily issues on GH and on Slack as well as all the rest.
I'm committed to Locust both for work and personal projects. You'll have to trust my words on this, but I'll be there to fix issues with my changes because they benefit myself as well.
It's out of scope for this PR, but I was thinking about it, and the distribution/dispatch logic could be somethings pluggable so that one can choose the "distributor/dispatcher" they need.
Anyway, I'll be going through the comments you made in the code in the upcoming days.

@dannyfreak, I published locust-mboutet-<IP_ADDRESS> which contains a bug fix. The bug lead to a non-respect of the spawn rate when going from n1 users to n2 users with n1 > 0 and n2 > n1. For example, going from 25 users to 50 users at 1/s went directly from 25 to 50 users instead of increasing linearly during 25s.
I also improved the dispatch logic in rare situations where some workers are running more users than what is desired. Prior to this improvement, the dispatcher would add users to these workers even though they already had users in excess. Now, the dispatcher doesn't add users to those workers.

@mboutet / @albertowar - we did run the scenario today that quite extensively is using different Locust features. Pacing, user classes (over 20), shapes and quite large volume of users (multiple masters and max single master with 20k VU).
We didn't spot any obvious issues on Locust side. The results got changed in our case by having proper distribution.
I believe that because of it we got higher CPU usage on workers (more heavy users). We still plan to execute few more scenarios to confirm it but at this stage I can't find any obvious defect.

@domik82, @dannyfreak, I published locust-mboutet-<IP_ADDRESS>. It does not contain bug fixes, new features, or change in behaviour, but I refactored some code. I also synced with the master branch containing the changes from pull requests #1788 and #1793. So if you have the chance to use this version instead, that'd be good.

@domik82, @dannyfreak, I just published locust-mboutet-<IP_ADDRESS> which incorporates the new custom message feature from pull request #1782. So use that instead if possible.

I'm liking the progress for this, and I have run a few basic tests myself again, and so far everything looks stable. A few more thoughts:

We still need to reduce the amount of logging somehow. Running headless is too noisy now. Moving the "Updating running test with x users" to debug level might be a start, but I'm not sure what would be the best approach. Maybe moving both to debug makes sense.

[2021-07-01 16:17:42,502] lars-mbp.local/INFO/locust.main: No run time limit set, use CTRL+C to interrupt.
[2021-07-01 16:17:42,503] lars-mbp.local/INFO/locust.main: Starting Locust <IP_ADDRESS>
Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregated                                                         0     0(0.00%)  |       0       0       0       0  |    0.00    0.00

[2021-07-01 16:17:42,505] lars-mbp.local/INFO/locust.runners: Updating running test with 1 users
[2021-07-01 16:17:42,505] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (0 already running)...
[2021-07-01 16:17:43,506] lars-mbp.local/INFO/locust.runners: Updating running test with 2 users
[2021-07-01 16:17:43,506] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (1 already running)...
Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET 1                                                              1     0(0.00%)  |     227     227     227     227  |    0.00    0.00
GET 2                                                              1     0(0.00%)  |     194     194     194     194  |    0.00    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregated                                                         2     0(0.00%)  |     210     194     227     194  |    0.00    0.00

[2021-07-01 16:17:44,506] lars-mbp.local/INFO/locust.runners: Updating running test with 3 users
[2021-07-01 16:17:44,506] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (2 already running)...
[2021-07-01 16:17:45,508] lars-mbp.local/INFO/locust.runners: Updating running test with 4 users
[2021-07-01 16:17:45,508] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (3 already running)...
Name                                                          # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
GET 1                                                              3     0(0.00%)  |     295     227     335     320  |    1.00    0.00
GET 2                                                              1     0(0.00%)  |     194     194     194     194  |    0.00    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
Aggregated                                                         4     0(0.00%)  |     270     194     335     230  |    1.00    0.00

[2021-07-01 16:17:46,508] lars-mbp.local/INFO/locust.runners: Updating running test with 5 users
[2021-07-01 16:17:46,508] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (4 already running)...
[2021-07-01 16:17:47,509] lars-mbp.local/INFO/locust.runners: Updating running test with 6 users
[2021-07-01 16:17:47,509] lars-mbp.local/INFO/locust.runners: Spawning additional 1 (5 already running)...
...

Should we reduce the default weight of users from 10 to 1? I think the old value (which doesnt make much sense) was set because of rounding issues or something (which the updated code seems to have no problems with). We should do this at the same time so we do all the breaking changes at once (I could do it in a separate PR though, if that is better for you)

We still need to reduce the amount of logging somehow

I published locust-mboutet-<IP_ADDRESS> in which I made a small refactor for the logging. I ran headless in both local and distributed mode and I think that the INFO logging is now more manageable and less verbose. If one wishes to see more details, DEBUG provides all the low-level logs. Let me know what you think.

Should we reduce the default weight of users from 10 to 1?

I was not even aware the default weights were set to 10 😛 I went back the history when weightwas added in 2011 (I haven't realised Locust was that "old" 😮) and it seems it's been set to 10 from the start. Anyway, we could probably change this in a separate PR. Perhaps  a v2 branch could be created in which all the PRs for v2 are merged in, what do you think? Thus, if there are fixes to be made to the master branch in the meantime, they can be done and you can publish other 1.x.y releases.

We still need to reduce the amount of logging somehow

I published locust-mboutet-<IP_ADDRESS> in which I made a small refactor for the logging. I ran headless in both local and distributed mode and I think that the INFO logging is now more manageable and less verbose. If one wishes to see more details, DEBUG provides all the low-level logs. Let me know what you think.

Looks good! Is there a better word than "Updating test with X users" though? Maybe "Target set to X users" or something. "Updating" (to me at least) implies that there was already some relevant state even before (which isnt really the case when running headless)

Should we reduce the default weight of users from 10 to 1?

I was not even aware the default weights were set to 10 😛 I went back the history when weightwas added in 2011 (I haven't realised Locust was that "old" 😮) and it seems it's been set to 10 from the start. Anyway, we could probably change this in a separate PR. Perhaps a v2 branch could be created in which all the PRs for v2 are merged in, what do you think? Thus, if there are fixes to be made to the master branch in the meantime, they can be done and you can publish other 1.x.y releases.

Hmm. @heyman , do you have the reason for the 10 default weight? I can worry about changing this (worst case it might be something for 3.x :) )
I think we're good to go soon, and having a "flat" structure of only PR:s to master is good for changelog etc, so I'm leaning towards no "v2" branch (possibly one or more 2.0bX prereleases, but that's all). I will also make a 1.6.x branch just before we merge this PR, in case we need to build a release on the old code.

Is there a better word than "Updating test with X users" though? Maybe "Target set to X users" or something.

I think that Updating test to %d users using a %.2f spawn rate or Updating user count to %d using a %.2f spawn rate could be better. The transitive verb Updating implies that the test will soon  be updated to reach the indicated target user count whereas Target set to X users could be interpreted as the test has already been updated to run X users. I'm not a native-speaker, but that's how I interpret it.

"Updating" (to me at least) implies that there was already some relevant state even before (which isnt really the case when running headless)

Perhaps our interpretation is different, but there is always a prior state. When this is the first spawning when the test just started, the prior state is "0 users".

do you have the reason for the 10 default weight?

For the new code, the weight doesn't matter. The weights could be small 0.00005, 0.007, etc. and it won't cause any issues (unless you go so small as to mess with the float precision). The only breakage it will cause is for people having user classes with and without explicit weights in which case they will need to explicitly set weight = 10 to keep the previous behaviour.

"Updating" (to me at least) implies that there was already some relevant state even before (which isnt really the case when running headless)

Perhaps our interpretation is different, but there is always a prior state. When this is the first spawning when the test just started, the prior state is "0 users".

Probably just a language thing. But as a user who is just starting a run (at least in my mind), the previous state is not that there are "zero users running", it is that there is not even a test running at all (so the user count is undefined, not zero). Maybe "Ramping to X users..."?

do you have the reason for the 10 default weight?

Just my 2 cents, but for the new code, the weight doesn't matter. The weights could be small 0.00005, 0.007, etc. and it won't cause any issues (unless you go so small as to mess with the float precision). The only breakage it will cause is for people having user classes with and without explicit weights in which case they will need to explicitly set weight = 10 to keep the previous behaviour.

👍 I've added a PR and will merge it after this.

Good for Ramping to %d users using a %.2f spawn rate. I'll push this tomorrow along with the warning on the spawn rate.

I published locust-mboutet-<IP_ADDRESS> with the modified log message and the reinstated worker spawn rate warning as discussed.

👍 looks really good now. I’ll try to get my colleague to try this out (check the slack thread @DennisKrone ) next week and if he finds no issues I’ll merge and make a first prerelease build (probably 2.0b0). I’ll merge the other breaking changes after that and make new prerelease builds that include those.
Also, I have looked into how to make prerelease builds for each commit, so once 2.0 (release version) is done I’ll add that.

Great! thanks for your valuable feedback these past weeks. And also thank you @domik82 and @dannyfreak for your extensive tests.
I've uploaded locust-mboutet-<IP_ADDRESS> containing the latest changes.

I'm ready to merge this now. There's nothing more that needs to happen first, right @mboutet ?

Bombs away!