Documentation: How can server broadcast to all connected clients at regular interval?

I'm trying to implement a WebSocket server that, once it's ready to accept connections, starts generating data periodically. Whenever data is generated (e.g. every 2 seconds), I want to broadcast the data to all connected clients.
I looked at the documentation and could not find an example like that. There is an example where the WebSocket server sends a message periodically to connected clients, but each client has its own "interval" (i.e. all clients won't receive the message at the same time).
It would be great if the documentation had such an example. I'm coming from Node.js and this is pretty easy to do with the 'ws' npm package.

You need to:

keep track of connected clients e.g. like this https://websockets.readthedocs.io/en/stable/intro.html#registration
start an asyncio task that broadcasts the data; this task must in parallel to your server

Well, it's certainly not a stupid question but I'd say it's more of an asyncio question, so I'm not sure it needs to be added to the docs. But here's how it could be done:
import asyncio

class MessageGenerator:
    def __init__(self):
        self.message = asyncio.Future()

    async def run(self):
        while True:
            await asyncio.sleep(1.0)
            self.message.set_result(b'meow')
            self.message = asyncio.Future()

async def handler(name, message_generator):
    while True:
        message = await asyncio.shield(message_generator.message)
        print(name, message)

async def main():
    message_generator = MessageGenerator()
    asyncio.create_task(message_generator.run())
    asyncio.create_task(handler('1', message_generator))
    asyncio.create_task(handler('2', message_generator))
    await asyncio.sleep(10.0)

asyncio.run(main())

Your server can now just do what the handler function does in its client handler function.

@lgrahl I'm not sure I understand your example.
Here's the sort of thing I had in mind (completely untested):
CLIENTS = set()

async def broadcast():
    while True:
        for ws in CLIENTS:
            await ws.send("woof")
        await asyncio.sleep(2)

asyncio.create_task(broadcast())

async def handler(websocket, path):
    CLIENTS.add(websocket)
    try:
        async for msg in ws:
            pass
    finally:
        # Unregister.
        CLIENTS.remove(websocket)

start_server = websockets.serve(handler, ...)

asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Ah, now I understand, you're having all clients wait on a shared future.

Probably worth giving this example in the docs.

Thank you @aaugustin and @lgrahl. I really appreciate it!
@aaugustin:
For your documentation example, the following line:
asyncio.create_task(broadcast())
generates a RuntimeError: no running event loop exception. I changed it to:
asyncio.get_event_loop().create_task(broadcast())
Which works. But again, I'm new to asyncio so I'm not certain it's the right solution.
Also, for some reason, whenever a client disconnects, I get the following message in the console:
Error in connection handler Traceback (most recent call last):   File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websockets/server.py", line 195, in handler     await self.ws_handler(self, path)   File "server.py", line 30, in handler     async for _ in websocket:   File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websockets/protocol.py", line 414, in __aiter__     yield await self.recv()   File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websockets/protocol.py", line 485, in recv     await self.ensure_open()   File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websockets/protocol.py", line 781, in ensure_open     raise self.connection_closed_exc()
However, the program continues running and everything works fine.

asyncio.get_event_loop().create_task(broadcast())

That's correct. Sorry.

Also, for some reason, whenever a client disconnects, I get the following message in the console:

If the client closes the connection summarily — without a WebSocket closing handshake — then you get an error. If you don't care, you can catch it and ignore it:
async def handler(websocket, path):
    CLIENTS.add(websocket)
    try:
        async for msg in websocket:
            pass
    except websockets.ConnectionClosedError:
        pass
    finally:
        CLIENTS.remove(websocket)

This makes me realize broadcast is too naive: if a client is slow or unresponsive, it will crash and exit.
This is better:
async def broadcast():
    while True:
        await asyncio.gather(
            *[ws.send("woof") for ws in CLIENTS],
            return_exceptions=False,
        )
        await asyncio.sleep(2)

Beautiful. And good to know for the closing handshake. Thanks again!

I'm going to keep this issue open to improve the documentation.

Roger that, my bad! Also, this makes me wonder... Does the "Synchronization example" in the "Getting started" section have the same naive broadcasting problem? That is, this line:
await asyncio.wait([user.send(message) for user in USERS])
Inside of notify_state() and notife_users().

I'll double check.

hi, I just use the broadcast example to make a test, while the server always hangs there after some time. Could someone give any advice? thanks. The attached is the code.
broadcast.zip

@willfu
Thanks for putting up a test code that can be useful for me. I think the reason for the hang is because of the global interpreter lock and the queue being full. The server hangs at workQueue.put(msg). You can easily reproduce the hangs by changing await asyncio.sleep(0.01) to await asyncio.sleep(0.02) so that the server runs slower than the client. Your server can be an easy target for DDoS.

I've been thinking about describing both patterns in the docs and (without actually experimenting) I came up with the following pros/cons:
Waiting on a shared future

Pro: elegant — asyncio takes care of the pub/sub, ultimately by looping through callbacks attached to the Future by each coroutine that's waiting on the Future
Con: can miss messages if the next message is generated before all subscribers have had the chance to run and grab the last message

Looping through connections

Pro: obvious — simplest thing that could possibly work
Con: it's easy to shoot yourself into the foot and write it in such a way that one slow client blocks every other
Con: under load, messages could arrive out of order — ordering relies on the fact that asyncio will start executing tasks in the order in which they are scheduled, which is an implementation detail.

Ideally we want the following properties:

clients receive all messages in order (while they're connected)
clients with good connections receive messages immediately

We do not want the following properties:

backpressure (i.e. slow consumers / clients with bad connections would slow down the rate of broadcasting)

You can pull Kafka but you may lose the "lightweight" aspect of websockets :trollface:
Or you can try the pattern proposed in #968:

for each connection, create a queue and start a task that constantly reads from the queue
broadcast by pushing messages in queues, which can be synchronous if you don't limit the size of the queue

Pro: has all the properties we want
Pro: harder to get wrong than "looping through connections"
Con: more pieces in the puzzle
Con: adds a level of buffering

Queues just contain pointers to messages so their memory use will be small compared to the outgoing TCP buffers. If memory use is a concern, the tasks that read messages from queues and send them can abort the connection if the queue reaches a certain size.
Here's untested code showing the idea:
import asyncio

QUEUE_MAX_SIZE = 100
QUEUES = {}

async def relay(queue, websocket):
    while True:
        message = await queue.get()
        await websocket.send(message)
        # Disconnect and exit if client is too far behind
        if queue.qsize() >= QUEUE_MAX_SIZE:
            await websocket.close()

async def handler(websocket, path):
    queue = asyncio.Queue()
    relay_task = asyncio.create_task(relay)
    QUEUES.add(queue)
    await websocket.wait_closed()
    QUEUES.remove(queue)
    relay_task.cancel()

async def broadcast():
    while True:
        for queue in QUEUES:
            queue.put_nowait("woof")
        await asyncio.sleep(2)

asyncio.create_task(broadcast())

@lgrahl I'm interested in hearing your opinion about this pattern :-) Often you see things that I miss!

I haven't thought about this very long, but it seems like rather than having a queue for every connection, you should be able to have a single queue, with each connection storing its relative position in the queue (the item it has consumed up to). And you can remove elements from the queue when either (1) all connections have read that element, or (2) you could have some expiration logic where an item will only stay in the queue for a certain amount of time, etc, whether or not all connections have read that item.

Yes, that's another possibility — closer do how Kafka operates :-)
I'm not sure there's a benefit worth the complexity, though.
My immediate thought was "it could be more memory efficient". However, before a queue starts filling up, asyncio's write buffer must be full, which is already 64kB. In my design, I give up at 100 items; I assume 1 item in the queue takes 8B so the total is 0.8kB. This isn't going to make a significant difference compared to the 70kB (connection object) + 64kB (full write buffers) baseline with default settings.

My immediate thought was "it could be more memory efficient".

I was also thinking about avoiding the need to loop through every connection every 2 seconds to populate their queues. With a single queue, you only need to do per-connection work when a connection pulls data, which could be considerably less frequent. Maybe that doesn't save much either, though. I'm not sure.

Oh, yes.
There would be just one iteration across all connections instead of two. It would be performed by asyncio: the single queue would provide a single Future on which all consumers would block (similar to Lennart's proposal above), and when that Future is set, asyncio would loop through coroutines to resume them.

I ran an benchmark to compare three broadcast methods.
1. Naive
Loop through connected clients and send messages sequentially.
Connection handler
CLIENTS = set()

async def handler(websocket, path):
    CLIENTS.add(websocket)
    try:
        async for _ in websocket:
            pass
    finally:
        CLIENTS.remove(websocket)

Broadcast task
async def send(websocket, message):
    try:
        await websocket.send(message)
    except websockets.ConnectionClosed:
        pass

            ...
            # Since the loop can yield control, make a copy of CLIENTS
            # to avoid: RuntimeError: Set changed size during iteration
            for websocket in CLIENTS.copy():
                await send(websocket, message)
            ...

2. Task-based
Loop through connected clients and send messages concurrently
Connection handler
CLIENTS = set()

async def handler(websocket, path):
    CLIENTS.add(websocket)
    try:
        async for _ in websocket:
            pass
    finally:
        CLIENTS.remove(websocket)

Broadcast task
async def send(websocket, message):
    try:
        await websocket.send(message)
    except websockets.ConnectionClosed:
        pass

            ...
            for websocket in CLIENTS:
                asyncio.create_task(send(websocket, message))
            ...

3. Queue-based
Loop through messages and push messages via a queue
Connection handler
CLIENTS = set()

async def relay(queue, websocket):
    while True:
        message = await queue.get()
        await websocket.send(message)

async def handler(websocket, path):
    queue = asyncio.Queue()
    relay_task = asyncio.create_task(relay(queue, websocket))
    CLIENTS.add(queue)
    try:
        await websocket.wait_closed()
    finally:
        CLIENTS.remove(queue)
        relay_task.cancel()

Broadcast task
            ...
            for queue in CLIENTS:
                queue.put_nowait(message)
            ...

Methodology
I start a server implementing each of these methods and broadcasting a 128 bytes messages every second. Compression is disabled (to minimize CPU overhead besides what I want to measure).
I started 4 processes, each of which connected 2.5k clients, for a total of 10k clients. Using 4 processes ensures that clients can keep up with the server.
Since this is a local benchmark, there are no slow clients, so the naive method works flawlessly.
For every iteration — so, every second — the server prints something like:
clients = 10000
load = 52.9% / average = 51.8%, late = 4.9 ms
broadcast = 326.2ms / average = 329.4ms, wait = 673.8ms

In this example there's 10k connected clients.
The ratio of process time / wall clock time is 52.9% over the last second. Since load can vary a lot, I smooth the curve; in this case, the average reports almost the same load, 51.8%.
The server started the current cycle 4.9ms after the target time, which means it's keeping up with the load. The task and queue methods require work after the broadcast operation, so even if the broadcast is fast, the server may not be able to keep up; in that case the cycle can start hundreds of ms late.
The broadcast operation took 326.2ms. I smooth the curve like for load. This yield almost the same figure, 329.4ms.
The server will wait 673.8ms before starting the next cycle.
Here are representative segments of the three methods.
1. Naive
clients = 10000
load = 34.6% / average = 45.6%, late = 5.5 ms
broadcast = 318.9ms / average = 328.5ms, wait = 681.1ms

clients = 10000
load = 30.3% / average = 44.1%, late = 4.8 ms
broadcast = 306.7ms / average = 326.4ms, wait = 693.3ms

clients = 10000
load = 29.4% / average = 42.6%, late = 5.7 ms
broadcast = 322.9ms / average = 326.0ms, wait = 677.1ms

clients = 10000
load = 50.3% / average = 43.4%, late = 4.2 ms
broadcast = 313.8ms / average = 324.8ms, wait = 686.2ms

clients = 10000
load = 58.9% / average = 44.9%, late = 1.8 ms
broadcast = 342.7ms / average = 326.6ms, wait = 657.3ms

clients = 10000
load = 42.7% / average = 44.7%, late = 3.1 ms
broadcast = 349.9ms / average = 328.9ms, wait = 650.1ms

clients = 10000
load = 63.3% / average = 46.6%, late = 2.9 ms
broadcast = 340.9ms / average = 330.1ms, wait = 659.1ms

clients = 10000
load = 50.2% / average = 46.9%, late = 2.3 ms
broadcast = 348.3ms / average = 331.9ms, wait = 651.7ms

clients = 10000
load = 41.6% / average = 46.4%, late = 1.8 ms
broadcast = 338.7ms / average = 332.6ms, wait = 661.3ms

clients = 10000
load = 50.1% / average = 46.8%, late = 3.8 ms
broadcast = 341.6ms / average = 333.5ms, wait = 658.4ms

2. Task-based
clients = 9998
load = 41.3% / average = 61.3%, late = 3.3 ms
broadcast = 59.3ms / average = 124.6ms, wait = 940.7ms

clients = 9998
load = 48.6% / average = 60.0%, late = 3.6 ms
broadcast = 559.1ms / average = 168.1ms, wait = 440.9ms

clients = 9998
load = 93.6% / average = 63.4%, late = 2.7 ms
broadcast = 55.4ms / average = 156.8ms, wait = 944.6ms

clients = 9998
load = 43.2% / average = 61.3%, late = 2.0 ms
broadcast = 51.6ms / average = 146.3ms, wait = 948.4ms

clients = 9998
load = 39.0% / average = 59.1%, late = 1.5 ms
broadcast = 55.7ms / average = 137.2ms, wait = 944.3ms

clients = 9998
load = 39.6% / average = 57.2%, late = 3.3 ms
broadcast = 59.1ms / average = 129.4ms, wait = 940.9ms

clients = 9998
load = 50.6% / average = 56.5%, late = 0.6 ms
broadcast = 565.4ms / average = 173.0ms, wait = 434.6ms

clients = 9998
load = 97.2% / average = 60.6%, late = 2.8 ms
broadcast = 55.6ms / average = 161.3ms, wait = 944.4ms

clients = 9998
load = 65.9% / average = 61.1%, late = 4.5 ms
broadcast = 73.9ms / average = 152.5ms, wait = 926.1ms

clients = 9998
load = 52.2% / average = 60.2%, late = 2.0 ms
broadcast = 82.2ms / average = 145.5ms, wait = 917.8ms

3. Queue-based
clients = 9999
load = 66.8% / average = 67.9%, late = 3.1 ms
broadcast = 571.7ms / average = 180.8ms, wait = 428.3ms

clients = 9999
load = 98.2% / average = 70.9%, late = 192.5 ms
broadcast = 53.7ms / average = 168.1ms, wait = 946.3ms

clients = 9999
load = 54.5% / average = 69.3%, late = 2.3 ms
broadcast = 123.0ms / average = 163.6ms, wait = 877.0ms

clients = 9999
load = 72.9% / average = 69.6%, late = 2.5 ms
broadcast = 67.6ms / average = 154.0ms, wait = 932.4ms

clients = 9999
load = 58.4% / average = 68.5%, late = 2.9 ms
broadcast = 616.1ms / average = 200.2ms, wait = 383.9ms

clients = 9999
load = 98.2% / average = 71.5%, late = 47.8 ms
broadcast = 101.3ms / average = 190.3ms, wait = 898.7ms

clients = 9999
load = 87.7% / average = 73.1%, late = 4.6 ms
broadcast = 66.7ms / average = 177.9ms, wait = 933.3ms

clients = 9999
load = 73.1% / average = 73.1%, late = 5.2 ms
broadcast = 579.7ms / average = 218.1ms, wait = 420.3ms

clients = 9999
load = 98.1% / average = 75.6%, late = 111.6 ms
broadcast = 87.7ms / average = 205.1ms, wait = 912.3ms

clients = 9999
load = 79.1% / average = 75.9%, late = 124.0 ms
broadcast = 58.4ms / average = 190.4ms, wait = 941.6ms

Summary
Load:

naive: 45%, some variability (I made no effort to stabilize the benchmark) but no spikes

task-based: ~60%, driven up by a spike to 100% every 5 seconds

queue-based: ~70%, often going close to 100%

So, in this benchmark, the naive method wins. This is probably because it never yields control to the event loop. Despite being a coroutine, send doesn't yield control to the event loop unless the write buffer is full. Since we're doing everything locally and have no slow clients, the write buffer remains empty.
Is the naive usable in practice? Well, maybe :-)
With default settings, the write buffer is 64kb and the ping timeout is 20s. So, if we send less than 3.2kB / sec on a connection, the connection will always timeout before the write buffer fills up!
 :-)

Benchmarking scripts are in #986.

I reran the tests with uvloop. Everything's generally a bit faster. Other than that I observe the same behavior.
"Representative" results below (i.e. unscientifically chosen roughly in the middle of the range.)

Naive

clients = 10000
load = 35.7% / average = 35.4%, late = 5.2 ms
broadcast = 142.4ms / average = 139.1ms, wait = 857.6ms

Task-based

clients = 10000
load = 45.1% / average = 45.1%, late = 7.3 ms
broadcast = 52.7ms / average = 148.4ms, wait = 947.3ms

Queue-based

clients = 9997
load = 45.7% / average = 55.0%, late = 1.5 ms
broadcast = 67.9ms / average = 108.4ms, wait = 932.1ms

@aaugustin Hello Aymeric, I reviewed your benchmark, There's an important point about the naive method :
As you mentioned , write_buffer will be filled for slow connections. In that case, the main send loop will await for one client's buffer to drain. that means adding a specific amount of latency to all next connections.

@aaugustin That benchmark is interesting but it is not realistic anyway. You have measured many parameters on the server side and there's not any parameter like real latency of each message in your benchmark. Of course CPU load is an important parameter but we have no sense of when these messages are arriving to our clients. So my suggestion is to send some kind of time to clients and calculate the real latency of messages on the client side, too.
But let me dive deeper and tell you what is happening here ( In my guess ) :
As you mentioned, create_task method causes CPU load to reach to 100% many times ( I faced the exact same result with create_task method ). Keeping CPU busy affects the overall server throughput and latency. I have a proof for what I said : In many cases when CPU load reaches and stucks at 100% some of connections will be dropped, and message latencies grow up to big numbers.

By the way, in terms of the use case, it would be good to know if the important thing is simply that each client receive each message, or is it equally important that each client receive each message quickly. Or is quickly better even if some messages are skipped?

@cjerdonek I think there's no need to say which one is more important, both are necessary. It would be much better if the benchmark tells us about count of sent and received messages and their latencies, So we can easily see which method works better in both sides. But I guess both methods have same message loss rate and the main point about queue method is its lower latency.

I think there's no need to say which one is more important, both are necessary.

Both might be necessary to you, but maybe not to someone else. How do you know what is necessary or important for someone else's use case? I was just making the point that if one is less important than the other, then it can affect what implementation would be best for that use case. I wasn't make the point for the purposes of the benchmark.

There's an important point about the naive method : As you mentioned , write_buffer will be filled for slow connections. In that case, the main send loop will await for one connection buffer to drain.

Yes, I'm aware of this I brought this up earlier: https://github.com/aaugustin/websockets/issues/653#issuecomment-514250856
(I still think it can work reliably in production if the message throughput is low enough to guarantee that a slow client will timeout before the buffer fills up.)

we are not even sure the how many messages per second are arriving to the client sides

WebSocket is running over TCP. All messages are arriving, or else the connection would drop and a ConnectionClosed exception would be raised eventually.

Keeping CPU busy affects the overall server throughput and latency.

Yes, this is why I'm benchmarking the CPU load. In #968 you wrote that going through a queue.

There's an important point about the naive method : As you mentioned , write_buffer will be filled for slow connections. In that case, the main send loop will await for one connection buffer to drain.

We're all clear on how this technique can fail. I brought it up earlier in this discussion: https://github.com/aaugustin/websockets/issues/653#issuecomment-514250856
So, let's forget about the naive method and focus on comparing the task-based and queue-based methods. In #968 you said that going through a queue was more efficient than calling create_task for each message to send. This is what I wanted to benchmark, because it wasn't obvious which one would have the lowest CPU load.

Of course CPU load is an important parameter but we have no sense of when these messages are arriving to our clients.

More accurately, since we're optimizing the server, we care about when they're leaving the server. What happens on the network and how long it takes to reach the client is off scope.
If I understand you correctly, you are proposing that the task-based method could be creating a lot of tasks and it could take time for the server to go through them, which would increase latency if the CPU saturates?
Exactly the same thing happens with the message-based method. Every time you push a message to a queue, the event loop must wake up the coroutine waiting on that queue. Latency would increase exactly in the same way if the CPU saturates.
Of course, adding a measure of latency in the benchmark could be the best way to settle this. I'll see what I can do.

create_task method causes CPU load to reach to 100% many times ( I faced the exact same result with create_task method ). Keeping CPU busy affects the overall server throughput and latency.

Yes. However, pushing messages to queue causes CPU to reach 100% even faster. It's worse.
So create_task is still the best we have and the recommendation we should make in the docs.

we are not even sure the how many messages per second are arriving to the client sides

Nothing to worry about here. WebSocket is running over TCP. All messages are arriving, or else the connection would drop and a ConnectionClosed exception would be raised eventually.

OK, I added a latency measurement to the benchmark. With asyncio and 10k clients:

Task-based: p99 ~ 400-500ms with occasional spikes to 800-900ms
Queue-based: p99 ~ 500-600ms with occasional spikes to 1000-1100ms

So it looks like the task-based method has better latency. For this benchmark, it's a logical consequence of lower CPU usage.

Latency measurement code is here.

Both might be necessary to you, but maybe not to someone else. How do you know what is necessary or important for someone else's use case? I was just making the point that if one is less important than the other, then it can affect what implementation would be best for that use case. I wasn't make the point for the purposes of the benchmark.

Yes, you are right but let me explain what I meant. as @aaugustin said, websocket uses TCP protocol which causes the message_loss to be equal to zero. When I wrote that comment, I was thinking about a crowded server with lots of clients which drops some connections ( because of ping timeout ), Then I called those messages which were going to be sent to these dropped connections lost messages. Yes I agree message_loss wasn't a suitable word there, But I meant the benchmark should counts transmitted messages to let us see if any method is leading to unexpected connection termination. I've also made some changes to the benchmark and I'm going to publish it in the next comment.

First of all I should appreciate @aaugustin for his benchmark. I ran that benchmark with different delays and client counts. In some conditions ( latency = 0.1 ) I found queue faster than create_task ( the difference was really small ). I can say you are right @aaugustin , create_task acts better than queue in most cases in your design. But If you are really searching for a more efficient way to do this there are better methods. I've made some changes to your benchmark. Source code is available here.
Benchmark's methodologies
1. Create task :
It is exactly as same as main benchmark's Task-based method.
2. Create task + batch sending :
This time I made an array for each connection and called it fake_buffer. Every time a producer wants to send a message to a specific client, message will be appended to its fake_buffer and buffer_send() will be called in a new task.
async def buffer_send(websocket,fake_buffer):
    try:
        for _ in range(len(fake_buffer)) :
            await websocket.send(fake_buffer.pop(0))
    except websockets.ConnectionClosed:
        pass

    ...
    for websocket,fake_buffer in CLIENTS:
        fake_buffer.append(message)
        asyncio.create_task(buffer_send(websocket, fake_buffer))
    ...

3. Queue + batch sending :
This method acts like the previous one but it uses a Queue-based design instead and avoids creating task for each send operation.
async def relay(queue, websocket):
    while True:
        message = await queue.get()
        await websocket.send(message)
        for _ in range(client_queue.qsize()):
            await websocket.send(client_queue.get_nowait())

    ...
    for queue in CLIENTS:
        queue.put_nowait(message)
    ...

Recording latencies
On the client side, I record all latencies to calculate average latency : avg_lat = sum_of_latencies / total_num_of_messages.
Finally, The results !
I ran benchmark with delay=0.2 , size=400 , num_of_clients=3000 , duration=30. I visualized the result on a scattered plot.
The X-AXIS represents latencies and each dot stands for one transmitted message. Y-AXIS has no meanings at all.

Message latency :

Create task Averege latency 			= 951416760.3328642 ns
Create task + Batch send Averege latency 	= 912500126.7984976 ns
Queue + Batch send Averege latency 		= 317753688.5 ns

Create task :

clients = 3000
load = 52.8% / average = 36.5%, late = 66.3 ms
broadcast = 90.3ms / average = 25.2ms, wait = 109.7ms

clients = 3000
load = 71.8% / average = 40.0%, late = 61.3 ms
broadcast = 14.6ms / average = 24.2ms, wait = 185.4ms

clients = 3000
load = 52.5% / average = 41.3%, late = 67.9 ms
broadcast = 15.3ms / average = 23.3ms, wait = 184.7ms

clients = 3000
load = 53.2% / average = 42.5%, late = 64.1 ms
broadcast = 17.6ms / average = 22.7ms, wait = 182.4ms

clients = 3000
load = 47.0% / average = 42.9%, late = 65.9 ms
broadcast = 94.6ms / average = 29.9ms, wait = 105.4ms

Create task + batch sending :

clients = 3000
load = 44.5% / average = 57.0%, late = 81.1 ms
broadcast = 16.4ms / average = 30.3ms, wait = 183.6ms

clients = 3000
load = 53.0% / average = 56.6%, late = 65.5 ms
broadcast = 16.7ms / average = 28.9ms, wait = 183.3ms

clients = 3000
load = 51.9% / average = 56.1%, late = 71.1 ms
broadcast = 97.8ms / average = 35.8ms, wait = 102.2ms

clients = 3000
load = 71.6% / average = 57.7%, late = 61.7 ms
broadcast = 16.5ms / average = 33.9ms, wait = 183.5ms

clients = 3000
load = 52.9% / average = 56.8%, late = 65.7 ms
broadcast = 17.1ms / average = 30.7ms, wait = 182.9ms

Queue + Batch sending :

clients = 3000
load = 5.9% / average = 1.9%, late = 66.6 ms
broadcast = 9.7ms / average = 10.9ms, wait = 190.3ms

clients = 3000
load = 0.0% / average = 1.7%, late = 62.9 ms
broadcast = 11.7ms / average = 11.0ms, wait = 188.3ms

clients = 3000
load = 0.0% / average = 1.5%, late = 66.9 ms
broadcast = 8.6ms / average = 10.7ms, wait = 191.4ms

clients = 3000
load = 0.0% / average = 1.4%, late = 65.4 ms
broadcast = 9.5ms / average = 10.6ms, wait = 190.5ms

clients = 3000
load = 5.6% / average = 1.8%, late = 80.0 ms
broadcast = 9.2ms / average = 10.5ms, wait = 190.8ms

As you can see there's not much difference between create task and create task + batch send. As you remember, the raw create_task was the winner of the first benchmark, But comparing to the new queue method the overhead of creating python Task object is significant.

OK - we reached different conclusions because we were optimizing for different conditions.
I'm optimizing a non-saturated event loop (CPU < 100%, late < 1% of cycle time). The task that sends a message is always scheduled before the next message is produced. Every message can be sent before the next one is produced.
You're optimizing a saturated event loop (CPU = 100%, late ~ 30-40% of cycle time). The task that sends a message isn't always scheduled before the next message is produced. (This is true regardless of whether the task with send multiple messages, in the queue-based design, or just one message, in the task-based design.) Not every message can be sent before the next one is produced, so sometimes you have more than one ready to send.

There are two "expensive" operations in our benchmark:

switching between tasks
writing to the network

In my version, I have 1 task switch and 1 write per event.
In your version, you have fewer than 1 task switch per event. This is where your gains come from. (If you batched multiple events into one message, you could also get fewer than 1 write per event).
Your design absorbs variability in an interesting way: if load increases, it compensates automatically by "batching more". Pretty cool!
I feel like running an event loop at 100% isn't a great idea but I don't have a proof. IMO it's the same level of hack as "the naive method works as long as you send less than 3.2kB / second / connection". If you tune it and monitor it carefully, you can get away with it.
The queue-based method still has the benefit of guaranteed ordering. I think I'll focus on this aspect.

Just to be clear, There's no need to push cpu load to 100% too see the effect of queue method. I'm sure in many cases ( e.g. having many producers in different coroutines with different broadcast cycles ) queue + batch method will acts better than create_task with CPU Load < 100% ( I can prove it with a benchmark if you want ). Anyway I ran my benchmark with clients=100 and delay=5 and here is the result :
Create task Averege latency 			= 50083886.202 ns
Queue + Batch send Averege latency 		= 43168221.42 ns

Create task

clients = 100
load = 0.6% / average = 0.1%, late = 7.4 ms
broadcast = 4.8ms / average = 1.1ms, wait = 4995.2ms

clients = 100
load = 0.5% / average = 0.1%, late = 7.2 ms
broadcast = 4.7ms / average = 1.4ms, wait = 4995.3ms

clients = 100
load = 3.9% / average = 0.5%, late = 3.9 ms
broadcast = 6.2ms / average = 1.9ms, wait = 4993.8ms

clients = 100
load = 0.5% / average = 0.5%, late = 5.4 ms
broadcast = 4.9ms / average = 2.2ms, wait = 4995.1ms

clients = 100
load = 2.3% / average = 0.7%, late = 3.7 ms
broadcast = 3.7ms / average = 2.4ms, wait = 4996.3ms

clients = 100
load = 0.2% / average = 0.6%, late = 7.8 ms
broadcast = 7.2ms / average = 2.8ms, wait = 4992.8ms

clients = 100
load = 0.2% / average = 0.6%, late = 5.7 ms
broadcast = 7.3ms / average = 3.3ms, wait = 4992.7ms

Queue

clients = 100
load = 0.0% / average = 0.0%, late = 4.4 ms
broadcast = 2.9ms / average = 0.3ms, wait = 4997.1ms

clients = 100
load = 0.5% / average = 0.1%, late = 7.6 ms
broadcast = 1.7ms / average = 0.4ms, wait = 4998.3ms

clients = 100
load = 0.1% / average = 0.1%, late = 8.3 ms
broadcast = 1.5ms / average = 0.5ms, wait = 4998.5ms

clients = 100
load = 3.6% / average = 0.4%, late = 3.5 ms
broadcast = 1.6ms / average = 0.6ms, wait = 4998.4ms

clients = 100
load = 0.1% / average = 0.4%, late = 8.1 ms
broadcast = 1.3ms / average = 0.7ms, wait = 4998.7ms

clients = 100
load = 1.4% / average = 0.5%, late = 2.9 ms
broadcast = 0.7ms / average = 0.7ms, wait = 4999.3ms

clients = 100
load = 0.0% / average = 0.4%, late = 8.2 ms
broadcast = 2.9ms / average = 0.9ms, wait = 4997.1ms

I'm currently working on a new design based on @cjerdonek idea.

I found a way to make the pattern proposed @lgrahl here reliable i.e. not lose messages even when not all clients have time to run between two events.
I came up with this.
4. Pubsub
Connection handler
class PubSub:
    def __init__(self):
        self.waiter = asyncio.Future()

    def publish(self, value):
        waiter, self.waiter = self.waiter, asyncio.Future()
        waiter.set_result((value, self.waiter))

    async def subscribe(self):
        waiter = self.waiter
        while True:
            value, waiter = await waiter
            yield value

    __aiter__ = subscribe

PUBSUB = PubSub()

async def handler(websocket, path):
    async for message in PUBSUB:
        await websocket.send(message)

Broadcast task
            ...
            PUBSUB.publish(message)
            ...

The load is ~70%, similar to the queue-based method:

This also implements the idea proposed by @cjerdonek:

have a single queue

The queue is modeled a singly-linked list of Futures.

with each connection storing its relative position in the queue (the item it has consumed up to)

The position is stored by the asynchronous iterator: it holds a reference to the Future for the current position.

And you can remove elements from the queue when either (1) all connections have read that element

The garbage collector takes care of this so I don't have to!

or (2) you could have some expiration logic where an item will only stay in the queue for a certain amount of time, etc, whether or not all connections have read that item.

Not implemented :-)

@aaugustin That's a nice solution! They way your PUBSUB method works is similar to asyncio.Queue. That's why PUBSUB gets same results as queue method. I was thinking about reduce yielding control to the event loop. I made a subclass of WebSocketServerProtocol and I've added a new method to it :
    def try_to_push(self, framed_message: bytes) -> bool:
        if self.state is WSState.OPEN and not self.transfer_data_task.done():
            if self._paused:
                return False
            else:
                self.transport.write(framed_message)
                return True
        return False

This method pushes data to the buffer if buffer is below the low watermark. I've made a loop like this in my server :
    async def send_loop(self):
        while not self.finished:
            for client in self.clients:
                if client.buffer:
                    if client.con.try_to_push(client.buffer):
                        client.buffer = b""
            await asyncio.sleep(0.1)

Which iterates over all connected clients every 0.1 seconds and tries to push data to their buffer. In this method there's no need to one Queue / Task / Future per client. I know this increases latency in first place, But according to my benchmarks it also reduces cpu load and gives freedom to developer to change the send_loop delay freely. I will not publish the full code until I am sure it works in production. But I like others to say their opinions about this method.

Thanks everyone for the discussion! It went much deeper than I expected :-) I wrote down a summary in a document that I am going to commit now.


New SKOS Concept(scheme)s for automated va

This issue describes SKOS CS (or missing Concepts in existing Schemes) additionally needed to provide for the VA process. The proposed method of solution is to provide the hiatus per comment entry, and provide the commit solving the hiatus in the reply to that comment.
Examples:

MISSING Scheme - for Mandatory Specification Sets for ERTMS. Propose name: ETCSMandatorySpecificationsSet.
MISSING Scheme - for Type of VA and VR Authorisation Cases . No proposed name.
MISSING Concept - Concept HistoricalAuthorisationCase in scheme treating the types of authorisation cases for vehicle authorisations. The concept must be used for 'virtual' authorisation cases' for which no link to OSS can be recreated. The vehicle type's area of use and authorisation process is then linked through that authorisation case.

[!IMPORTANT]
Please fork and provide the missing Schemes under the appropriate folder in the repo.

MISSING SCHEME - for the ERATV VehicleType Registration Methods. No proposed name.
Source: ERATV - 1.0 General Information, Registration method, which depends on the legal framework used to authorise the type.
Values to be provided, preferably grouped per legal baseline in which they exist:

New Type of a registered Platform
New Type
New Version of a registered Type Directive 2008/57/EC
etc ... (asked internally on 2024/07/09).

MISSING SCHEME - - for Mandatory Specification Sets for ERTMS. Propose name: ETCSMandatorySpecificationsSet.
Source: CCS EC Certificates must register this value (NBRail requirement).
Proposed values are in SKOS CS.

IMPROVE SCHEME - for the 5 types of Vehicle(Type) Authorisation Cases, some of which are allowed to be combined, the following CS exists VehicleAuthorisationCase. The concepts must be used for instances of era:VAAuthorisationCase, a subclassof vpa:Case, which provides this detail under vpa:permissionType.
Improvement needed: the allowed combinations must be SHACL-validated.

MISSING SCHEME - for the CLD type, as foreseen with property dcterms:type.  Propose name CLDTypes.
Proposed modelling

The intermediate statements could be grouped in a separate skos:Collection
The EC Certificates (not the QMSA) could be grouped in a disjoint collection.
Each ISV-member of that Collection could be skos:narrower than its counterpart member, as the skope it covers is legally speaking narrower.

Example
:CLDTypes a skos:ConceptScheme ;
    dct:title "Types of CLDs, NoBo's can issue."@en .

:CertificateTypes a skos:Collection ;
    skos:member :ECTypeExaminationCertificate , :ECDesignExaminationCertificate ; ECCertificateOfVerification ; ... .

:ISVTypes a skos:Collection ;
   skos:member  :ECTypeExaminationISV81 ,  :ECDesignExaminationISV82, :ISV86 , ... .

:ECTypeExaminationCertificate
    skos:inScheme :CLDTypes ;
    skos:prefLabel "EC Type Examination Certificate" ;
    skos:notation "1. (B,CB, SB)" ;
    skos:broader :ECTypeExaminationISV81 .

:ISV86
    skos:inScheme :CLDTypes ;
    skos:prefLabel "Intermediate Statement of Verification" ;
    skos:notation "8.6. (SD, SF, SG, SH1, SH2)" ;
    skos:narrower ECCertificateOfVerification .

Values
Rail Notified Bodies may issue CLDs of the following types:
1 EC Type Examination Certificate (B,CB, SB)
2 EC Design Examination Certificate (CH1, H2, SH1, SH2)
4 Quality Management System Approval (CD, CH, CH1, D, SD, H1, H2, SH1, SH2)*
5 EC Certificate of Conformity (A1, CA1, CA2, CF, F)
6. EC Certificate of Verification (SD, SF, SG, SH1, SH2)
7 EC Certificate of Suitability for Use (CV, V)**
8.1. Intermediate Statement of Verification – EC Type Examination (SB)
8.2. Intermediate Statement of Verification – EC Design Examination (SH1, SH2)
8.4. Intermediate Statement of Verification – Quality Management System Approval (SD, SH1, SH2)
8.6. Intermediate Statement of Verification (SD, SF, SG, SH1, SH2)

(The Modules that lead to the indicated CLD types are given in brackets.)

Source:  RFU-STR-001 (NBRail)


Support Request 2

Received this request from a place trying to embed.  Can we help from this?  Or do I need to get more?

Hi the code we downloaded isn’t running when we embed it.
Any help for that?
This is the code:

This is the page:
http://www.liunaforhillary.org/get-out-and-vote

Updated. A Javascript library on an embedding page was causing conflicts with our code. We added an extra wrapper.


Added fancy buttons

Hi there!
I've added few fancy styles :)

Great Work!
Thanks for Contribution!

Great Work!
Thanks for Contribution!


'Ctrl+S/Ctrl+Shift+S/F5' do not take effect under query builder

Storage Explorer Version: 1.29.0-dev
Build Number: 20230406.3
Branch: main
Platform/OS: Windows 10/Linux Ubuntu 22.04/MacOS Ventura 13.3 (Apple M1 Pro)
Architecture: ia32/x64
How Found: From running test cases
Regression From: Previous release (1.28.1)
Steps to Reproduce

Expand one storage account -> Tables.
Create a table -> Click 'Query'.
Press 'Ctrl+S'.
Check whether there is a system dialog.

Expected Experience
There is a system dialog.

Actual Experience
No system dialog pops up.

@v-xianya The system dialog will only pop up if no save file has been selected before. I'm assuming this is the first save?

@craxal Yes, it is the first save.

Verified this issue on the build main 20230413.5. Reproduce. So, we are going to reopen it and verify it on a new build.

There was a bad package update. Published a new version.

Verified this issue on the build main 20230414.2. Fixed.


⚠ Bofrost.lu has degraded performance

In 9e1cb08, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1116 ms

Resolved: Bofrost.lu performance has improved in a409a4e after 13 minutes.


[release-1.8] Fix HTTPs on HTTP port passthrough

This is an automated cherry-pick of #28166

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-multicluster-k8s-tests_istio_release-1.8
7bb12332f3e36775e0f8be7965da005b4f97b99b
link
/test integ-multicluster-k8s-tests_istio_release-1.8

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-multicluster-k8s-tests_istio_release-1.8


Reorganize CLI code

Is your feature request related to a specific problem?
The CLI code doesn't currently really have a clear structure as far as the files in commands/. Refactor it to be one file per command group to logically match the CLI itself

moving to done


Custom tasks modeling

What does this PR do?

This PR adds Custom tasks modeling through a new class OVModelForCustomTasks which allows exporting (through custom export config) and using those models.
Fixes #611
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[x] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@echarlaix I added an openvino variant of bert-with-pooler in IlyasMoutawwakil/sbert-all-MiniLM-L6-v2-with-pooler and added a custom export test that creates it


[fix-bug] image with same URL should not add twice more reliable id image tag fix

EmbededImage add same URL how many times it exist in HTML file that make unnecessary big file and some other problem. this PR download and add image to epub just once and reference all of same url image to the same file inside epub.

not download image twice
better id for images in opf file
fix content of image tag inside HTML to show correctly in eBooks reader.

@fmartingr my problem with update unit test is each time we generate new id so how should i test that in unit test. how can i get that from run to check at the end of test and if there same or not in final file?

Maybe you can use a v5 UUID? Instead of creating a random UUID, you can create one derivated from the filename:
// You need a namespace for V5 uuids, we can generate one based on the project URL
// Some namespaces are predefined in the UUIDs RFC, like the one used for URLs
namespace := uuid.NewV5(uuid.NameSpaceURL, "github.com/go-shiori/go-epub")
fileIdentifier := uuid.NewV5(namespace, filename)

More info: RFC 4122 - A Universally Unique IDentifier (UUID) URN Namespace - Some Name Space IDs

@fmartingr 3838225 is it what you want?
do you have any idea how should i fix unit test?

I understood that the problem with the test was that you were creating random UUIDs for images. If you change that to use deterministic UUIDs (uuidv5) you can generate those at the end of your tests and the result will be the same UUID.
I don't see any test in this PR, is there any code I can check?

you need to modify the test if you modify the ID generation.
Merge latest changes from master into this branch and let's check if theres a problem with #4

now you can see that test in CI but yet i don't have idea how should i fix that.

Since you are modifying the way IDs are created, you need to edit the test here: https://github.com/go-shiori/go-epub/blob/main/epub_test.go#L845-L851
That raises the question. Should we store the files with the same filename as the ID? images/{uuid}.extension ?

i change that unit test:

get any idUUID format in manifest
count them and check with what we expected
be sure nav is exist in manifest

not prefect but it is better than noting.

I'm thinking that since the files are hardcoded, you can hardcode the IDs as well, since they are deterministic.
It would also be interesting to check if a duplicate <item id="x"> is allowed by the EPUB spec.

@fmartingr i back to old method just replace with new id.
about duplicate id. when we check that with fix variable if any different show up duplicate similar etc this test faild (current method).
do we need check duplication separately?

I was thinking on a separate test that adds the same image twice, to see how the generation behaves and confirm that everything works as expected. Do you think this would be useful?


Handle port numbers in host header

From: https://github.com/mitmproxy/netlib/pull/121
Also incorporates the comment by @mhils

Does this affect this part in any way? Or should it?
https://github.com/mitmproxy/mitmproxy/blob/be02dd105b7803b7b2b6942f9d254539dfd6ba26/mitmproxy/mitmproxy/protocol/http2.py#L297-L320

I don't think so. There's no host header in HTTP2, so this should not affect that part. Looking at the code though, authority.partition(':') may fail for IPv6 (see how we parse the header up here), but that is an unrelated issue.

Tests are failing :(

Thanks! :smiley:


Aogonek missing anchors?

In a recent project for me, @colophon-foundry had to add in an anchor manually for the Aogonek - it wasn't automatically added as expected, suggesting its missing from the default GlyphsData.xml

Why do you need an anchor in the Aogonek?

To prevent mis-rendering the character.

So what anchors would be needed and in what contexts?
Are there any cases where there would another mark below? So a top anchor should be sufficient?


Enhancement: Allow export of genesets to various file formats (.csv, .gmt, etc.)

GMT (Gene Matrix Transposed) files  is a standard format used by gene set enrichment analysis software such as GSEA.
The GMT format is tab-delimited with one geneset per row containing:  geneset name, followed by a description, then a list of tab-separated gene IDS:
Name (tab) Description (tab) Gene (tab) Gene (tab)
Perhaps it would be useful for users to be able to export gene sets in this format with their identifiers of choice.

Other potentially useful formats are:

.csv
.tsv
.xml
.json
.txt
.gmx

This feature is addressed on the frontend in this PR: https://github.com/biothings/mygeneset.info-website/pull/21
On the backend, Biothings allows response formatting in several file formats: json, yaml, html, msgpack (code)
Example: http://mygeneset.info/v1/query?q=_id:GO_0004568_9606&format=yaml
However, writing custom parsers for csv, tsv, and gmt file formats would likely be very specific to mygeneset, and not apply to the other biothings APIs.

This is already implemented in the frontend. I don't think we have plans to do so in the backend.


docs: recommendation APIM enable caching to reduce latency

Overview/Summary
APRL recommendation to improve latency with caching for APIM
Related Issues/Work Items
https://dev.azure.com/CSUSolEng/Accelerator - AI/_sprints/taskboard/Accelerator - AI Team/Accelerator - AI/December '24?workitem=38369
Breaking Changes

Recommendation added APIM

As part of this pull request I have

[X] Read the Contribution Guide and ensured this PR is compliant with the guide
[X] Checked for duplicate Pull Requests
[X] Associated it with relevant GitHub Issues or ADO Work Items (Internal Only)
[X] Ensured my code/branch is up-to-date with the latest changes in the main branch
[X] Ensured PR tests are passing
[X] Performed testing and provided evidence (e.g. screenshot of output) for any changes associated to ARG queries
[X] Updated relevant and associated documentation (e.g. Contribution Guide, Docs etc.)

@ejhenry added kql file

@ejhenry - The new recommendation looks good. Only comment is to have the impact as Medium instead of High.


Migration in Example Failing with Syntax Error

honeydew_fields seems to be generating invalid SQL. In the example project I get a postgres syntax error when I run ecto.migrate. I'm using postgres 10.3 but none of this has changed recently so I'm a bit mystified how you didn't have the same result.
The reason for the failure is because the syntax of extract is wrong, it should be EXTRACT('milisecond' from NOW())).  This is generated fragment:
(EXTRACT('millisecond', (CAST('1994-03-26 04:20:00' AS TIMESTAMP))) + CAST((CAST('1994-03-26 04:20:00' AS TIMESTAMP)) AS  INT) * 1000)
I fixed that in the macro but then it complains that datetime cannot be cast to an int - the second part. I think maybe what you are after here with the cast is implemented in the epoch function.
[] ~/repos/elixir/honeydew/examples/ecto_poll_queue <master> ✗ mix ecto.create
Compiling 6 files (.ex)
Generated ecto_poll_queue_example app
The database for EctoPollQueueExample.Repo has been created
[] ~/repos/elixir/honeydew/examples/ecto_poll_queue <master> ✗ mix ecto.migrate

17:53:01.570 [info]  == Running EctoPollQueueExample.Repo.Migrations.CreatePhotosAndUsers.change/0 forward

17:53:01.572 [info]  create table photos
** (Postgrex.Error) ERROR 42601 (syntax_error): syntax error at or near ","
    (ecto) lib/ecto/adapters/sql.ex:200: Ecto.Adapters.SQL.query!/5
    (ecto) lib/ecto/adapters/postgres.ex:96: anonymous fn/4 in Ecto.Adapters.Postgres.execute_ddl/3
    (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/adapters/postgres.ex:96: Ecto.Adapters.Postgres.execute_ddl/3
    (ecto) lib/ecto/migration/runner.ex:104: anonymous fn/2 in Ecto.Migration.Runner.flush/0
    (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/migration/runner.ex:102: Ecto.Migration.Runner.flush/0
    (stdlib) timer.erl:181: :timer.tc/2

I guess your tests are using cockroachdb, is why they passed?  Cockroach is only compatible at the protocol level, it is not source or feature compatible with postgres. For some reason in the case of extract function they chose not to even be compatible with ANSI SQL.

just pushed proper postgres support in https://github.com/koudelka/honeydew/commit/8134472246711d7bc3f5704620ddca14e4788f09 on master, try it out and let me know how it goes. :)

@koudelka It works, thanks! Glad to have this in the main repository already.
I have a couple of suggestions based on the path I was headed down.
The SQL in your Postgres module looks like pure ANSI, no special Postgres features. I'd call the module "ANSI". I'd also suggest changing the option you pass to honeydew_fields as dialect rather than database. Dialects now would be :cockroach and :ansi, in the future maybe someone will add :cql for Cassandra etc.
Second, you could have a single example project support multiple database adapters and run the test suite on both of them by using a separate MIX_ENV. I had started to make a separate MIX_ENV=pg_test, which changed the adapter - you could also use it in the migration to select the database/dialect depending on the value of Mix.env. I didn't get that far since you mentioned you were working on this, but the first step of the mix.config I did here: https://github.com/smartmetals/honeydew/commit/e0edef042dfe4b951c592ba72387cd2c87cd6609
If you take both these suggestions adding support for Maria is completely trivial I think, it would just be another MIX_ENV for test and I don't think would require any different SQL code but I didn't try it yet.

Can you see a reason to want to select which dialect is used, outside of needing to manually select it when the adapter is ambiguous?
Thanks for the MIX_ENV idea! I really didn't like the idea of duplicating example project. :)

@koudelka No, I think if the macros can get access to the applications' Repo configuration you could save the user the trouble, but I think also if a single ANSI dialect will cover Postgres and MySQL (as written it would work with SQL Server and Oracle too, I think) you can just default to that, which will cover 95% of users.

combined the example into a single project, thanks again for the suggestion. :)


CLOUDSTACK-8818: Use MySQL native connector with Python

MySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/

Pylint result:
Global evaluation
-----------------
Your code has been rated at 6.48/10 (previous run: 6.48/10, +0.00)

I also wrote a small test script:
from db import Database

db = Database(username='root', password='XXXX')

try:
    db.testConnection()
    print("Succes!")
except:
    raise

The result:
wido@wido-laptop:~/repos/cloudstack/python/lib/cloudutils$ python smallthin.py
Succes!
wido@wido-laptop:~/repos/cloudstack/python/lib/cloudutils$

"Apache CloudStack Server" has failed says Jenkins. I don't see a failed test though?

@wido
Running org.apache.cloudstack.networkoffering.CreateNetworkOfferingTest

A fatal error has been detected by the Java Runtime Environment:

SUREFIRE-859: #  SIGSEGV (0xb) at pc=0xadc1552c, pid=22362, tid=4136426304

JRE version: 7.0_25-b15
Java VM: Java HotSpot(TM) Server VM (23.25-b01 mixed mode linux-x86 )
Problematic frame:
C  [libnet.so+0x352c]  _init+0x620

SUREFIRE-859: # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

An error report file with more information is saved as:
/home/jenkins/jenkins-slave/workspace/cloudstack-pull-analysis/server/hs_err_pid22362.log

SUREFIRE-859: # If you would like to submit a bug report, please visit:
http://bugreport.sun.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.

also see INFRA-10703 and vote for it

@DaanHoogland Thanks! Now, that failure has nothing to do with this Python code :)

LGTM

@borisroman @DaanHoogland Anybody else? :)

code lgtm but i have really no time to test this anytime soon

Can we already merge this one?

I need one more LGTM for this one and I would like to run CI to make sure nothing breaks.  Otherwise I think this is in a good place.

@swill I don't exactly know how to test this with CI. I verified it with the test script I created.
Don't know how we can automate this in CI.

@wido we'll need to fix codebase wide mysqldb usage in python scripts and build both rpms/deb to show installation, setting up of db and setting up mgmt server works out of the box. Other than this, I'll also need to fix a python based migration script used in dynamic roles PR that also uses the current mysqldb. We'll also need to fix travis to install the new dependency and fix marvin's dbclient to use it instead of mysqldb.

@wido I just want to verify that the standard CI run does not have any issues.  However, @bhaisaab does bring up some good points.  We should try to fix this everywhere if we can.  I will try to do some greping today to see if I can find everywhere this is used.

Ok, leave this PR open or closed?
Keep in mind, the API of both packages is rather similar. Just the connection creating is different. Atferwards it is getting a cursor and execute.

Let's leave this open for now. I want to understand what is needed to fix this project wide.

@swill @rhtyd: I looked through all the code and fixed it. Is was just two more scripts.
wido@wido-desktop:~/repos/cloudstack$ grep -R "MySQLdb" *
wido@wido-desktop:~/repos/cloudstack$
API-wise the syntax is the same between MySQLdb and this native connector.

@wido that is true, so will simply fixing the imports and usage work?
I found using git grep MySQLdb:
client/bindir/cloud-update-xenserver-licenses.in:import MySQLdb
client/bindir/cloud-update-xenserver-licenses.in:       conn = MySQLdb.connect(host=host,user=username,passwd=password)
python/lib/cloudutils/db.py:import MySQLdb
python/lib/cloudutils/db.py:                txn = MySQLdb.Connect(host=self.host, user=self.username,
python/lib/cloudutils/db.py:                txn = MySQLdb.Connect(host=self.host, user=self.username,
python/lib/cloudutils/db.py:                db = MySQLdb.Connect(host=self.host, user=self.username,
python/lib/cloudutils/db.py:                db = MySQLdb.Connect(host=self.host, user=self.username,
setup/bindir/cloud-migrate-databases.in:import MySQLdb
setup/bindir/cloud-migrate-databases.in:                self.conn = MySQLdb.connect(host=self.host,

@rhtyd I grepped after my modifications :)
Yes, the API between MySQLdb and mysql.connector is the same. The only difference is the import and the connection creation.
Afterwards you can get a cursor and execute queries.
When you do more advanced/complex things there are API differences, but that's not being used by CloudStack.

So are we happy with the state of this?  How should I adjust my CI to validate this fix?
I currently have:
yum -y install maven tomcat mkisofs python-paramiko jakarta-commons-daemon-jsvc jsvc ws-commons-util genisoimage gcc python MySQL-python openssh-clients wget git python-ecdsa bzip2 python-setuptools mariadb-server mariadb python-devel vim nfs-utils screen setroubleshoot openssh-askpass java-1.7.0-openjdk-devel.x86_64 rpm-build rubygems nc
yum -y install http://mirror.karneval.cz/pub/linux/fedora/epel/epel-release-latest-7.noarch.rpm
yum --enablerepo=epel -y install sshpass mariadb mysql-connector-python

and when marvin is installed:
pip install --upgrade tools/marvin/dist/Marvin-*.tar.gz --allow-external mysql-connector-python

But --allow-external has been deprecated, so I don't actually think that does anything anymore...

@swill for the CI you can install using pip with: (this is also in tools/travis/install.sh)
pip install --upgrade --egg http://cdn.mysql.com/Downloads/Connector-Python/mysql-connector-python-2.0.4.zip#md5=3df394d89300db95163f17c843ef49df

I see this in the output when Marvin is installed.
Requirement already up-to-date: mysql-connector-python>=1.1.6

If I check the version I have it is:
# pip freeze | grep mysql-connector-python
mysql-connector-python==1.1.6

Is that appropriate or should we be updating the Marvin requirements?

Running CI now.  Will have results later tonight or tomorrow...

CI RESULTS
Tests Run: 85
  Skipped: 0
   Failed: 1
   Errors: 0

Summary of the problem(s):
FAIL: test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 265, in test_03_vpc_privategw_restart_vpc_cleanup
    self.performVPCTests(vpc_off, True)
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 331, in performVPCTests
    self.check_pvt_gw_connectivity(vm1, public_ip_1, vm2.nic[0].ipaddress)
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
    "Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_G9N9AW/results.txt

Associated Uploads
/tmp/MarvinLogs/DeployDataCenter__Apr_28_2016_22_47_49_Z93Y1T:

dc_entries.obj
failed_plus_exceptions.txt
runinfo.txt

/tmp/MarvinLogs/test_network_G9N9AW:

failed_plus_exceptions.txt
results.txt
runinfo.txt

/tmp/MarvinLogs/test_vpc_routers_V1YYOR:

failed_plus_exceptions.txt
results.txt
runinfo.txt

Uploads will be available until 2016-06-29 02:00:00 +0200 CEST
Comment created by upr comment.

CI has passed.  The only issue we see is a connectivity issue I sometimes see in my environment.

LGTM, let's merge if CI has passed; though we need to check installable packages wrt all sort of scripts using it (like cloud-setup-databases and others etc.)

tag:mergeready

@wido can you rebase for me?  We apparently have merge conflicts now.  Thanks...

Done @rhtyd !

Could it be possible the file debian/control should be updated to depend on mysql-connector-python  deb package instead of python-mysql.connector ?
I have this PR ready for the Debian side:
https://github.com/pdion891/cloudstack/blob/b6ca215b17b7d65fd08cf567c8b584cc76aa7e52/debian/control
I'm just not sure if it's wanted to not have Debian updated to use the new driver?
Regards,

The package is called python-mysql.connector in Ubuntu and Debian: http://packages.ubuntu.com/xenial/python-mysql.connector
The mysql-connector-python package name is CentOS/RHEL.

what @pdion891 says is correct. I just tried building deb packages on ubuntu 12:04
Even after installing http://dev.mysql.com/downloads/connector/python/ its complaining about python mysql connector

dpkg-buildpackage -uc -us
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): -Wl,-Bsymbolic-functions -Wl,-z,relro
dpkg-buildpackage: source package cloudstack
dpkg-buildpackage: source version 4.10.0-SNAPSHOT
dpkg-buildpackage: source changed by the Apache CloudStack project<EMAIL_ADDRESS>dpkg-buildpackage: host architecture amd64
dpkg-source --before-build cloudstack
dpkg-checkbuilddeps: Unmet build dependencies: python-mysql.connector
dpkg-buildpackage: warning: Build dependencies/conflicts unsatisfied; aborting.
dpkg-buildpackage: warning: (Use -d flag to override.)

after editing debian/control using :%s/python-mysql.connector/mysql-connector-python/g, it worked fine

Why is t hat required? The package is also available for 12.04: http://packages.ubuntu.com/precise/python-mysql.connector

Are you saying it's not like CentOS which require to install mysq-connector-python from mysql repo [1], on ubuntu the package would have not been renamed ?
[1] https://github.com/cloudops/cloudstack-docs-rn/blob/4.9/source/upgrade/_mysql_connector.rst

No, it isn't indeed.
apt-get dist-upgrade
That will show you that new dependencies are required for install and it will download them. No manual need for downloading a .deb file and installing it.

ok, thanks, I will make sure the doc reflect that.
thanks!


Fix wrong supergroup migration handling

Fixes #36

If it is ok with you I would like to merge this.
CC: @LukasKalbertodt

so?

We are rewritting everything from scratch, so i'm closing it.


Add run tests section

Current coverage is 96.53% (diff: 100%)

Merging #11 into master will not change coverage

@@             master        #11   diff @@
==========================================
  Files             2          2
  Lines           173        173
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits            167        167
  Misses            6          6
  Partials          0          0

Powered by Codecov. Last update 5401f42...19e62ff

Nothing before Pkg.test("DBAPI") is necessary, and some of it is wrong (DBI doesn't need to be installed and cloning it wouldn't cause local filesystem clone).

 Corrected now.


Herbaria #Records reversed

Fixes a bug in Herbaria by #Records reversed

Delivers https://www.pivotaltracker.com/story/show/176833509
The bug was that sorting links were not displayed.
The links were not displayed because Query#num_results returned the wrong value (1) for #Records reversed. (It also returned the wrong value for #Records not reversed.

Suggested Manual Test

http://localhost:3000/herbarium or http://localhost:3000/herbaria?flavor=all
Sort by: #Records
Sort by: Reverse Order
Expected Result: Page includes sort tabs

This works, but seems like a kluge.
It feels like the correction should be made somewhere in one of the Query Herbaria modules.
But I can't figure out where or how.

Coverage remained the same at 92.984% when pulling b25bcec5adde63af07eaf87ac332cf1cdf28ac7d on herbaria-records-reversed-176833509 into 4c587f656a5cb30aa67abc28dcd5fad21d437855 on master.

This is low priority. It should probably wait until @pellaea can eyeball it.

You're right, Joe, I don't like this solution.  The underlying problem is the interaction of selecting COUNT(*) from a query which has a GROUP BY clause in it.  Ultimately, this was a bit of sloppiness / laziness on my part.  I knew that it was unnecessary to include the sorting criteria which simply counting the number of results, but I thought (incorrectly) that it would be harmless and therefore I left the code simple.  But of course, I hadn't anticipated a complex case like sorting by number of herbarium_records when I first wrote Query.  So now I should finally deal with it correctly.  I think it will be a simple fix.  Let me play with it for a minute, see if the naive approach works right...  More in a minute...

You're right, Joe, I don't like this solution.  The underlying problem is the interaction of selecting COUNT(*) from a query which has a GROUP BY clause in it.  Ultimately, this was a bit of sloppiness / laziness on my part.  I knew that it was unnecessary to include the sorting criteria which simply counting the number of results, but I thought (incorrectly) that it would be harmless and therefore I left the code simple.  But of course, I hadn't anticipated a complex case like sorting by number of herbarium_records when I first wrote Query.  So now I should finally deal with it correctly.  I think it will be a simple fix.  Let me play with it for a minute, see if the naive approach works right...  More in a minute...
[By the way, something similar could potentially occur when sorting images by name... at least it also introduces a GROUP BY clause, so should therefore also mess up the results of the COUNT(*) query.]

Jason Hollinger<EMAIL_ADDRESS>Mon, Aug 9, 2021 at 11:56 AM
To: mo-developers<EMAIL_ADDRESS>Joseph Cohen<EMAIL_ADDRESS>Sorry, Joe, I can't get github to work for me any more today, so I'll just write it in a comment. Please revert your changes to num_results() in high_level_queries.rb, and instead use this fix:

 # Number of results the query returns.
 def num_results(args = {})
   @num_results ||=
     if @result_ids
       @result_ids.count
     else
       # Explicitly disable GROUP BY and ORDER clauses for the purposes of
       # simply counting the number of results.  This is important because
       # GROUP BY in particular will mess up the COUNT(*) spec.  Passing in
       # empty strings for group and order will tell it to explicitly override
       # anything in self.group and self.order.
       rows = select_rows(args.merge(select: "COUNT(*)", group: "", order: ""))
       begin
         rows[0][0].to_i
       rescue StandardError
         0
       end
     end
 end

I'm still running the tests, but it looks like it does what it is supposed to do. And this fix should guard against similar gotchas in the future (this is the "correct" solution I should have been doing all along)... within the context, of course, of a ginormous complex class like Query which we would never write nowadays!!. And further, it should improve performance: it will no longer unnecessarily sort queries which are simply requesting the number of results.
Uh oh, there is now one failure associated with listing images of an advanced search. I'll look into it. Hopefully this just caught another bug we hadn't noticed and that's all that's going on..

Jason Hollinger<EMAIL_ADDRESS>Mon, Aug 9, 2021 at 12:22 PM
To: mo-developers<EMAIL_ADDRESS>Joseph Cohen<EMAIL_ADDRESS>Okay, no worries, the fix just interacted badly with an earlier kludge in the image advanced search query (intended to make advanced image searches with nonempty "content" field not crash the server!) The solution is trivial. Change execute_content_search() in app/classes/query/image_advanced_search.rb to:

  def execute_content_search(args)
+    # [Sorry, yes, this is a mess.  But I don't expect this type of search
+    # to survive much longer.  Image searches are in desperate need of
+    # critical revision for performance concerns, anyway. -JPH 20210809]
    args2 = args.dup
    args2.delete(:select)
+    args2.delete(:order)
+    args2.delete(:group)
    params2 = params.dup
    params2.delete(:by)


Warning - CreateWebHostBuilder -

Hey MS,
You have forgotten to change - CreateWebHostBuilder - to - CreateHostBuilder - (I guess ?) during "Initialize DB with test data".
It's not really important because Visual Studio auto-generate few lines after the right method, but for a real beginner it can be a little weird.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3ad06677-5842-c1aa-153f-7993a9660dfa
Version Independent ID: 8daf79df-eac4-358b-e6f0-6def45bd54da
Content: Tutorial: Get started with EF Core in an ASP.NET MVC web app
Content Source: aspnetcore/data/ef-mvc/intro.md
Product: aspnet-core
Technology: aspnetcore-data
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Per the tutorial
This tutorial has not been updated to ASP.NET Core 3.0.


Datepicker errordescription

Is your feature request related to a problem? Please describe.
Datepicker component has a state field just like the other form inputs.
However when using it to to show validation error with the error-state of the input it doesn't color red.
Describe the solution you'd like
Color the description red when state="error"
Describe alternatives you've considered
Naming should be the same everywhere to keep it consistent, Datepicker has description, Textfield has errorDescription.
Preferably all form input would have description and errorDescription and support both use cases this way.

This was added in v6.4.0


Implement Work Cell Materials with BOM Cross Reference

Work cell materials lists now need to be implemented, with transferrable BOM cross reference.

This is complete and in this branch.


Vec<T> in newtype doesn't impl ToHeader even though T and Vec do

Hi, first off, sample code:
$ cat Cargo.toml
[package]
name = "eh"
version = "0.1.0"
edition = "2021"

[dependencies]
poem = "1"
poem-openapi = "2"

$ cat src/main.rs
use poem_openapi::NewType;

#[derive(Clone, Debug, PartialEq, NewType)]
pub struct HexEncodedBytes(Vec<u8>);

fn main() {
    println!("Hello, world!");
}

When you try to compile this, you get the following error:
$ cargo check
    Checking eh v0.1.0 (/Users/dport/eh)
error[E0277]: the trait bound `Vec<u8>: ToHeader` is not satisfied
 --> src/main.rs:3:35
  |
3 | #[derive(Clone, Debug, PartialEq, NewType)]
  |                                   ^^^^^^^ the trait `ToHeader` is not implemented for `Vec<u8>`
  |
  = help: the following other types implement trait `ToHeader`:
            &T
            Arc<T>
            Box<T>
            HexEncodedBytes
            MaybeUndefined<T>
            Uri
            bool
            f32
          and 13 others
  = note: this error originates in the derive macro `NewType` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0277`.
error: could not compile `eh` due to previous error

This is sort of surprising to me, because I see that ToHeader is implemented both for u8 and Vec:

poem-openapi/src/types/external/vec.rs
poem-openapi/src/types/external/integers.rs

Could you help me understand why this doesn't work?
Thanks!

Okay looking closer, Vec does not actually implement ToHeader. I'm looking at adding it now but I realise perhaps this doesn't make sense to do, we'd have to come up with a custom way to represent the vec as ascii.
Any ideas @sunli829? I wonder how I'm meant to include Vec<u8> in my Object.

Okay I see in this example (https://github.com/poem-web/poem/blob/master/examples/openapi/upload/src/main.rs) that you can wrap it in Binary. However this still doesn't quite work.
Code:
use aptos_types::account_config::AccountResource;
use serde::{Deserialize, Serialize};
use poem_openapi::{Object as PoemObject, payload::Binary};

#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, PoemObject)]
pub struct AccountData {
    pub sequence_number: U64,
    pub authentication_key: Binary<Vec<u8>>,
}

Error:
error[E0277]: the trait bound `poem_openapi::payload::Binary<Vec<u8>>: ParseFromJSON` is not satisfied
  --> api/types/src/account.rs:10:59
   |
10 | #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, PoemObject)]
   |                                                           ^^^^^^^^^^ the trait `ParseFromJSON` is not implemented for `poem_openapi::payload::Binary<Vec<u8>>`

You should use types::Binary. 😂

Ahhh oops, okay that works, thanks!


template is="dom-if" throws if template contains comment

Given an dom-if template with a comment:
<template is="dom-if" if="{{fetchingMail}}">
  <!-- div class="layout vertical center-center" style="height:100%">
    <paper-spinner active></paper-spinner>
  </div -->
  <div id="refresh" class="fit layout vertical center-center">
    ...
  </div>
</div>
</template>

Polymer 1.0.5 throws an error trying to display: none it:

I worked a little bit on the dom-if yesterday. This error happens if the dom-if contains a node without style property (like text_node, comment_node). I'll update my pull request for this case. Please, see the PR #2027 .

Here is a simple reproduction (code here.
The _showHideChildren method in dom-if needs to correctly handle comment nodes - what would the desired behavior be for this situation?
I can file a PR if you like given guidance on the proper implementation here.

This is a dupe of #1786.
This PR will be merged this week: https://github.com/Polymer/polymer/pull/1992

Should I make a new PR to keep only the first commit of the PR #2027 who solves the issue #1725 ?


Transient failure to build libHSrts.a

(Note, if relevant, that I'm using GHC from 2 days ago but shakeing from right now.)
Error when running Shake build system:
* .build/stage1/rts/build/libHSrts.a
user error (Development.Shake.cmd, system command failed
Command: C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe q .build/stage1/rts/build/libHSrts.a @C:\Users\ndmit_000\AppData\Local\Programs\stack\i386-windows\msys2-20150512\tmp\extra-file-504133661-1694121724
Exit code: 1
Stderr:
C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe: creating .build/stage1/rts/build/libHSrts.a
C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe: unable to rename '.build/stage1/rts/build/libHSrts.a'; reason: No such file or directory

And checking that assertion:
$ ls .build/stage1/rts/build/*.a
.build/stage1/rts/build/libCffi-6.a  .build/stage1/rts/build/libHSrts_thr.a

Rerunning just that gave:
$ shake-build/build.bat --configure=--enable-tarballs-autodownload -j .build/stage1/rts/build/libHSrts.a
| Run configure (--enable-tarballs-autodownload)...
... seemed to almost be starting from scratch ...

Which was a bit weird. I complete configure. Why is it rerunning configure etc?
That run ended up failing with:
build: .build/stage0/compiler/package-data.mk: openFile: does not exist (No such file or directory)

But that file does exist...
So I did a clean and tried again. This time it worked. I've got failures to build libHSrts.a before, in roughly the same place, so I thought it worth raising.

I've never seen this libHSrts.a failure before. Will try to reproduce on my machine.

Why is it rerunning configure etc?

Because you asked it to with --configure=--enable-tarballs-autodownload. Remember: this is currently a direct command to the build system to rerun configure. It is supposed to be used only on the first run. The right way to build this file only is to run shake-build/build.bat -j .build/stage1/rts/build/libHSrts.a.

Yep, I figured out the configure thing, and ticketed that separate as #204. I'll do some soak test builds, clean/rebuild, and see what I get.

Not sure what's going on. Here is how I invoke Ar when @file is supported:
useAtFile :: FilePath -> [String] -> [String] -> Action ()
useAtFile path flagArgs fileArgs = withTempFile $ \tmp -> do
    writeFile' tmp $ unwords fileArgs
    cmd [path] flagArgs ('@' : tmp)

I may be running into the problem of lazy writeFile', but I don't understand how this leads to the reported errors. Looking at the error message, the command was
Command: C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe q .build/stage1/rts/build/libHSrts.a @C:\Users\ndmit_000\AppData\Local\Programs\stack\i386-windows\msys2-20150512\tmp\extra-file-504133661-1694121724

which seems fine to me.
I've no idea why Ar needs to do any intermediate renaming of .build/stage1/rts/build/libHSrts.a that is mentioned in stderr and appears to be the cause of the error.
Stderr:
C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe: creating .build/stage1/rts/build/libHSrts.a
C:/Neil/ghc/conf3/ghc/inplace/mingw/bin/ar.exe: unable to rename '.build/stage1/rts/build/libHSrts.a'; reason: No such file or directory

Maybe this is the way Ar complains about empty @file (due to lazy writeFile')?

writeFile is not lazy, only readFile, so that's not the problem. I'll keep soak testing and if it doesn't reproduce we can close as not being that common, or perhaps being an artefact of something else.

writeFile is not lazy, only readFile, so that's not the problem

Ah, indeed.
I failed to reproduce. I suspect that withTemp is somehow related, but can't quite figure it out.

GHC's temp file functions are buggy. Shake's have been stress tested a lot. But it's a tricky race condition prone problem, so I should test some more.

GHC's temp file functions are buggy.

This is quite concerning to hear. @ndmitchell Could you open a GHC bug for this if you have a test case (or even a rough idea of where the issue may be)?

My memory is that I already did raise a bug, but I'll double check...

https://ghc.haskell.org/trac/ghc/ticket/10731

Thanks @thomie, yes, that was what I was thinking of. Test harnesses and a fully implemented workaround, what more could you want :)

Got hit by this today, for the first time I think.
@ndmitchell Since we only use withTempFile a couple of times in the whole build system, perhaps, we can just avoid them at all and keep temporary files among build artefacts? A bit inelegant, but doesn't seem to be a big issue.

We are using Shake's temporary files, which are super robust, and in this case just keyed off an IORef in process so will never fail. It's unlikely to be that, or at least obviously and simply that...

Ah, I somehow assumed this issue is related to the GHC bug. Sorry for the noise. Still no idea how to properly fix this (apart from trying to get rid of temporary files).

If you rm the file then rebuild in a loop, does that trigger it? Without the ability to trigger hard to experiment with what is the culprit.

I've not seen this in a lot of soak testing. Shall we hopefully assume its gone?

Yep, let me close this.


Creating "duplicate" rows instead of list of strings in column

Hello!
I have what I hope isn't too hard or impossible to do. Essentially, I've got a column of serial numbers on my invoice table. Each row can have multiple serial numbers. This is done by doing a few joins and a "LinkingSetRelation". While that is cool and all, my troubles start when I try and enable a quick filter to search through the invoices by serial number. It's proving to be impossible, or at least extremely difficult to do this way because the serial numbers are stored as List.
My question is whether or not it'd be possible to get it so each row has has only one serial number, even if it means other information is duplicated. That way, I can enable a quick filter to search by serial number.

Let me know if I can provide more code or details.

UPDATE: I did figure out the correct Join and Expression to do what I wanted, which is great! The issue now is that some of my queries take too long.


add metric to compute fraction of max amplitude; change summary stati…

Changed the name of create_avg_dict to create_statistics_dict as this is a more accurate representation of this function
Added per-twitch-estimate of fraction-of-max-amplitude along with summary statistics of amplitude and fraction-of-max-amplitude

Also, I might have used the wrong branch here


Multiple filters can not be submitted on bs_smart_collection

If more than one filter is applied to bs_smart_collection, the filters are not submitted on ENTER as it's happening when only applying one filter

This is still an open issue and present in version 3+. Can I assign this to myself and work on it? @keithrowell @jonasjabari
I've actually developed a fix/workaround for this in a client project and can port it over to the gem. The W3C spec explains that this is natural behavior of forms:

Form Submission
...
...
When there is only one single-line text input field in a form, the user agent should accept Enter in that field as a request to submit the form.

A Stackoverflow question was also asked about this and the accepted answer directed to the above section of the W3C spec.

@aaron-contreras yes please assign it to yourself and have fun working on it :)


Move to CalVer

All Autobahn's will move to http://calver.org/ beginning with the next release. See also:

https://github.com/crossbario/autobahn-js/issues/284
https://github.com/crossbario/autobahn-python/issues/803

well, that has happened some time ago;)


Auto-registration support

Fix last change to support automatic registration and add some method built-in to do it.

Now support 3 ways to auto add handlers
By the initial mediator creation
services.AddMediator(<list of assemblies to scan>)
After mediator creation
services.AddHandler(<list of assemblies to scan>)
services.AddHandler(<filter function of types to include>, <list of assemblies to scan>)

LGTM 👍


Issue with initial Isotope layout

Playing with infinitescroll and Masonry/isotope, I have noticed an issue (other than the initial image overlap already reported). Masonry works well but I've found that if I use isotope.js instead of masonry.js , but initialize the grid with masonry code ( a coding accident) , it still works (!) but if I use  isotope.js  and initialize isotope as one should, the masonry layout does not layout properly until the grid is resized, as if imagesloaded  is not present, i.e., it loads in one column initially
(Not sure how masonry initialization works with isotope.js ?)
Mike
@jquerymac

Thanks for reporting this issue.

"It still works" because isotope.pkgd.js includes everything in masonry.pkgd.js. Isotope is a superset of Masonry.
When using the Masonry layout mode with Isotope, Masonry options columnWidth and gutter need to be set in a masonry object (this allows various layout modes to have similar option names but not overwrite one another).

 var $grid =  $('.grid').isotope({
  itemSelector: '.photo-item',
  masonry: {
    columnWidth: '.grid__col-sizer',
    gutter: '.grid__gutter-sizer',
  },
})

See demo https://codepen.io/desandro/pen/e9cb00062329d59e0fff54db7915d46d/

Thanks. Darn,  I knew that about isotope but had a brain fart!


az devops security group membership list --relationship memberof returns invalid json

Hi,
Using the membership list --relationship memberof option, but I believe the 'members' option does the same, I come across the issue that powershell cannot convert the output from json.
Code: $MemberOf = (az devops security group membership list --id $GroupId --relationship memberof) | ConvertFrom-Json
In our case I see three groups returned, but the $MemberOf only holds one object, not three. Even if I force the output in Json format.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 52ec4f1d-e2b6-2397-ea6f-d8ceb8a3b315
Version Independent ID: 58fbf7ef-2e9f-d56f-3e75-d37195202350
Content: az devops security group membership
Content Source: latest/docs-ref-autogen/ext/azure-devops/devops/security/group/membership.yml
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

devops

Hi, can you share screenshots of your issue for a detailed explanation.
you can try az devops security group membership list --id $GroupId --relationship memberof -o table to see how many groups is this group a member of.
What we get is a single json object containing information for all the 3 groups, which I guess is what you are seeing. If there is something else you wanted to tell, please reply back with some screenshots of the issue.
Thanks

Hi, see attached files. I have added the powershell statements I executed and replaced the most important GUIDs with text to avoid exposing too much details.
JSONFormat.zip

Same problem here.

@smunj Has there been any work on correcting this issue?


fix(freestyle-writer) - listdetail fix type errors in List.controller.ts

fix https://github.com/SAP/open-ux-tools/issues/896

Adjust packages/fiori-freestyle-writer/templates/listdetail/add/webapp/controller/List.controller.ts
Add test

Thanks all


Control size of lightbox image?

Hi.  The lightbox image is trying to fill my entire window, no matter the size of the image.
My code:
        <div         className="image"         style={{ width: "100px", display: "block", margin: "auto" }}       >         <ModalImage           small={displayImg}           medium={displayImg}           alt={description + " #" + number}         />       </div>
In my test case, the image in displayImg is a jpg of 500px x 276px .    I display the small image at 100px wide (styled within the div)  and want the image in the lightbox to be the full 500x276.
The small image displays fine, but the lightbox image blows up to the full size of the window.
Is there any way to control the image size in the lightbox?

Hi, I'm running into the same issue and haven't found a solution yet. In the demo, this doesn't seem to be an issue, but there I'm not seeing anything in the code that would control the size of the image in the lightbox.

Hi. The lightbox image is trying to fill my entire window, no matter the size of the image.
My code:
<div className="image" style={{ width: "100px", display: "block", margin: "auto" }} > <ModalImage small={displayImg} medium={displayImg} alt={description + " #" + number} /> </div>
In my test case, the image in displayImg is a jpg of 500px x 276px . I display the small image at 100px wide (styled within the div) and want the image in the lightbox to be the full 500x276.
The small image displays fine, but the lightbox image blows up to the full size of the window.
Is there any way to control the image size in the lightbox?

you can control max-width of image in css:
.__react_modal_image__modal_content {
img {
max-width: 900px;
}
}


antctl get network-policy output not very friendly

Describe the bug
antctl get network-policy outputs data in json by default, which is a little inconvenient to have an overall view of the data, the default format should be more human friendly.
However, even with -o table, the output is not friendly enough:
# antctl get network-policy -o table
name                                 rules                                                                                appliedToGroups
0044b144-8d49-4621-8cc3-9a4ddf2c35f9 [{"direction":"In","from":{"addressGroups":["32ef631b-6817-5a18-86eb-93f4abf0467c"]},"services":[{"port":"http","protocol":"TCP"}],"to":{}}] ["c4c59cfe-9160-5de5-a85b-01a58d11963e"]

# antctl get network-policy -o table
rules                                                                                appliedToGroups                          name
[{"direction":"In","from":{"addressGroups":["32ef631b-6817-5a18-86eb-93f4abf0467c"]},"services":[{"port":"http","protocol":"TCP"}],"to":{}}] ["c4c59cfe-9160-5de5-a85b-01a58d11963e"] 0044b144-8d49-4621-8cc3-9a4ddf2c35f9

# antctl get applied-to-group -o table
name                                 pods
c4c59cfe-9160-5de5-a85b-01a58d11963e [{"pod":{"name":"nginx-6db489d4b7-vgv7v","namespace":"default"}},{"pod":{"name":"nginx-6db489d4b7-bh74d","namespace":"default"}},{"pod":{"name":"nginx-6db489d4b7-m272t","namespace":"default"}},{"pod":{"name":"client1-5dfbb959dc-f5dp7","namespace":"default"}},{"pod":{"name":"nginx-6db489d4b7-57jfj","namespace":"default"}}]

# antctl get address-group -o table
name                                 pods
32ef631b-6817-5a18-86eb-93f4abf0467c [{"ip":"<IP_ADDRESS>"}]

The fields order changes randomly, it should be consistent and start from identifer.
There is no length limit for a line and multiple rules can be extremely long, it should have a limit and generate summary when there are too many contents.
Fields are still json format, it should be more human friendly.
Table header should be capitalized and meaningful.

To Reproduce
Steps to reproduce the behavior.
Expected
The following output will be more friendly to me:
# antctl get network-policy -o table
NAME                                 APPLIED-TO                                       RULES
0044b144-8d49-4621-8cc3-9a4ddf2c35f9 c4c59cfe-9160-5de5-a85b-01a58d11963e             2
32ef631b-6817-5a18-86eb-93f4abf0467c c4c59cfe-9160-5de5-a85b-01a58d11963e + 2 more... 2

# antctl get applied-to-group -o table
NAME                                 PODS
c4c59cfe-9160-5de5-a85b-01a58d11963e default/nginx-6db489d4b7-vgv7v,default/nginx-6db489d4b7-bh74d + 2 more...

# antctl get address-group -o table
NAME                                 PODS
c4c59cfe-9160-5de5-a85b-01a58d11963e <IP_ADDRESS>,<IP_ADDRESS>
32ef631b-6817-5a18-86eb-93f4abf0467c <IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS> + 4 more...

Actual behavior
See above
Versions:
Please provide the following information:

Antrea version (Docker image tag). v0.4.1
Kubernetes version (use kubectl version). If your Kubernetes components have different versions, please provide the version for all of them.
Container runtime: which runtime are you using (e.g. containerd, cri-o, docker) and which version are you using?
Linux kernel version on the Kubernetes Nodes (uname -r).
If you chose to compile the Open vSwitch kernel module manually instead of using the kernel module built into the Linux kernel, which version of the OVS kernel module are you using? Include the output of modinfo openvswitch for the Kubernetes Nodes.

Additional context

@ZhangYW18 please see if you can improve this.

I read the code involved, the above problems are applied to all commands with table output, not just 'antctl get network-policy'.  Do we need to modify the format for all commands?

We can use Golang tags in command response struct definition to deal with most of the issues.

Use tags to specify the order. Sort according to field names in the alphabet order in default.
Use tags to specify constraints (length, maximum contents) when defining the return struct of commands.
Modify the response types in /pkg/antctl/transform. For a specific response struct, we need to remain only the data we need to show. If there is a field whose type is a self-defined type, then still output json.
Same as problem 2, we can use tags to specify the title associated with a field. The tags do not need to be capitalized and we can process them together when printing the results.

We can use Golang tags in command response struct definition to deal with most of the issues.
For the 4 problems:

Use tags to specify the order. Sort according to field names in the alphabet order in default.
(1) Want the length of rows and columns varies for different commands and structs: Use tags to specify constraints (length, maximum contents) when defining the return struct of commands. Also, we can recognize the maximum length of elements as the maximum length of the column.
(2) Want the length of rows and columns to be a fixed number: Just control the number of spaces in the output.
Modify the response types in /pkg/antctl/transform. For a specific response struct, we need to remain only the data we need to show. If there is a field whose type is a self-defined type, then still output JSON.
Same as problem 2, we can use tags to specify the title associated with a field. The tags do not need to be capitalized and we can process them together when printing the results.

@tnqn please see if I can utilize the methods above.

@ZhangYW18 The solutions sound good to me.
For 2, how is it related to rows?
I prefer the column width is not fixed and not filled with spaces.
It should be short when the content is short but have a maximum length, after which it should be truncated. The maximum length can be fixed for all columns.

@ZhangYW18 The solutions sound good to me.
For 2, how is it related to rows?
I prefer the column width is not fixed and not filled with spaces.
It should be short when the content is short but have a maximum length, after which it should be truncated. The maximum length can be fixed for all columns.

This requirement can be solved automatically by https://github.com/jedib0t/go-pretty.
Also, for issue 4, a more concise and universal solution is to put on a map to store the titles which are needed to be converted when getting headers. This solution applies to all response structs with the same field names. It is not necessary to use tags.

I think the codes can be done within this week.


No mapping to the Request object

Describe the Bug
When using the Body Annotation the request body object will map 1:1 with the request body and no mapping occurs.
To Reproduce
Create an endpoint with the Body Annotation and see what request body reaches the object given.
Expected behavior
It should use the inverse mapping of the Response object mapping.
Screenshots / Logs
If applicable, add screenshots and/or logs to help explain your problem. You can use pastefy.ga to upload large text snippets.
Environment Information:

Java Language Level 11

Additional Context
@JulianFun123 Can confirm the error

Set naming policies from default to camel_case

Unfortunately, it seems like it didn't fix the issue.

At some point some solved magically.


mac-36 is dead.

Type of Request
bug
Infrastructure Environment
LUCI, Github, Cocoon scheduler, Autosubmit, etc...
What is happening?
mac-36 is dead.

Steps to reproduce
Step 1:
Step 2:
..
Step n:
Expected results
I expect to see X when Y is finished.

mac-36 is out of sync with Salt master, I can login remotely from my end. Ran command "salt-call state.apply" to re-sync with Salt master.
flutter@flutter-devicelab-mac-36 ~ % uptime
14:58  up 14:20, 4 users, load averages: 1.27 1.12 1.07


util: enable golang profiling

Add support for golang profiling.
Standard tools like go tool pprof and curl
work. example:
$ go tool pprof http://localhost:8080/debug/pprof/profile
$ go tool pprof http://localhost:8080/debug/pprof/heap
$ curl http://localhost:8080/debug/pprof/heap?debug=1
https://golang.org/pkg/net/http/pprof/ contains
more details about the pprof interface.
Updates: #1699
Signed-off-by: Yati Padia<EMAIL_ADDRESS>

/retest ci/centos/mini-e2e-helm/k8s-1.19

/retest ci/centos/mini-e2e-helm/k8s-1.20

/retest ci/centos/mini-e2e-helm/k8s-1.19

/retest ci/centos/mini-e2e-helm/k8s-1.20

/retest ci/centos/k8s-e2e-external-storage

/retest ci/centos/k8s-e2e-external-storage

/retest ci/centos/k8s-e2e-external-storage

/retest ci/centos/k8s-e2e-external-storage

/retest all

/retest ci/centos/k8s-e2e-external-storage/1.20

/retest ci/centos/k8s-e2e-external-storage/1.21

/retest all

/retest ci/centos/k8s-e2e-external-storage/1.20

/retest ci/centos/mini-e2e-helm/k8s-1.21

@yati1998 please rebase pr on top of devel for ci fix.


Fix methods without permissions

Added missed permission checks.
This PR depends on #537. Should be review after the dashboard modularization
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I've found 1 more TODO comment related to missing permission: https://github.com/nikpodsh/aws-dataall/blob/fix_methods_without_permissions/backend/dataall/modules/dataset_sharing/services/share_item_service.py#L175
Otherwise Lgtm!