libuv performance experiments (Network IO) - request feedback

This issue is opened to gather early feedback on an experiment within libuv loop to investigate improvement in network troughput by exploiting the operating system tunables in the TCP stack and leveraging them in the libuv.
Background:
Node.js Big data use cases (with / without streaming and piping) may involve large chunks of data xfer between endpoints. The current unit of data xfer in Node is 64 KB (see uv__read for example) and is hard-coded.
This may be good enough if the underlying TCP stack supports only windows of small size. Modern operating systems support larger TCP data windows. (ref: RFC 1323 TCP high performance extensions, https://tools.ietf.org/html/rfc7323).
Goal:
Investigate improvement in Node's network throughput by exploiting the operating system tunables in the TCP stack and leveraging them in the libuv.
Non-goal:

Improvements realized in the most common use case (single page webserver etc.) is not a goal of this experiment.
Enhancing the OS tunables to fit the workload is also not a goal, node being run under normal user permissions can only query the values and enhance the TCP chunking accordingly.

Externals:
A --bigdata flag to be introduced to node.js for usage in large data xfer scenarios. Using this flag will cause node to query the platform to read the maximum possible TCP read/write buffer sizes, and the TCP sockets thus created will be made capable of transferring chunk sizes equal to that.
Testing:
stream-bench module which is a benchmark for measuring streaming performance can be used for measurement, with few combinations of client-server proximity (short and long RTT systems)

Client and Server in the same system, using loopback address.
Client and Server in the same system, using IP address.
Client and Server in different systems, under same subnet.
Client and Server in different systems, under different subnet.
while [1] may not reflect a real use case, [2] (cluster), [3] (collective cluster / db / cloud), [4] (db / microservices / cloud) are.

Some early experimet in Linux revealed the following: (I paln to experiment with AIX, Windows, MAC etc.)
A test case with 128 concurrent clients accessing 5MB of data across network showed this strace statistics: node at the client side, so read() is at focus.
Original node (reading in terms of 64KB): total runtime: 13500 millis.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
98.00    0.786124        2629       299        34 futex
  0.31    0.002518          18       142           mmap
  0.20    0.001574           1      1349           read
  0.17    0.001331          23        59           mprotect
  0.14    0.001158          33        35           rt_sigaction
  0.14    0.001084           2       514       128 epoll_ctl
.......
------ ----------- ----------- --------- --------- ----------------
100.00    0.802162                  3420       310 total

Modified node (reading in terms of 256KB): total runtime: 7639 millis.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
99.35    2.773892       11276       246        28 futex
  0.32    0.009062          16       566           read
  0.05    0.001328           9       140           mmap
  0.05    0.001260           2       514       128 epoll_ctl
  0.03    0.000977           7       131           write
  0.03    0.000835           2       384           setsockopt
  0.03    0.000727           4       169           epoll_wait
  0.03    0.000700           7        96           munmap
  0.02    0.000583           4       142           close
  0.02    0.000497           4       128       128 connect
....
------ ----------- ----------- --------- --------- ----------------
100.00    2.792172                  3087       304 total

*Few observations [and their explanations]:

Amount of time spent in read has increased (1.5 millis vs. 9 millis).
[due to the increase work in the kernel for allocating and copying in large chunks ]
Amount of time spent in the kernel has increased (800 vs 2700 ms).
[due to the increase work in the kernel for allocating and copying in large chunks ]
But eventually, the large buffer node performs better, (7.6 secs vs 13.5 secs).
[number of callbacks reduced, OS buffers are efficiently utilized, other parties in the uv loop is sufficiently atteneded to.]
The performance gain is a function of concurrence in the system. If a dedicated client attempts to read, the gain is not visible, instead the read overhead surfaces.
[ No code can match the speed of a single-threaded, dedicated, blocking C client-server data xfer]
The performance gain is also a function of the actions in the read callback. If an empty callback is employed, the performance gain is not visible.
[ When the read callback is under execution, the read buffer in the kernel gets accumulated with data. The more time the callback runs, the more the buffers are filled, making the next readrelatively optimal. ]
In summary my inference is that in an application where the uv loop is sufficiently excercised, the large-buffer-read-overhead is masked by the gain we get by: i) better utiliation of n/w resources in the system, ii) reduced callbacks, iii) reduction in the starvation in the loop.

Please let me know what you think.

The current unit of data xfer in Node is 64 KB (see uv__read for example) and is hard-coded.

Not that those 64K libuv passes are just a suggestion.  Node (and any other application) are free to ignore it and pick a better value.  In fact, we are removing that for v2 :-)
So, how did you modify libuv, if at all?  What kind of feedback do you expect? On the methodology?  Any actionable items?
Cheers!

thanks @saghul. the approach is to query the OS to confirm the window scaling option, and then pick up /proc/sys/net/core/rmem_max for the socket receiver buffer size, and then modify the chunk size in uv__read to this value. [ improvement here would be to take into account rmem_max and wmem_max, to cater to full-duplex sockets (most common cases).
the performance is 'really' a function of what we do in the read callback and how loaded the uv look are. So the feedback requested are: i) does the customization of chunk size for workloads which demands it sound good to you? ii) is deriving the chunk sizes from the underlying system appear reasonable to you? iii) any specific workload which you know of this experiment can bank on?

@gireeshpunathil Did you profile node.js or plain libuv + something else?
Also, can I suggest you use perf instead of strace?  strace is ptrace-based and has rather significant overhead; it's not a good tool for benchmarking.

@bnoordhuis - I used node.js. sure, will use perf and come up with observations. thanks.

test details:
data xfer: 10MB
no. of concurrent clients: 128
network: short RTT.
uv__read with 64KB chunk, and tcp auto-tuning(no SNDBUF and RCVBUF adjustments):
Performance counter stats for 'node_old down.js [IP] 25000 128':

      25918.928625 task-clock                #    0.744 CPUs utilized
             2,545 context-switches          #    0.098 K/sec
               133 cpu-migrations            #    0.005 K/sec
            23,658 page-faults               #    0.913 K/sec
   <not supported> cycles
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
   <not supported> instructions
   <not supported> branches
   <not supported> branch-misses

      34.841539894 seconds time elapsed

uv__read with 256KB chunk, (SNDBUF = RCVBUF = 256KB with window scaling ON):
Performance counter stats for 'node down.js [IP] 25000 128':

       7615.776219 task-clock                #    0.472 CPUs utilized
             1,680 context-switches          #    0.221 K/sec
                52 cpu-migrations            #    0.007 K/sec
            22,619 page-faults               #    0.003 M/sec
   <not supported> cycles
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
   <not supported> instructions
   <not supported> branches
   <not supported> branch-misses

      16.144609213 seconds time elapsed

Also ran with appmetrics (https://www.npmjs.com/package/appmetrics) to see how the uv loop latency for pending tasks (time elapsed between requesting a task to be run on the loop and its execution commencement - such as process.nextTick(), setImmediate(), setTimeout()) are faring:
64KB case:
4916
4154
4452
289
0

256KB case:
615
185
0

leads to an inference that the loop residents are better attended to, in the second case.

By the <not supported> markers I'm guessing you're testing in a VM.
For reliable benchmark numbers, bare metal is still best but for now, try perf record -c 40000 -i -g node --perf_basic_prof to get call stacks.  If you want to filter/group on system calls, use perf probe.

I am working with @gireeshpunathil on this. The top CPU consumers for original and modified node are as shown below. I am still exploring to get fine tuned info with the callstack as well as filter probes.
Top 10 consumers in original node (full profile):

+  23.63%  node_org  perf-11287.map        [.] 0x000037d67f60bb77
+   2.29%  node_org  node_org              [.] v8::internal::Scanner::Scan()
+   2.20%  node_org  node_org              [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
+   1.66%  node_org  node_org              [.] v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8
+   1.54%  node_org  node_org              [.] v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpression(
+   1.52%  node_org  node_org              [.] void v8::internal::String::WriteToFlat<unsigned short>(v8::internal::Strin
+   1.47%  node_org  node_org              [.] v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationSpace, v8::in
+   1.27%  node_org  node_org              [.] void v8::internal::String::WriteToFlat<unsigned char>(v8::internal::String
+   1.26%  node_org  node_org              [.] v8::internal::StringReplaceGlobalRegExpWithString(v8::internal::Isolate*,
+   1.14%  node_org  node_org              [.] v8::internal::AstValueFactory::GetOneByteStringInternal(v8::internal::Vect

Top 10 consumers in modified node (full profile):
+  22.18%  node  perf-10270.map        [.] 0x0000122fc5408529
+   2.99%  node  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
+   2.15%  node  node                  [.] v8::internal::Scanner::Scan()
+   2.07%  node  node                  [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
+   1.41%  node  node                  [.] void v8::internal::String::WriteToFlat<unsigned short>(v8::internal::String*,
+   1.40%  node  node                  [.] v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpression(bool
+   1.38%  node  libc-2.17.so          [.] memchr
+   1.38%  node  node                  [.] v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::in
+   1.28%  node  node                  [.] v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationSpace, v8::intern
+   1.21%  node  node                  [.] void v8::internal::String::WriteToFlat<unsigned char>(v8::internal::String*, u

The key observation is that buffer copy (copy_user_enhanced_fast_string) overhead has increased, due to the large read. libc:memchr also shows up, which is root caused by the Buffer.includes() call from the test case, which takes more time to search in large data.
Top 10 consumers in the kernel (original node):
   0.81%  node_org  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
    0.49%  node_org  [kernel.kallsyms]     [k] iowrite16
    0.17%  node_org  [kernel.kallsyms]     [k] free_hot_cold_page
    0.14%  node_org  [kernel.kallsyms]     [k] __do_softirq
    0.11%  node_org  [kernel.kallsyms]     [k] put_page
    0.08%  node_org  [kernel.kallsyms]     [k] get_page_from_freelist
    0.08%  node_org  [kernel.kallsyms]     [k] run_timer_softirq
    0.07%  node_org  [kernel.kallsyms]     [k] tcp_recvmsg
    0.05%  node_org  [kernel.kallsyms]     [k] clear_page_c_e
    0.04%  node_org  [kernel.kallsyms]     [k] _raw_spin_unlock_irqrestore

Top 10 consumers in the kernel (modified node):
     2.99%     node  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
     0.59%     node  [kernel.kallsyms]     [k] iowrite16
     0.46%     node  [kernel.kallsyms]     [k] free_hot_cold_page
     0.41%     node  [kernel.kallsyms]     [k] get_page_from_freelist
     0.38%     node  [kernel.kallsyms]     [k] __do_softirq
     0.28%     node  [kernel.kallsyms]     [k] put_page
     0.25%     node  [kernel.kallsyms]     [k] clear_page_c_e
     0.15%     node  [kernel.kallsyms]     [k] __do_page_fault
     0.13%     node  [kernel.kallsyms]     [k] _raw_spin_unlock_irqrestore
     0.09%     node  [kernel.kallsyms]     [k] tcp_recvmsg

This is in alignment with the full profile where the TCP read and memory copy takes more CPU. [ I need to figure out who calls iowrite16 ].

@sreepurnajasti Can you paste the output of perf report --stdio?  In the first two, this...

23.63%  node_org  perf-11287.map        [.] 0x000037d67f60bb77

...is the (or at least: a) thing of interest.

Last time I have misplaced the command --perf_basic_prof in the command line position and hence the JS symbols were not resolved, and accumulated into one single entity, with their total cpu consumption.
Currently, I am not able to produce the correct profile due to internal network problem on the machine we are running. I will update as soon as I have data.

when perf report –stdio command is executed,
In Modified node: copy_user_enhanced_fast_string, scan, ScanIdentifierOrKeyword()  are the top 3 consumers
In Original node: Scan, ScanIdentifierOrKeyword()  are the top 2 consumers
As updated earlier, the single most cpu consumer

23.63% node_org perf-11287.map [.] 0x000037d67f60bb77

Is now (with correct command line options) is now distributed all round
NOTE: I am not pasting each command in detail as it is very huge
The following are the observations below:
Modified Node:

Overhead        Command         Shared Object
# ........  ....................  ..................................................
3.02%     node  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
     |
     --- copy_user_enhanced_fast_string
        |
        |--99.70%-- skb_copy_datagram_iovec
        |tcp_recvmsg
        |inet_recvmsg
        |sock_aio_read.part.7
        |sock_aio_read
        |do_sync_read
        |vfs_read
        |sys_read
        |system_call_fastpath
        |0x7f8a6746325d
        |uv__stream_io
        |uv__io_poll
        |uv_run
        |node::Start(uv_loop_s*, int, char const* const*, int, char const* const*)
        |node::Start(int, char**)
        |__libc_start_main
         --0.30%-- [...]

1.99%     node  node        [.] v8::internal::Scanner::Scan()
     |
     --- v8::internal::Scanner::Scan()
        |
        |--89.08%-- v8::internal::Scanner::Next()
        ||
        ||--17.08%-- v8::internal::ParserBase<v8::internal::Parser>::Expect(v8::internal::Token::Value, bool*)

(this continues …)

1.98%     node  node        [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
     |
     --- v8::internal::Scanner::ScanIdentifierOrKeyword()
        |
        |--98.19%-- v8::internal::Scanner::Scan()
        ||
        ||--99.43%-- v8::internal::Scanner::Next()

(this continues …)

Original Node:
# Overhead   Command         Shared Object
# ........   ....................  .....................................................
  2.26%  node_org  node_org    [.] v8::internal::Scanner::Scan()
  |
  --- v8::internal::Scanner::Scan()
     |
     |--88.64%-- v8::internal::Scanner::Next()
     ||
     ||--17.27%-- v8::internal::ParserBase<v8::internal::Parser>::Expect(v8::internal::Token::Value, bool*)
(this continues...)

2.21%  node_org  node_org    [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
  |
  --- v8::internal::Scanner::ScanIdentifierOrKeyword()
     |
     |--98.00%-- v8::internal::Scanner::Scan()
     ||
     ||--99.44%-- v8::internal::Scanner::Next()
(this continues…)

0.97%  node_org  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
  |
  --- copy_user_enhanced_fast_string
     |
     |--99.63%-- skb_copy_datagram_iovec
     |tcp_recvmsg
     |inet_recvmsg
     |sock_aio_read.part.7
     |sock_aio_read
     |do_sync_read
     |vfs_read
     |sys_read
     |system_call_fastpath
     |0x7f24860f925d
     |uv__stream_io
     |uv__io_poll
     |uv_run
     |node::Start(uv_loop_s*, int, char const* const*, int, char const* const*)
     |node::Start(int, char**)
     |__libc_start_main
      --0.37%--

when perf report –stdio command is executed,
In Modified node: copy_user_enhanced_fast_string, scan, ScanIdentifierOrKeyword()  are the top 3 consumers.
In Original node: Scan, ScanIdentifierOrKeyword()  are the top 2 consumers.
As updated earlier, the single most cpu consumer

23.63% node_org perf-11287.map [.] 0x000037d67f60bb77

is now (with correct command line options) is now distributed all round.
NOTE: I am not pasting each command in detail as it is very huge.
The following are the observations below:
Modified Node:
Overhead        Command         Shared Object
# ........  ....................  ..................................................
3.02%     node  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
     |
     --- copy_user_enhanced_fast_string
        |
        |--99.70%-- skb_copy_datagram_iovec
        |tcp_recvmsg
        |inet_recvmsg
        |sock_aio_read.part.7
        |sock_aio_read
        |do_sync_read
        |vfs_read
        |sys_read
        |system_call_fastpath
        |0x7f8a6746325d
        |uv__stream_io
        |uv__io_poll
        |uv_run
        |node::Start(uv_loop_s*, int, char const* const*, int, char const* const*)
        |node::Start(int, char**)
        |__libc_start_main
         --0.30%-- [...]

1.99%     node  node        [.] v8::internal::Scanner::Scan()
     |
     --- v8::internal::Scanner::Scan()
        |
        |--89.08%-- v8::internal::Scanner::Next()
        ||
        ||--17.08%-- v8::internal::ParserBase<v8::internal::Parser>::Expect(v8::internal::Token::Value, bool*)

(this continues …)

1.98%     node  node        [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
     |
     --- v8::internal::Scanner::ScanIdentifierOrKeyword()
        |
        |--98.19%-- v8::internal::Scanner::Scan()
        ||
        ||--99.43%-- v8::internal::Scanner::Next()

(this continues …)

Original Node:

# Overhead   Command         Shared Object
# ........   ....................  .....................................................
  2.26%  node_org  node_org    [.] v8::internal::Scanner::Scan()
  |
  --- v8::internal::Scanner::Scan()
     |
     |--88.64%-- v8::internal::Scanner::Next()
     ||
     ||--17.27%-- v8::internal::ParserBase<v8::internal::Parser>::Expect(v8::internal::Token::Value, bool*)
(this continues...)

2.21%  node_org  node_org    [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
  |
  --- v8::internal::Scanner::ScanIdentifierOrKeyword()
     |
     |--98.00%-- v8::internal::Scanner::Scan()
     ||
     ||--99.44%-- v8::internal::Scanner::Next()
(this continues…)

0.97%  node_org  [kernel.kallsyms]     [k] copy_user_enhanced_fast_string
  |
  --- copy_user_enhanced_fast_string
     |
     |--99.63%-- skb_copy_datagram_iovec
     |tcp_recvmsg
     |inet_recvmsg
     |sock_aio_read.part.7
     |sock_aio_read
     |do_sync_read
     |vfs_read
     |sys_read
     |system_call_fastpath
     |0x7f24860f925d
     |uv__stream_io
     |uv__io_poll
     |uv_run
     |node::Start(uv_loop_s*, int, char const* const*, int, char const* const*)
     |node::Start(int, char**)
     |__libc_start_main
      --0.37%--

NOTE: I am not pasting each command in detail as it is very huge

Instead of posting large text inline, you can create a gist at gist.github.com, and post links.

@sreepurnajasti If you see Scanner and ParserBase in there, the application can't have been doing anything very interesting.  That's the JS source code parser and it normally takes up an insignificant fraction of the total running time, especially in programs that run for longer than a second or two.
You wouldn't happen to be spawning child processes, would you?  Because in that case you've probably been profiling the parent process while it was waiting for the child process.

The test case and the full logs are available here
The test case is an attempt to simulate a real world workload from the UV loop perspective. Suggestions to improve the test case is much appreciated!
Here is an explanation to what I am doing in the code:

Three sites are accessed 128 times in conjunction with the main streaming, to make sure that there are other I/O operations in the work queue.

An empty function is called at 100ms interval and 100 times to make sure that the UV loop is attended to, and to make the program completion depend on this.

App-metrics is used to measure the latency incurred in the loop.

Following actions are performed per callback in the down.js:
I. The individual chunk lengths are stored - which can be used to measure how much we read every time.
II. An object serialization which incurs a constant amount of work (invariant of the chunk volume) to encapsulate any general CPU bound activities.
III. A raw search is performed in the received chunk - effort linearly proportional to the chunk volume.

The test does not spawn any child process. As I have observed earlier, the more work we do in the callback, the performance gain increases, and vice versa. My reasoning on that is as follows:
I. When more CPU is spent in the callback, the time to return to UV__read increases, and as a result, I/O buffer accumulates more incoming data. (potentially > 64KB)
II. When next iteration of read is attempted (in UV__read), we get more data than earlier, and thereby reducing the callback count.
III. While reading amount is increased, the probability of bytes read to be less than requested length (256KB in this case) is still more than the old case. This causes the thread to return to uv loop, and attend other parties, more frequently.

That's not how I would describe what the gist does. I would say it does this:

sets an interval timer to force node to stay alive for at least 10 seconds, whether it is actively doing any work or not, not sure what it means for the uv loop to be "attended to", or why this is here

and starts a whole bunch of http.gets at the same time, so they run in parallel:

makes (128 * 3) http.gets across 3 well-known web sites, and doesn't bother reading their responses, which takes < 4 seconds on my machine over slow wifi, most of it i/o blocked, only 1 sec spent in cpu
makes 128 http.gets to a specific URL from cmd line, collects the response, and does two pointless calls (util.inspect, and Buffer.includes) - I assume this URL is to a server that supports large TCP windows?

The whole test looks like it runs too fast to learn anything, but then pauses doing nothing but a set interval for 5 or 6 seconds.
I'd suggest setting off hundreds of http.gets in parallel, but when each one completes, have it do another http.get, and have the entire thing run for a few hundred seconds at least, and then call process.exit.

@sam-github, thanks.
I guess your description matches the scenario when the main client read takes very less time. The data being streamed is of importance here, and the program makes meaning with some 'big' data.
Can you please try with a 100MB or more being sent from a server? (in short RTT or in a high speed n/w, otherwise 10MB or more over a long RTT).
In terms of what the code is doing: agree that they look meaningless apparently. But this is simulation testing, meant to do nothing externally, but make their impact visible in the area of code of our interest (libuv in general, the uv__read in particular).
The interval timer: Meant to make sure the timer is kicked in every 100ms. If there is a starvation due to tight-reading of chunks, this will be delayed, and show off in the overall performance.
Reading from 3 well-known web sites: again, they don't do any meaningful work, but causes some tasks to be pending in the pre-loop-queue (the loop watcher queue) and / or some I/O ready fds in the OS queue. The overall run time depends on their completion, which in turn get influenced by the chunk reading logic in uv__read.
Two point-less calls: Please suggest how do we otherwise 'encapsulate' variety of streaming / piping use cases in few lines here. My rational is to combine an invariant and a variant operation (w.r.t the incoming data). If we don't do anything in the callback, we deviate from the most common use cases, and if we over-do here, we go far from piping scenarios. Care has been taken to strike a right balance, but am happy to amend, with a better abstraction if available.
Yes, server supports the large TCP Windows.
If I read back-to-back (as opposed to parallel), the overhead of socket creation, connection etc. can surface and may cause noise in the read. On the other hand, there is a limit on the number of sockets which can be established between two end-points, and hence 128 clients.
The intent of the test is to see how efficiently we are reading, efficiency here refers to the combination of speed and concurrency.
Having said that, I will follow your suggestion to increase the iterations and the running time and post the results.

As per the earlier suggestions on testcase, I have made the changes and here is the link
The modified  test case does the following:

Generates 128 http.get connections to the server which sends a data of about 93 MB in short RTT
At each connection:

Run a custom word count transformation to the incoming data, and pipe the result into a file.
Repeat this for 99 more times, at the end of each connection.

With this test case, an improvement of 12% is visible with modified node.

@sreepurnajasti Can you post the output from perf?  Intuitively, I would expect that benchmark to spend some (possibly quite a lot) of its CPU time doing work that isn't directly related to network I/O, e.g. the d.toString() call and the regex, plus I can see it getting backlogged on DNS queries and disk writes because they go through the thread pool.

@bnoordhuis Output of top 20 consumers of perf is here. (The perf was run without the call graph as it produced huge data.)  At high level, this seem to match with your expectations. Is there any other info you are keen on from this run?

@sreepurnajasti No, that about confirms my intuition.  I'd say it's hard to draw meaningful conclusions from a benchmark where <1% is spent doing the thing we're actually interested in.
I imagine the 12% speedup is because you enjoy some secondary benefits from being able to operate on larger chunks of data (less garbage to collect, better cache efficiency, etc.) so it might still be worthwhile to pursue.
One last quick gauge: what does /usr/bin/time (not the time bash builtin) print for both binaries?  The sys/user breakdown and number of pagefaults might tell us something.

@sreepurnajasti  I know there are pros and cons of microbenchmarks, and you are trying to emulate some kind of real-world work-load, but I wonder if a straight-line program to read and discard data as quickly as possible from a TCP socket (in node) with data coming from a server with large windows sizes would give a better initial assesment of what kinds of improvements are possible, if any?

@bnoordhuis, thanks.

I imagine the 12% speedup is because you enjoy some secondary benefits from being able to operate on larger chunks of data (less garbage to collect, better cache efficiency, etc.) so it might still be worthwhile to pursue.

Agreed on garbage, cache etc., but how about potential improvements which came through reduced number of callbacks, JIT factors (flow based optimizations when code running on larger volume of data), added effort for call linkage (C -> C++ -> JS -> C++ -> C), object creation overhead etc.?
For example, we see 7531830 read callbacks in the modified case as opposed to 19161368 with the old one.
Here is the time profile, which shows reduction in both system+user space as well as in the page faults.
Modified Node:
[bash node]$ /usr/bin/time ./node_org exp.js <IP_ADDRESS> 2000 128
2602101 millis.
1665.49user 740.30system 43:22.29elapsed 92%CPU (0avgtext+0avgdata 86888maxresident)k
0inputs+64944outputs (0major+78803minor)pagefaults 0swaps

Original Node:
[bash node]$ /usr/bin/time ./node exp.js <IP_ADDRESS> 2000 128
2401625 millis.
1553.76user 673.00system 40:01.76elapsed 92%CPU (0avgtext+0avgdata 97508maxresident)k
0inputs+45120outputs (0major+457397minor)pagefaults 0swaps

@sreepurnajasti Re: potential improvements, I agree, they're the "etc." :-)

@sam-github, thanks.
If we are reading and discarding data as fast as we can, I guess the performance becomes a function of the network bandwidth alone. For example, in most of the dev systems I tested (with varying degree of network proximity), the chunks arrived are in the order of few KBs. In such setups, the current logic of 64 KB read is sufficient (and works better) to cater to back-to-back reading.
Networks which offer better bandwidth, would receive data at better rate than the socket can read, then large chunk reading shows up improvements (reason explained in previous comments). Result of a co-located (server and client in the same system) test with empty read callback (all other factors kept in-tact) is as below:
Original Node: 1035620 millis (total run time)
Modified Node: 1010135 millis (2.4% improvement over the default)
My bottom-line is that when the code tends towards real-world workload scenarios, the large buffer brings benefit.
Now, I am investigating on couple of other platforms.

Purpose of this issue is solved:

Experiment with the approach and make performance observations
Gather review comments on the community on the usefulness.
As PR is place to drive this work to conclusion, closing this issue.


PCSX2 Games not getting controller input

When I add PCSX2 as a non steam game manually, it works. I get a steam overlay when I manually start a game when I run "C:\Games\PCSX2 1.4.0\pcsx2.exe".
When I manually change this shortcut to "C:\Games\PCSX2 1.4.0\pcsx2" --fullscreen --nogui "D:\ROMs\PS2\Kingdom Hearts.iso" It works fine, as well. Controller support, steam overlay, everything.
I HAVE disabled "Allow Desktop Configuration in launcher" and it has worked perfectly.
When I add these games through ICE, however, with the config
[PCSX2]
location=C:\Games\PCSX2 1.4.0\pcsx2
command=%l --fullscreen --nogui %r
I start the game, same setting as the first, and have no steam overlay, and no controller support whatsoever.
Any help here would be greatly appreciated, as adding all of these games manually kind of defeats the purpose of ICE.

Nevermind. I just had to make pcsx not start as Admin.

hey @phantomrtw   You should try SRM. It's a great successor to ICE.
https://steamcommunity.com/groups/SteamRomManager


expose refreshToken

Can we expose this @apechimp

I think so, but this is not what you want to expose because this will do a check to see if the token is expired or not. I think if you factor out the bit that just refreshes the token, that'd be fine. Also, you'll need to write a test and add it to the readme 😝

✨ ✨ ✨ ✨
✨ LGTM✨
✨ ✨ ✨ ✨


Add subscribe_to_webhooks, search_company and request_company.

Comment
This implements subscribe_to_webhooks, search_company and request_company.
Let me know if you see any issues.
Cheers.

Looks good! Although could we make app_secret and app_id configurable through config.exs/runtime.exs?

@crova I know I specified that we should pass in the key / ID each time but I've changed my mind 🙂 Let's use runtime.exs.

Hopefully I got this request right.
Let me know if you see any issues.
Cheers.


No module named 'utils'

hi,i can't find the file 'utils' ~Could u please upload the file?

Just added.
Thanks for noticing it.
Sorry about that.


del/rename + clean up + check ping after connection

Closes https://github.com/Galeontz/URedis/issues/10
I also fixed the documentation and check for proper connection with ping after connect (_instance may be valid even if the redis server is down - ping is better).

Nope, exceptions are not supported in packaged games - so this code will
run in editor but it won't compile in packaged builds (that's one of the
weird unreal things)
On Sun, Jun 2, 2024, 13:12 EliLentz @.***> wrote:

@.**** commented on this pull request.
In Source/URedis/Private/URedis.cpp
https://github.com/Galeontz/URedis/pull/13#discussion_r1623386340:

@@ -24,11 +24,10 @@ void FURedis::Connect(const FStringView host, const int32 port) {
opts.host = TCHAR_TO_UTF8(host.GetData());
opts.port = port;
_instance = MakeUniquesw::redis::Redis(opts);

check(_instance);

check(Ping() == "PONG");

I was talking about using like that:
FURedis& URedisModule = FModuleManager::LoadModuleChecked("URedis");
if(URedisModule != nullptr)
{
try {
URedisModule.Connect(TEXT("<IP_ADDRESS>")):
}
catch(...) {
UE_LOG(LogTemp, Fatal, TEXT("Can't connect to Redis Database"));
}
}
I can't normally imagine program, which runs and then crashes because of
the lost connection instead of safely logging and leaving current process
(e.g leave to main menu)
Just tell me if it possible to do it out of the plugin. That's what I'm
trying to understand.
—
Reply to this email directly, view it on GitHub
https://github.com/Galeontz/URedis/pull/13#discussion_r1623386340, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACAJSDDCCJERYGS7JDSRGPTZFL4Z3AVCNFSM6AAAAABIUE3ITGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAOJSGI3TKOBTHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Nope, exceptions are not supported in packaged games - so this code will run in editor but it won't compile in packaged builds (that's one of the weird unreal things)
…
On Sun, Jun 2, 2024, 13:12 EliLentz @.> wrote: @.* commented on this pull request. ------------------------------ In Source/URedis/Private/URedis.cpp <#13 (comment)>: > @@ -24,11 +24,10 @@ void FURedis::Connect(const FStringView host, const int32 port) { opts.host = TCHAR_TO_UTF8(host.GetData()); opts.port = port; _instance = MakeUniquesw::redis::Redis(opts); - check(_instance); + check(Ping() == "PONG"); I was talking about using like that: FURedis& URedisModule = FModuleManager::LoadModuleChecked("URedis"); if(URedisModule != nullptr) { try { URedisModule.Connect(TEXT("<IP_ADDRESS>")): } catch(...) { UE_LOG(LogTemp, Fatal, TEXT("Can't connect to Redis Database")); } } I can't normally imagine program, which runs and then crashes because of the lost connection instead of safely logging and leaving current process (e.g leave to main menu) Just tell me if it possible to do it out of the plugin. That's what I'm trying to understand. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAJSDDCCJERYGS7JDSRGPTZFL4Z3AVCNFSM6AAAAABIUE3ITGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAOJSGI3TKOBTHA . You are receiving this because you authored the thread.Message ID: @.***>

I found out what exceptions are disabled by default, but can be enabled. By the way, it's out of our project.


Upgrade the openebs/tests-tpcc-client container to perform parallel database load in tpcc-runner.sh

The openebs/tests-tpcc-client container currently performs the following step to load the database with entries (as per the  number of warehouses specified in the tpcc.conf benchmark config file) before running the actual benchmark test (tpcc_start) :-
./tpcc_load -h <DB_SERVER_IP> -P3306 -d<DB_NAME> -u <DB_USER> -p <DB_PASSWORD> -w
This is a sequential process that increases in duration with the number of warehouses specified, thereby delaying the start of the actual benchmark.
This process can be made quicker by running multiple parallel tpcc_load operations on a subset of warehouses. There is already a script (load.sh) provided  by percona in the tpcc-mysql repo which does this - we need to use it in the tpcc-runner.sh with the logic to identify completion of all tpcc_load processes before proceeding to start the benchmark

Moved issue to openebs/test-tools repo: https://github.com/openebs/test-tools/issues/19


#56723 - upgrade Test.Core to newest version and fix analyzers

No eSW.Core package changes, so that version is not being upgraded

Already done by @tabman83 in https://github.com/eShopWorld/core/pull/22


Exhaust Task List Generation

Supersede of #227.
Fixes #215
This PR:

Updates symbiflow and quicklogic toolchain versions.

Adds symbiflow_a200t toolchain to make env.

Changes exhaust.py to by default build all possible combinations of board/toolchain/project.

Adds --only_required parameter to exhaust.py to build only boards that are defined in project.json as required for CI to be green.

Adds nexys-video board.

Limits exhaust.py exception message lenght to 300 characters (without this, log from CI weights ~200MB).

I see in CI that the test combination matrix is generated also for the following tests:

multiple_tests
parameter_tests
seed_tests

I think it might be good to add the --only-prefix parameter when invoking these tests as well.


Adds project reactor

This PR introduces a dependency on Project Reactor to facilitate the use of Flux for streaming responses.
It was mentioned in #689
As suggested in this discussion, a new module has been added to this repository. However, it's worth noting that this addition isn't directly associated with Spring since it solely utilizes the Project Reactor dependency and doesn't rely on Spring Webflux. Therefore, if you believe the new module would be better suited in another repository, I'm open to relocating it.

@ilopezluna thank you for this PR, but it is not very clear how exactly this will help if no model integrations are going to implement FluxStreamingChatLanguageModel, at least those in the main repo.
We plan to support Flux on the AI Service level, so model integrations won't need to change their API:
@AiService
interface Assistant {

    Flux<String> chat(String userMessage);
}


Fixing latest windows version in choose-a-uwp-version.md

Adjust latest build of Windows to 16299.

@QuinnRadich : Passed PR checklist Please review proposed change and merge if appropriate.

Good catch. Thanks.


Compile ofxCcv on raspberry pi

Can you make a library for raspberry pi?
because i can't compile it, cause too many errors :(

Anyway, i've compiled this addon for raspberry pi.
this issue can be closed now, thank you

@khangprolxag if you have ofxCcv working for raspberry pi, this would be a very welcome pull request for this repo.

yes i wil but i still have some error on Rasberry Pi Zero, but work perfectly on Rasberry Pi 2 or 3...

that's fantastic! how fast do the examples run? did you just make a new static library for pi, or did you also have to modify any of ofxCcv's code?  if the first one, then perhaps just PR it to the main repo. if changes to ofxCcv were needed, might be good to make a new branch.

i haven't tried example yet but i wrote my code and tested, it worked. i don't have to modify ofxCcv's code, i just base on your tutorial how do you make static library for linux64 and then i tried on my rasberry pi
here this is libccv.a that i have compiled: https://drive.google.com/file/d/0B2csDFRMXb34dWdyTHFCOEVwcW8/view?usp=sharing
Do you have any idea for the "segmentation fault" error ? Openframeworks doesn't show any errors detail else...

Segmentation fault can be caused by many things, but first one I can think of is that the Pi doesn't have enough memory to load the image-net module.

At the moment I don't have a Pi, so I may not be able to test this immediately.

i think so, i can turn on public SSH on my raspberry if you would like. I really really need this library gets to work on my raspberry Pi Zero

there may be some interest in this project from @AndreasRef. pinging him here...

Thanks, I'm really interested in getting ofxCCV running on Raspberry as well, so any progress @khangprolxag  would like to share would be highly appreciated  :)

@AndreasRef  ya i will, do you have raspberry pi? Can you help me to test the library that i've compiled ?
[create a directory like this "ofxCcv/libs/ccv/lib/armv6l" and place libccv.a into]
https://drive.google.com/file/d/0B2csDFRMXb34dWdyTHFCOEVwcW8/view?usp=sharing
Thank you so much.

Not currently, but can have a look later this month.

Hey,some advance?, how can i help?, im interested in compile ofxCcv in Rpi Zero. Help me please.

Its working on pi zero but the speed is ultra slow. I think you should try
another library

Its working on pi zero but the speed is ultra slow. I think you should try another library

Hi there, have you tried another suitable library so far? I am also trying to use such an addon on my RPi Zero.


fix(collection-of): fix issue when the linter fix rules

:tada: This PR is included in version 1.1.10 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Fix build path

The snapshotDirPath path in the current version is different from the one used in deno task build.
Under normal circumstances, there is no problem, but if you move dev.ts to the src/ directory, a bug will occur.

Good catch, let's add a test case for that to ensure that it works.

I added a test case based on build_test. Do you think it's okay?


Problems with repeat

I'm submitting a ...  (check one with "x")
[x] bug report => searched github for a similar issue or PR
[ ] feature request
[ ] support request

Current behavior
When I click the Add button, Input is not displayed and errors appear in the console.
Expected behavior
When I click an Add button, The input should be displayed.
Minimal reproduction of the problem with instructions
UI Material: Stackblitz
What is the motivation / use case for changing the behavior?
I have a model with these properties:
{
 restricted: boolean,
 access_restriction: {
      emails: string[],
      email_domains: string[]
    }
}

I want the form to display the access_restrictions only when restricted is true and be able to add new restrictions (either emails or email_domains).
Please tell us about your environment:
OS: macos 10.13.6
IDE: Visual Studio Code
Package.json dependencies:
"dependencies": {
    "@angular-devkit/core": "^0.7.5",
    "@angular/animations": "6.1.6",
    "@angular/cdk": "^6.4.7",
    "@angular/common": "6.1.6",
    "@angular/compiler": "6.1.6",
    "@angular/core": "6.1.6",
    "@angular/flex-layout": "^6.0.0-beta.18",
    "@angular/forms": "6.1.6",
    "@angular/language-service": "6.1.6",
    "@angular/material": "^6.4.7",
    "@angular/platform-browser": "6.1.6",
    "@angular/platform-browser-dynamic": "6.1.6",
    "@angular/router": "6.1.6",
    "@ngrx/effects": "6.1.0",
    "@ngrx/entity": "6.1.0",
    "@ngrx/router-store": "6.1.0",
    "@ngrx/store": "6.1.0",
    "@ngrx/store-devtools": "6.1.0",
    "@ngx-formly/core": "^4.7.2",
    "@ngx-formly/material": "^4.7.2",
    "@nrwl/nx": "6.3.0",
    "angular-text-input-highlight": "^1.3.0",
    "class-transformer": "^0.1.9",
    "core-js": "^2.4.1",
    "deep-equal": "^1.0.1",
    "flag-icon-css": "^3.0.0",
    "i18n-iso-countries": "^3.7.4",
    "jasmine-marbles": "^0.3.1",
    "libphonenumber-js": "^1.1.7",
    "moment": "^2.22.0",
    "ngrx-store-logger": "^0.2.2",
    "ngx-loading": "^1.0.14",
    "ngx-webstorage-service": "^3.1.1",
    "normalize.css": "^8.0.0",
    "resolve": "^1.7.1",
    "rxjs": "^6.3.1",
    "rxjs-tslint": "^0.1.5",
    "shallow-equals": "^1.0.0",
    "smooth-scrollbar": "^8.2.7",
    "tslib": "^1.9.3",
    "zone.js": "^0.8.26"
  }

Angular version: 6.1.6

Browser:  Chrome, Firefox, Safari

Language: Typescript

When I click the Add button, Input is not displayed and errors appear in the console.

the raised error is correct you have to define email validator https://stackblitz.com/edit/angular-t3xk4l


Remove outdated RVV 2.0 notice from document #1402

For reasons I describe here, I think it's best to remove the RVV 2.0 version notice from the specification, there does seem to be no indication that what the notice says is actually planned.
I replaced it with text derived from the Vector Crypto Introduction, both should ideally be reworded before an actual release, but this should remove the confusion for now.
See also past discussion on the v-spec repo: https://github.com/riscv/riscv-v-spec/pull/946
It sounded like the change would be merged, but the repo was archived before that got through. My understanding is that the riscv-v-spec repo wasn't supposed to be a source of truth, hence it was archived.

I'm dropping the extensive warning about what Ratified means, as it's a global property of RVIA policy, not of this extension.  Yes, I understand there's a tortured history in this case, but I'd rather not flog a dead horse.

Looking at the other extensions (with the exception of vector crypto), the consistent thing to do is to remove any draft/freeze warnings and update the preface.  That's what we'll do here.

I'm dropping the extensive warning about what Ratified means, as it's a global property of RVIA policy, not of this extension

Exactly, that was the motivation behind this change. Everything I've heard from RVI in any official capacity other than this NOTE referred to RVV 1.0 being ratified. Maybe I've missed some part of the communication.


Guard against invalid kw lists

a small check if the first element of the list is a tuple when non-empry

😄

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:


Update Docker test container

This fixes #96.

Codecov Report

Merging #97 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #97   +/-   ##
=======================================
  Coverage   84.82%   84.82%
=======================================
  Files          31       31
  Lines        1858     1858
  Branches      264      264
=======================================
  Hits         1576     1576
  Misses        245      245
  Partials       37       37

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5332ee1...f7fb1ab. Read the comment docs.

GPU03 has the container built if you want to check it.

Successfully built on my mac and passed all 80 tests on execution.


dashboard/app: optimize pollCompletedJobs()

We currently query all finished non-reported Jobs every reporting_poll_bugs request.
That's very slow and inefficient -- there are more than 50k of such jobs.
In reality, we are never interested in reporting old jobs, so let's only query jobs finished within last month. That's only a few thousands jobs.

I'm not sure 1 month is a correct estimate here.

I think it's a good starting point. We won't query too much and we'll definitely notice that reporting is broken if it's been broken for a month.


[Device Support Request] Siren Model HS2WD manufactured by Heiman

Is your feature request related to a problem? Please describe.
Hi everybody, I just bought an heimann warding device (siren) and unfortunatly its recognized as a sensor device in HA.
Model HS2WD manufactured by Heiman
Describe the solution you'd like
A clear and concise description of what you want to happen.
I saw that the only quirks is for the heimann smoke sensor and alarm device so i guess this is the reason for HA to integrate the device as a sensor device.
What i would like to be integrated :

On/Off binary button for the sounding of the siren
On/Off binary button for the lighning (flash) of the siren
Battery Level sensor
Sensor tempering
Options for type of sounding if possible
Options for type of flashing if possible

Zigbee signature of the device :
{
"node_descriptor": "NodeDescriptor(byte1=1, byte2=64, mac_capability_flags=142, manufacturer_code=4619, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=0, maximum_outgoing_transfer_size=82, descriptor_capability_field=0, *allocate_address=True, *complex_descriptor_available=False, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False, *is_valid=True, *logical_type=<LogicalType.Router: 1>, *user_descriptor_available=False)",
"endpoints": {
"1": {
"profile_id": 260,
"device_type": "0x0403",
"in_clusters": [
"0x0000",
"0x0001",
"0x0003",
"0x0004",
"0x0009",
"0x0500",
"0x0502"
],
"out_clusters": [
"0x0003",
"0x0019"
]
}
},
"manufacturer": "Heiman",
"model": "WarningDevice",
"class": "zigpy.device.Device"
}
zha logs of the pairing process:
[0x0000:zdo] ZDO request ZDOCmd.Mgmt_Permit_Joining_req: [60, <Bool.false: 0>]
Device 0x637f (00:0d:6f:00:15:3a:3b:f7) joined the network
[0x637f:zdo] ZDO request ZDOCmd.Device_annce: [0x637F, 00:0d:6f:00:15:3a:3b:f7, 142]
[0x637f] Requesting 'Node Descriptor'
Tries remaining: 2
[0x637f] Extending timeout for 0x47 request
[0x637f] Node Descriptor: NodeDescriptor(byte1=1, byte2=64, mac_capability_flags=142, manufacturer_code=4619, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=0, maximum_outgoing_transfer_size=82, descriptor_capability_field=0, *allocate_address=True, *complex_descriptor_available=False, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False, *is_valid=True, *logical_type=<LogicalType.Router: 1>, *user_descriptor_available=False)
[0x637f] Discovering endpoints
Tries remaining: 3
[0x637f] Discovered endpoints: [1]
[0x637f:1] Discovering endpoint information
Tries remaining: 3
[0x637f:1] Discovered endpoint information: SizePrefixedSimpleDescriptor(endpoint=1, profile=260, device_type=1027, device_version=0, input_clusters=[0, 1, 3, 4, 9, 1280, 1282], output_clusters=[3, 25])
[0x637f:1:0x0000] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=77 command_id=Command.Read_Attributes_rsp>
[0x637f:1] Manufacturer: Heiman
[0x637f:1] Model: WarningDevice
Checking quirks for Heiman WarningDevice (00:0d:6f:00:15:3a:3b:f7)
Considering <class 'zhaquirks.xbee.xbee_io.XBeeSensor'>
Fail because endpoint list mismatch: {232, 230} {1}
Considering <class 'zhaquirks.xbee.xbee3_io.XBee3Sensor'>
Fail because endpoint list mismatch: {232, 230} {1}
Considering <class 'zhaquirks.smartthings.tag_v4.SmartThingsTagV4'>
Fail because device_type mismatch on at least one endpoint
Considering <class 'zhaquirks.smartthings.multi.SmartthingsMultiPurposeSensor'>
Fail because device_type mismatch on at least one endpoint
Considering <class 'zhaquirks.netvox.z308e3ed.Z308E3ED'>
Fail because device_type mismatch on at least one endpoint
Considering <class 'zhaquirks.gledopto.soposhgu10.SoposhGU10'>
Fail because endpoint list mismatch: {11, 13} {1}
Considering <class 'bellows.zigbee.application.EZSPCoordinator'>
Fail because device_type mismatch on at least one endpoint
device - 0x637F:00:0d:6f:00:15:3a:3b:f7 entering async_device_initialized - is_new_join: True
device - 0x637F:00:0d:6f:00:15:3a:3b:f7 has joined the ZHA zigbee network
0x637F: started configuration
0x637F:ZDO: 'async_configure' stage succeeded
[0x637F:1:0x0000]: finished channel configuration
[0x637F:1:0x0019]: finished channel configuration
[0x637F:1:0x0502]: bound 'ias_wd' cluster: Status.SUCCESS
[0x637F:1:0x0502]: finished channel configuration
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=81 command_id=Command.Read_Attributes_rsp>
[0x637F:1:0x0500]: started IASZoneChannel configuration
[0x637F:1:0x0500]: bound 'ias_zone' cluster: Status.SUCCESS
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=85 command_id=Command.Write_Attributes_rsp>
[0x637F:1:0x0500]: wrote cie_addr: 00:21:2e:ff:ff:07:13:aa to 'ias_zone' cluster: [WriteAttributesStatusRecord(status=<Status.SUCCESS: 0>)]
[0x637F:1:0x0500]: Sending pro-active IAS enroll response
[0x637F:1:0x0500]: finished IASZoneChannel configuration
[0x637F:1:0x0502]: 'async_configure' stage succeeded
[0x637F:1:0x0000]: 'async_configure' stage succeeded
[0x637F:1:0x0500]: 'async_configure' stage succeeded
[0x637F:1:0x0019]: 'async_configure' stage succeeded
0x637F: completed configuration
0x637F: stored in registry: ZhaDeviceEntry(name='Heiman WarningDevice', ieee='00:0d:6f:00:15:3a:3b:f7', last_seen=1623740712.0014222)
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=87 command_id=Command.Default_Response>
[0x637f:1:0x0003] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=89 command_id=Command.Default_Response>
[0x637F:1:0x0003]: executed 'trigger_effect' command with args: '(2, 0)' kwargs: '{}' result: [64, <Status.UNSUP_CLUSTER_COMMAND: 129>]
0x637F: started initialization
0x637F:ZDO: 'async_initialize' stage succeeded
[0x637F:1:0x0502]: initializing channel: from_cache: False
[0x637F:1:0x0502]: finished channel configuration
[0x637F:1:0x0000]: initializing channel: from_cache: False
[0x637F:1:0x0000]: finished channel configuration
[0x637F:1:0x0500]: initializing channel: from_cache: False
[0x637F:1:0x0019]: initializing channel: from_cache: False
[0x637F:1:0x0019]: finished channel configuration
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=91 command_id=Command.Read_Attributes_rsp>
[0x637F:1:0x0500]: finished channel configuration
[0x637F:1:0x0502]: 'async_initialize' stage succeeded
[0x637F:1:0x0000]: 'async_initialize' stage succeeded
[0x637F:1:0x0500]: 'async_initialize' stage succeeded
[0x637F:1:0x0019]: 'async_initialize' stage succeeded
0x637F: power source: Mains
0x637F: completed initialization
[0x637f:zdo] ZDO request ZDOCmd.NWK_addr_req: [00:21:2e:ff:ff:07:13:aa, 0, 0]
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=72 command_id=1>
[0x637f:1:0x0500] ZCL request 0x0001: [<ZoneType.Standard_Warning_Device: 549>, 4619]
[0x637f:1:0x0500] No handler for cluster command 1
[0x637F:1:0x0500]: Enroll requested
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=94 command_id=Command.Default_Response>
[0x637f:1:0x0001] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=73 command_id=Command.Configure_Reporting>
[0x637f:1:0x0001] ZCL request 0x0006: [[AttributeReportingConfig(direction=1, attrid=33, timeout=600)]]
[0x637f:zdo] ZDO request ZDOCmd.Match_Desc_req: [0xFFFD, 260, [25], []]
[0x637f:1:0x0019] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=74 command_id=1>
[0x637f:1:0x0019] ZCL request 0x0001: [0, 4619, 8458, 18, None]
[0x637f:1:0x0019] OTA query_next_image handler for 'Heiman WarningDevice': field_control=0, manufacture_id=4619, image_type=8458, current_file_version=18, hardware_version=None
[0x637f:1:0x0019] No OTA image is available
[0x637f:1:0x0500] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=75 command_id=0>
[0x637f:1:0x0500] ZCL request 0x0000: [<ZoneStatus.Supervision_reports: 16>, <bitmap8.0: 0>, 0, 0]
[0x637f:1:0x0500] No handler for cluster command 0
[0x637F:1:0x0500]: Updated alarm state: ZoneStatus.0
Tries remaining: 3
[0x0000] Delivery error for seq # 0x13, on endpoint id 0 cluster 0x0031: Couldn't enqueue send data request: API is not running
Tries remaining: 2
[0x0000] Delivery error for seq # 0x15, on endpoint id 0 cluster 0x0031: Couldn't enqueue send data request: API is not running
Tries remaining: 3
[0x0000] request status: Status.SUCCESS. response: Neighbors(entries=5, start_index=0, neighbor_table_list=[Neighbor(extended_pan_id=XXXXXXXX, ieee=XXXXXXXX, nwk=0x637F, packed=37, permit_joining=<PermitJoins.Accepting: 1>, depth=1, lqi=145, *device_type=<DeviceType.Router: 1>, *relationship=<Relationship.Sibling: 2>, *rx_on_when_idle=<RxOnWhenIdle.On: 1>)])
Tries remaining: 1
[0x0000] Delivery error for seq # 0x17, on endpoint id 0 cluster 0x0031: Couldn't enqueue send data request: API is not running
[0x0000] Querying next starting at 1
Tries remaining: 3
[0x0000] request status: Status.SUCCESS. response: Neighbors(entries=5, start_index=1, neighbor_table_list=[Neighbor(extended_pan_id=XXXXXXXX, ieee=XXXXXXXX nwk=0x018E, packed=18, permit_joining=<PermitJoins.NotAccepting: 0>, depth=1, lqi=252, *device_type=<DeviceType.EndDevice: 2>, *relationship=<Relationship.Child: 1>, *rx_on_when_idle=<RxOnWhenIdle.Off: 0>)])
[0x0000] Querying next starting at 2
Tries remaining: 3
[0x0000] request status: Status.SUCCESS. response: Neighbors(entries=5, start_index=2, neighbor_table_list=[Neighbor(extended_pan_id=XXXXXXXX, ieee=XXXXXXXX, nwk=0x9686, packed=18, permit_joining=<PermitJoins.NotAccepting: 0>, depth=1, lqi=252, *device_type=<DeviceType.EndDevice: 2>, *relationship=<Relationship.Child: 1>, *rx_on_when_idle=<RxOnWhenIdle.Off: 0>)])
[0x0000] Querying next starting at 3
Tries remaining: 3
[0x0000] request status: Status.SUCCESS. response: Neighbors(entries=5, start_index=3, neighbor_table_list=[Neighbor(extended_pan_id=XXXXXXXX, ieee=XXXXXXXX, nwk=0xE2CA, packed=18, permit_joining=<PermitJoins.NotAccepting: 0>, depth=1, lqi=186, *device_type=<DeviceType.EndDevice: 2>, *relationship=<Relationship.Child: 1>, *rx_on_when_idle=<RxOnWhenIdle.Off: 0>)])
[0x0000] Querying next starting at 4
Tries remaining: 3
[0x0000] request status: Status.SUCCESS. response: Neighbors(entries=5, start_index=4, neighbor_table_list=[Neighbor(extended_pan_id=XXXXXXXX, ieee=XXXXXXXX, nwk=0x61AF, packed=37, permit_joining=<PermitJoins.Accepting: 1>, depth=1, lqi=253, *device_type=<DeviceType.Router: 1>, *relationship=<Relationship.Sibling: 2>, *rx_on_when_idle=<RxOnWhenIdle.On: 1>)])
[0x0000] Done scanning. Total 5 neighbours
[0x61af:1:0x0006] ZCL deserialize: <ZCLHeader frame_control= manufacturer=None tsn=97 command_id=Command.Report_Attributes>
[0x61af:1:0x0006] ZCL request 0x000a: [[Attribute(attrid=0, value=)]]
[0x61af:1:0x0006] Attribute report received: on_off=0
Additional context
I know how to code in python so if i can help writing tests or the quirck i will do it gladly,  but i'm a total noobs in zigbee network programming so i need directions from the devs.
I saw that there is a siren.py in the tuya quircks which is probably a similar device so maybe it is a good starting point ?
Any direction or help will be greatly appreciated.
Thanks again for your time.

I've just received a siren from Heiman, model is HS2WD-Y and seems now to be a Tuya siren.
Signature is:
{   "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4619, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",   "endpoints": {     "1": {       "profile_id": 260,       "device_type": "0x0403",       "in_clusters": [         "0x0000",         "0x0001",         "0x0003",         "0x0500",         "0x0502",         "0x0b05"       ],       "out_clusters": [         "0x0004",         "0x0019"       ]     },     "242": {       "profile_id": 41440,       "device_type": "0x0061",       "in_clusters": [],       "out_clusters": [         "0x0021"       ]     }   },   "manufacturer": "_TYZB01_8scntis1",   "model": "TS0216",   "class": "zigpy.device.Device" }

I can control this device using a script like:
service: zha.warning_device_warn data:   mode: 1   level: 3   duration: 1   ieee: 80:4b:50:ff:fe:xx:xx:xx
The duration works.  The level is ignored.

@MattWestb Could you please help me in trying to create a quirk for this device?
I don't know if it's a good idea but I would like to start with an already existing quirk. I just don't know which one ¯_(ツ)_/¯
This is this device:
https://www.zigbee2mqtt.io/devices/HS2WD-E.html
https://github.com/Koenkk/zigbee-herdsman-converters/blob/19bb57b7bc2f686e9bf05e28a5b1b6219e8a7d46/devices/heiman.js#L263
Thank you

Start with one copy of https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/heiman/smoke.py and name it siren.py.
Configure local quirks in HA and putting the new quirk in the folder.
Then updating the device signature with all information and cluster functions.
Then you must do import of all new clusters and deleting the you is not needing.
If not all imports is not OK ZHA is not starting but you is getting one error in the log that showing what is wrong.
Your device type is IAS_WARNING_DEVICE = 0x0403
Your new cluster is:
https://github.com/zigpy/zigpy/blob/7c168c1c2ca2a338676e5956d2277af7c40971d6/zigpy/zcl/clusters/security.py#L62
https://github.com/zigpy/zigpy/blob/7c168c1c2ca2a338676e5956d2277af7c40971d6/zigpy/zcl/clusters/security.py#L346
Then you can start fixing the replace part and hope all functions is OK and if not you must finding how to fixing it.

Thank you so much! I've started something and I currently have a quirk that match my device! \o/
Now how to get information from the embedded battery?

From device card manage cluster and cluster 0x0001 and attribute 0x0021 click get zigbee attribute and waking the device so it can receiving the command.
Some device is not doing it OK and you must reading the voltage instead 0x0020.

This siren is powered by main but has a battery inside in order to still work for some time even if unplugged. So in ZHA, "Power Source" is equal to "Mains" but I would like ZHA to create a sensor for the battery.
0x0021 is 200 and 0x0020 is 41

0x0012 is send with * 2 so in real its 100%
0x0020 is send in 0.1 volts so its 4.1 V in real.
In https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/tuya/ts0201_zemismart.py its one sensor that is saying its main powered and the quirk is changing it to battery so ZHA is showing the battery indicator.

No way to create a sensor based on these value without rewriting Node descriptor?
Do you know where I can find Node descriptor values?
Thank you

I think you is getting one then paring the device if you have debug logging.

class _NodeDescriptorEnums: and under.

https://github.com/zigpy/zigpy/blob/a19a2a063d5edacb46722aeb56d538badd37fcc0/zigpy/zdo/types.py#L14-L18

OK I have the battery working!!! (this is not trivial at all). Thank you again :)
Now to get entities for the siren sensor like done for this quirk: https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/tuya/ts0601_siren.py

I think you is getting one then paring the device if you have debug logging.
Sorry wrong answer :-((

That was true :)

tuya ts0601 device is bad if you have one more normal devices then they is using one own protocol (DP = tuya MQTT 2 cloud) and have very strange functions "converted" to Zigbee standard in the quirk.
You can trying it but i think its better looking of some other siren that have getting quirks if you can using the function they is have done.

I would like to be able to set mode, level, strobe, etc. from ZHA like it’s possible in Z2M. Maybe I need to wait for ZHA to support siren entities?

ZHA have getting one alarm panel for IAS but no Siren card.
If you can steering the functions with cluster commands and attributes you can making trample switches that is sending the commands in HA GUI.
If you good in wiring code you can also making one device card that have all your nice functions (i cant ;-(( ).

FYI, support for siren entities in ZHA integration component has now been added to the Home Assistant core:
https://github.com/home-assistant/core/pull/60920
(that will maybe not make it into HA 2021.12 release? https://rc.home-assistant.io/blog/2021/12/03/release-202112/ )
Are you also looking at adding device support for this new siren entity platform for Zigbee sirens (which also includes doorbells and chimes) to the ZHA integration component in Home Assistant core?
As per the feature requested here:
https://community.home-assistant.io/t/zha-integration-to-use-siren-entity-platform-for-zigbee-sirens-doorbells-and-chimes/343075
Request ZHA integration 1 use recently new siren entity platform introduced in Home Assistant 2021.8
" A siren entity is a device whose main purpose is to control siren devices like a doorbell or chime."
https://www.home-assistant.io/blog/2021/08/04/release-20218/#siren
https://www.home-assistant.io/integrations/siren/
https://developers.home-assistant.io/docs/core/entity/siren/
So far Z-Wave JS, DeCONZ, and Tuya integrations (Devolco is WIP) has added siren entity platform:
https://github.com/home-assistant/architecture/issues/375
https://github.com/home-assistant/core/pull/52948
https://github.com/home-assistant/core/pull/56397
https://github.com/home-assistant/core/pull/57780
https://github.com/home-assistant/core/pull/53400

It is in 2021.12. I’m currently running the beta and my Heiman siren is now recognized as a siren \o/

It is in 2021.12. I’m currently running the beta and my Heiman siren is now recognized as a siren \o/

Hi mbo18,
I have acquired the same siren, and it's recognized by zha as you mentioned. But, I don't see the other info like battery, etc.
Could you please share your quirk code so I can test it too?
Thanks.

The cluster 0x0502 is having little different attribute then Zigbee standard i was finding them in some tuya dev docks.
0x0502 IasWd Cluter
* 0xF000 (int8)

  * 0 Disarm the alarm
  * 1 Sound alarm
  * 2 Light alarm
  * 3 Sound and light alarm

@MattWestb what do you mean about this ? is 0xF000 a new command or a new attribute ?

Its tuya version attributes that need being converted to Zigbee standard or ZHA need adding them as custom attributes for getting the right function in (Z)HA.
The standard Zigbee cluster you can see its not easy matching:
https://github.com/zigpy/zigpy/blob/16ec072eb5e1e94dac33e5d408f939cf6746d652/zigpy/zcl/clusters/security.py#L326

Just to be sure. The only attribute in 0x502 cluster is 0x0000 max_duration. Do you mean that this item has another one that is  0xF000 ?


SQL Tables Not Found

I'm getting some table not found errors at various points. I tried the entire installation process twice on one machine and setup a new computer and tried it again but keep getting the same errors.
When I click on client config I get this
500 SQLSTATE[42S02]: Base table or view not found: 1146 Table 'solder.clients' doesn't exist (SQL: select * from clients)
Whenever I try to add an user or edit the existing I'm getting this

500 SQLSTATE[42S22]: Column not found: 1054 Unknown column 'solder_keys' in 'field list' (SQL: update user_permissions set solder_full = 1, solder_users = 0, mods_create = 0, mods_manage = 0, mods_delete = 0, updated_at = 2015-02-05 12:13:40, solder_keys = 0, solder_clients = 0, modpacks_create = 0, modpacks_manage = 0, modpacks_delete = 0 where id = 1)*

Any ideas?

Are you updating from 0.6 or creating from scratch?

Its a new install with a fresh Nginx and Mysql install as well.

This looks similar: https://github.com/TechnicPack/TechnicSolder/issues/259

I was able to run this command as referenced in 259 and all the tables were added. Thanks!
php artisan migrate:refresh

Anytime


Add an empty logs directory to unblock test runs on the DRT VMs.

@atulkatti,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Test/featuretestcase

added 'patch' to allowed HTTP verbs in RouteCollection
added resetRoutes() in RouteCollection, to support FeatureTestCase
added output capturing to FeatureTestCase::call
added more methods to tests/_support/Controllers/Popcorn, to generate all the different kinds of responses
added note to testing/features in user guide

Related to #1767, #1692 & possibly #1697
Closes #1767

Three SQLite3 tests are failing, and there are some warnings for SQLite3 too.
I don't see these being related to the current PR, though they pass travis-ci for other PRs.
This should be ready to go, as far as I can tell.

Doesn't look like those tests are affected by anything you did. They seem to be missing a $refresh = true or similar on those test classes.

I think there is something screwy with travis-ci.
I re-ran the test suite that failed, and now three different tests fail - Validation ones.
It seems that the db_user table is sometimes there for SQLite3, and sometimes not.


Support ALTER ONLINE TABLE syntax

This PR supports an experimental syntax: ALTER ONLINE TABLE ..., which has the same syntax as ALTER TABLE except for the addition of ONLINE.
This syntax isn't supported by MySQL: it is a Vitess experiment (PR link to follow) with special magic syntax. To that effect, gh-ost will silently accept --alter="alter online table ..." syntax as valid. It otherwise doesn't do anything special if the word ONLINE appears.

Context: https://github.com/vitessio/vitess/pull/6547

There is no need for this PR. Vitess can send the alter_options part of the ALTER TABLE statement.


Link to download "Release" not present.

the readme states to download a release, however does not provide a link, attempted downloading the zip of the git however that results in error below... please update instructions so we can install.
Failed to load extension
File
~\Desktop\wayfarerplus\WayfarerPlusUltra-main
Error
Manifest file is missing or unreadable
Could not load manifest.

It's right there, mate! 😄
I'll add the latest release ( 0.0.4 ) there as well.

Same issue, says Manifest file is missing or unreadable

Same issue, says Manifest file is missing or unreadable

Have you followed the steps listed here https://github.com/mariomc/WayfarerPlusUltra#installation-guide-chrome ?
For Google Chrome, you need to uncompress the folder and load the folder where the manifest.json is present.

You said it's right there, but still didn't provide a link to the page you showed a screenshot of, that's the page we need the link to please....

OH! I just found it!
Here is the link for others to follow, please add this to your wiki!
https://github.com/mariomc/WayfarerPlusUltra/releases

I've followed the instructions, but get this error in Chrome when trying to load extension:

Missing 'manifest_version' key. Its value must be an integer either 2 or 3. See developer.chrome.com/extensions/manifestVersion for details.

0.0.0-alpha2 loaded up fine for me. Something with the manifest was changed with 0.0.4


add number of days to tdata

We specify calendar_start_time, we should als so add the maximum number of days available in the timetable. Now this is always 32.

Taken care of in PR #174

Merged in tdata4


Feat: Add scroll-to-top button in contributor's page

Describe the feature
There is no scroll-to-top button in the contributor's page. As the number of contributors increases, having a scroll-to-top button is always helpful to the user.
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@Harshdev098 Please assign this issue to me and add 'gssoc-ext' and a 'level' labels to it.


How to determine the value of genie.server.max.running.jobs

Hello,
We currently run a single genie instance to support a relatively small hadoop cluster.  We recently started getting 503's and tracked the issue to exceeding the number of max running jobs set in genie.properties.  The default is set to 30, after which the Genie customization docs start talking about using Eureka and multiple instances of Genie.
From genie.properties:
com.netflix.genie.server.max.running.jobs=30
The long-term goal is to pull in Eureka or a tool like it, but for our current use case we're really interested in how this default is determined and if we can get more mileage out of a single node.
For single-instance deployments of Genie, how far does genie server scale with hardware (more RAM, CPU, etc).  What are the variables?
Regards,
Joe Reid

Hello,
The default is determined from the property you mentioned in you email.
That along with a few other properties in genie.properties controls the
number of jobs. Genie is not aware of the characteristics of the jobs that
it runs. So it does not have statistics like Memory, CPU etc. You as an
admin need to figure out what is a typical workload for your job and based
on the resources available on the genie node, how many can it run
concurrently. For example, at Netflix the metric we focus on is 1.5GB of
jvm size per client process. So if we have 30GB of memory, we leave aside
some for system processes and allocate around 22.5GB for Genie jobs. So we
set the max running jobs to 15.
Hope this helps.
Thanks,
Amit
On Mon, Mar 16, 2015 at 8:38 AM, JoeReid-Korrelate<EMAIL_ADDRESS>
wrote:

Hello,
We currently run a single genie instance to support a relatively small
hadoop cluster. We recently started getting 503's and tracked the issue to
exceeding the number of max running jobs set in genie.properties. The
default is set to 30, after which the Genie customization docs
https://github.com/Netflix/genie/wiki/Customization-and-Options start
talking about using Eureka and multiple instances of Genie.
From genie.properties:
com.netflix.genie.server.max.running.jobs=30
The long-term goal is to pull in Eureka or a tool like it, but for our
current use case we're really interested in how this default is determined
and if we can get more mileage out of a single node.
For single-instance deployments of Genie, how far does genie server scale
with hardware (more RAM, CPU, etc). What are the variables?
Regards,
Joe Reid
—
Reply to this email directly or view it on GitHub
https://github.com/Netflix/genie/issues/111.

Hi Joe,
As Amit said currently Genie doesn't force any restrictions on the forced child processes. We have plans in the future to change how jobs are executed and this may be a feature we introduce at that time.
Setting that com.netflix.genie.server.max.running.jobs property is your first step but since you're running in a single node instance you should also set com.netflix.genie.server.forward.jobs.threshold to be a number higher than com.netflix.genie.server.max.running.jobs. This will disable job forwarding and just start rejecting jobs if your node is overwhelmed.
We've never tried to run a ton of jobs on a single Genie node so can't speak to how it will perform if you place it on one huge piece of hardware and crank up the jobs to a large number. We suspect it will work but it's not a use case we currently have. We run ~30 jobs per Genie node internally.
Hope this helps.
Also as an FYI it's probably better to ask questions in the Genie Google Group in the future so others can see answers and they're not lost in Github Issues.
Thanks,
Tom

@tgianos @amitsharma1981 - Great, thanks guys!  I appreciate your help.


Make vulkan queue selection only choose transfer queues in the same family as the dispatch queue.

We don't model queue transfers perfectly and this would (in theory) lead to correctness issues or validation layer nagging. Something we can support eventually but has caveats around things like the unaligned fill emulation. By construction we end up with relatively few transfers anyway.

We aren't really making good use of transfer queues today and we're at least correct now thanks to forcing all work on dispatch queues done in #7391. Whenever we start to look at this stuff again we can rework it all properly.


Setting Light hex color not working (400 bad request)

Describe the bug
I've updated huemagic to 4.2.2 and now my flow is failing.
There are 2 lights that change color every hour using the following msg.payload:
msg.payload = {
    hex: "b387ff"
}

return msg;

Since the last update, this (or any other color/rgb: [255, 255, 255]/etc) results in a 400 - Bad request.
Even the alert does not working anymore.
Flow to Reproduce

Update to 4.2.2
Create a payload using a function block
Send payload including hex/rgb to the destined light
Observe debug window/logs

Expected behavior
The light to change to specified color
Screenshots

Please complete the following information:

Node-RED version: 2.2.2
NodeJS version: v14.18.2
Device running Node-RED: Docker on Raspberry Pi 4 Model B

Hi @GRX,
are the affected bulbs original Signify / Hue ones?

Hi,
Yes, this affects all bulbs. I have Ikea bulbs as original Signify/Hue bulbs and spots and none of them seem to work.
However; I've converted the Hex colors to XY, because colorXY seems to be working :)

How to convert hex to colorXY?

How to convert hex to colorXY?

You can use This tool, set the Input system to Hex.
Once you've pasted the Hex color, you can use the first 2 coordinates at XYZ (so XY) to use in your payload.

XY colors are the actual way to send them to the lamps, since HUE changed the API. The RBG / hex values in HueMagic just got actually translated into XY.
To run it reliable, I recommend to use only XY colors.


youtube -dl --write thumbnail downloading but not writing to video file

Checklist

[ ] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2<IP_ADDRESS>
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bug tracker for similar issues including closed ones

Verbose log

C:\Users\fahad>youtube -f 160+140 -v https://www.youtube.com/watch?v=daefaLgNkw0
[debug] System config: []
[debug] User config: ['--write-thumbnail', '-o', 'C:/Users/fahad/Documents/youtube-dl/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['-f', '160+140', '-v', 'https://www.youtube.com/watch?v=daefaLgNkw0']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg 2020-11-18-git-e3081d6f4f-full_build-www.gyan.dev
[debug] Proxy map: {}
[youtube] daefaLgNkw0: Downloading webpage
[youtube] daefaLgNkw0: Downloading thumbnail ...
[youtube] daefaLgNkw0: Writing thumbnail to: C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.webp
[debug] Invoking downloader on 'https://r8---sn-2uja-pnck.googlevideo.com/videoplayback?expire=1606135391&ei=_1m7X9GCIMTaWeHMtDA&ip=<IP_ADDRESS>&id=o-AHFGQfwu9Gh1nm66KMu0oNNs3Pd83TvCOF7FuH3V5WQj&itag=160&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=Di&mm=31%2C29&mn=sn-2uja-pnck%2Csn-4wg7ln7e&ms=au%2Crdu&mv=m&mvi=8&pl=18&initcwndbps=265000&vprv=1&mime=video%2Fmp4&ns=HSdZOM-Hkr7VCPSQcSV_izYF&gir=yes&clen=581735&dur=598.531&lmt=1577563577666521&mt=1606113635&fvip=5&keepalive=yes&c=WEB&txp=5432432&n=lSmlsJ6KyREV_gU&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgAszQ8nyre_Hk9j2KTiFCIC8XOYvg46zjgbapD78mjuACIEiBRZeJYxl9wssDyorfUxdgfxW1abm5ofFrqq9SvPfN&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALJ6n1BoPnYl6Fo348jqdRIF8zc4SM5ZRuRs0m0f3iIhAiEAnfueygRgVJtwnqxPmmxGJVX03yd5NhzK_GACfaHw4Fw%3D&ratebypass=yes'
[download] Destination: C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f160.mp4
[download] 100% of 568.10KiB in 00:02
[debug] Invoking downloader on 'https://r8---sn-2uja-pnck.googlevideo.com/videoplayback?expire=1606135391&ei=_1m7X9GCIMTaWeHMtDA&ip=<IP_ADDRESS>&id=o-AHFGQfwu9Gh1nm66KMu0oNNs3Pd83TvCOF7FuH3V5WQj&itag=140&source=youtube&requiressl=yes&mh=Di&mm=31%2C29&mn=sn-2uja-pnck%2Csn-4wg7ln7e&ms=au%2Crdu&mv=m&mvi=8&pl=18&initcwndbps=265000&vprv=1&mime=audio%2Fmp4&ns=HSdZOM-Hkr7VCPSQcSV_izYF&gir=yes&clen=9688135&dur=598.587&lmt=1577563560452382&mt=1606113635&fvip=5&keepalive=yes&c=WEB&txp=5431432&n=lSmlsJ6KyREV_gU&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAN87Q1qGqHMg6i80pvw0urMeJX6fRUU96KUbiTq8NPUNAiEA9qDLmcvwA05E-RQVN_VjwUNKlHTuwdY1k1o-NN2uifc%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALJ6n1BoPnYl6Fo348jqdRIF8zc4SM5ZRuRs0m0f3iIhAiEAnfueygRgVJtwnqxPmmxGJVX03yd5NhzK_GACfaHw4Fw%3D&ratebypass=yes'
[download] Destination: C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f140.m4a
[download] 100% of 9.24MiB in 00:05
[ffmpeg] Merging formats into "C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f160.mp4" -i "file:C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f140.m4a" -c copy -map "0:v:0" -map "1:a:0" "file:C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.temp.mp4"
Deleting original file C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f160.mp4 (pass -k to keep)
Deleting original file C:\Users\fahad\Documents\youtube-dl\NA\NA - Python Tutorial for Beginners 5 - Dictionaries - Working with Key-Value Pairs.f140.m4a (pass -k to keep)
C:\Users\fahad>python --version
Python 3.9.0

Description

I am downloading a playlist but the thumbnail is not writing to the video file. I tried different pc and many videos but I get the same result.

--embed-thumbnail                Embed thumbnail in the audio as cover art


Broken link to Azure Cache for Redis

The link: "How to create a Web App with Azure Cache for Redis" https://docs.microsoft.com/api/Redirect/documentation/articles/cache-web-app-howto is broken.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 33a55fba-7e30-02ff-b525-f00f19610b15
Version Independent ID: 42905489-5272-bee8-531f-b4be826eeb7f
Content: Scalable Episerver marketing website - Azure Solution Ideas
Content Source: docs/solution-ideas/articles/digital-marketing-episerver.md
Service: architecture-center
Sub-service: solution-idea
GitHub Login: @adamboeglin
Microsoft Alias: pnp

Thanks for reporting - this issue is under review.  This is a Microsoft Internal DevOps Bug ID AB#219914


Plugins request

Hi Hannes, thanks for the plugins so far.
Please consider also porting the following:

Doublelay
Air2
Air3

Air2 and Air3 have just been released in version 30.0.
My ported version of Doublelay still has some issues (and I don't know why), so I didn't publish it yet.

Doublelay is now also ported with 065ff4cbcb8ddaa8fa99f337f46f749adf11b973 and will be included in the next release.


Why does vim always resolve symbolic-link directory?

Describe the bug
It seems that vim always resolve symbolic-link directory to physical directory!
To Reproduce
$ mkdir /tmp/d1
$ touch /tmp/d1/f1.txt
$ ln -s /tmp/d1 /tmp/d2
$ vim --clean -c 'set statusline=%F | set laststatus=2' /tmp/d2/f1.txt

Notice that the status line shows "/tmp/d1/f1.txt".
Expected behavior
The status line should show "/tmp/d2/f1.txt".
Environment:

Vim 8.1.1968
OS: Arch Linux 5.2.14
Terminal: xterm

Well, I for one appreciate it, that Vim does correctly resolve the file to be edited. In the end it is probably a matter of taste. I do not consider this a bug.

Well, I for one appreciate it, that Vim does correctly resolve the file to be edited. In the end it is probably a matter of taste. I do not consider this a bug.

If so, there should be a setting to disable this.

I don't think this warrants another setting. I believe you can achieve this by a custom 'statusline' function.

This is not about formatting. If autochdir is set, current directory is /tmp/d1 instead of /tmp/d2.

i noticed that too, feeling a little strange to me, i'd like to keep 'xx/d2/yy' if you opened 'd2' which not 'd1'.
sometime, that would be issue if vim auto resolve to 'd1' for a 'link' (#4832 maybe)
:)

@chrisbra

Well, I for one appreciate it, that Vim does correctly resolve the file to be edited. In the end it is probably a matter of taste. I do not consider this a bug.

The behaviour is not consistent, though -- if we continue with the original example:
md /tmp/d3
ln -s /tmp/d1/f1.txt /tmp/d3/
vim --clean +'set statusline=%F | set laststatus=2' /tmp/d3/f1.txt

And now notice that the status line shows "/tmp/d3/f1.txt".
Me, I'm actually with you here, Christian, I do like that the file is resolved :)
However, unfortunately, it seems that it's really not always the case. (Or, may be I'm missing something?)

There are cases where the resolution of symbolic names causes problems.
I have to use a cloud file share  service for work, and it insists on using a long path that has spaces and parentheses in it.  If I use a symbolic name to make the path manageable vim resolves it, and that causes various problems interacting with scripts, as well as annoyingly long file names.
The service provider says, just map a drive letter, but that's a Windows feature. So, I use a bind mount to get a similar result, and vim doesn't know about it.  The bind mount can confuse some utilities, like find, but using a "hidden" directory (a directory with a leading ".") in the real path minimizes this, and it works for me really well.

On 2021-09-19, John Little wrote:

There are cases where the resolution of symbolic names causes problems.

Agreed.
I have a plugin that needs to search for parent directories and for
files in those parent directories.  I used to use finddir() and
findfile() for this, but then discovered that they, too, resolve
symbolic links.  This was no good when running the plugin in
a subdirectory of one of the /usr/src/linux-headers-* directories.
Many of those subdirectories are symbolic links to some other
/usr/src/linux-headers-* subdirectory, so my plugin would find the
wrong parent directory.  I wound up writing my own FindDir() and
FindFile() that textually traverse $PWD/.

I would think that findfile() and finddir() could return the file or
directory as-is, but when editing a file, then the symbolic link resolve
would apply.
--
A fine is a tax for doing wrong.  A tax is a fine for doing well.
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

On 2021-09-20, Bram Moolenaar wrote:

On 2021-09-19, John Little wrote:

There are cases where the resolution of symbolic names causes problems.

Agreed.
I have a plugin that needs to search for parent directories and for
files in those parent directories. I used to use finddir() and
findfile() for this, but then discovered that they, too, resolve
symbolic links. This was no good when running the plugin in
a subdirectory of one of the /usr/src/linux-headers-* directories.
Many of those subdirectories are symbolic links to some other
/usr/src/linux-headers-* subdirectory, so my plugin would find the
wrong parent directory. I wound up writing my own FindDir() and
FindFile() that textually traverse $PWD/.

When I found the problem, I made notes on it that I thought would be
enough to reproduce it.  But now I can't.  And I fixed the problem
before I put the plugin in version control, so I can't just revert
to that earlier version, although I can probably reconstruct it.
I'll take another stab at it tomorrow or Wednesday.

I would think that findfile() and finddir() could return the file or
directory as-is, but when editing a file, then the symbolic link resolve
would apply.

That's what I'm seeing now, too, although it contradicts my notes on
the problem.  I may have misinterpreted my findings at the time.
I'll let you know what I find.
Regards,
Gary

I have just encountered this issue.
Every time I run a script it produces a new timstamped output directory. I make symbolic links from the latest built files back to the current directory. I want to be able to reopen these linked files in vim and for it to pick up the latest files through the symbolic link.
Instead the files are not opened via the link, and I get either the old files or nothing if the old files are gone.
I think we ought to be able to save the symlink paths to a vim session file to restore via the updated links.

I want to add to the report.
vim ~/symlink-to-dir/file resolves the symlink and shows me ~/real-dir/file in the status line and in echo expand('%').
The conditions for the bug are (~/.vim/ and ~/.vimrc are absent or empty):

the symlink must be a link to a directory; direct link to a file doesn't trigger the bug; vim ~/symlink-to-file shows ~/symlink-to-file;
the bug is triggered only in shell command line; vim with :ed ~/symlink-to-dir/file shows the symlink, not the resolved target;
the path must be absolute: vim ~/symlink-to-dir/file triggers the bug, vim ../symlink-to-dir/file doesn't.

I reproduced the bug with different versions of vim on different Linux systems. I reproduced the bug with vim 7.4 on CentOS with absent ~/.vim/ and ~/.vimrc. I triggered the bug with vim 9.0 at Debian 12.4 with additional condition: :set laststatus=2 in ~/.vimrc; absent laststatus or other values (0 or 1) don't trigger the bug.
A script to reproduce the bug:
#! /bin/sh
set -e
cd "$HOME"

trap 'set +e; rm -rf .vim .vimrc test-dir test-syml;
    mv .vim-backup .vim; mv .vimrc-backup .vimrc' EXIT HUP INT QUIT TERM

mv .vim .vim-backup
mv .vimrc .vimrc-backup
rm -rf test-dir test-syml

echo set laststatus=2 > .vimrc

mkdir test-dir
echo Test >test-dir/test-file

ln -s test-dir test-syml
vim -c "echo expand('%')" "$HOME"/test-syml/test-file

Expected output (both in the status line and from expand('%')) in vim: ~/test-syml/test-file; the actual output is ~/test-dir/test-file.

Just to add weight to the argument that vim ought to have an option not resolve symlinks:
I often set up directory structures more or less like this:
~
└─ projects
   └─ foo-project
      ├─ repo  -> ~/git/foo-project
      ├─ cloud -> ~/securely-backed-up/projects/foo-project
      │  ├─ cfg      # my dev configuration
      │  └─ secrets  # keys and stuff banished from the repo
      └─ # logs and unimportant temporary files

But paths that get logged are often relative to the directory that I run the project in, which is typically ~/projects/foo-project/repo. I'd like it, when vim has been launched from that same directory, to be able to quickly view those paths, but when they're something like ../cloud/cfg/blah.conf, that doesn't work, because when vim launches it sets getcwd() to ~/git/foo-project instead. 😒
For a decade or two, I've just gritted my teeth and corrected the path manually, but today is one of those days when I've also done a websearch, to see whether there's finally an option to get vim to do what I want.
I hope there will be some day. I disagree with @chrisbra: I think such a setting is quite warranted. Part of the beauty of vim is how configurable it is, and this setting would make many users' lives easier.


fix the legacy counter namespace

The prefix for counters in statsd was "counters" which is not reflected in the legacy namespaces for graphite in gostatsd. The reference to this in statsd can be found here

From the reference:
In the legacy setting rates were recorded under stats.counter_name directly, whereas the absolute count could be found under stats_counts.counter_name.

This (with an optional suffix) is how it's emitted currently when legacy is enabled:

rate: stats.name
absolute count: stats_counters.name

It would change to:

rate: stats.counters.name
absolute count: stats_counters.name

Personally I don't think either is great, and if I was running a graphite backend, I'd likely turn off legacy mode entirely (with suitable migration, etc)

Thank you for the note @tiedotguy! After looking through the code again, I believe setting the legacy_namespace variable to false in the config is exactly what I needed. Thank you for pointing me in the right direction with this. I am closing this PR as it is not needed.


Fix default scenario links

The http links for XKCD and Apple trailers don't work which gives a bad first impression when the default thing doesn't work :)
Using HTTPS they work fine.

These were the errors I had: https://gist.github.com/Draiken/9ed1fd210158e316b43696d4af5ca0e6
Maybe they were unrelated to the URL and I just got lucky that after I changed to HTTPS they started working again...

I think those errors are not related to using SSL or not, but changing to SSL also saves a redirect 😄
Thanks!


fix: 'promises must be awaited' sonar issues

Issue
Sonar Cloud introduced a new TypeScript rule 'Promises should not be misused' and this makes our quality gate fail due to new issues reported.
Description

Introduce the corresponding eslint rule @typescript-eslint/no-floating-promises so developers see the issue locally
Add .eslintrc files in each module to fix the following error when enabling the above eslint rule:

Error: Error while loading rule '@typescript-eslint/no-floating-promises': You have used a rule which requires parserServices to be generated. You must therefore provide a value for the "parserOptions.project" property for @typescript-eslint/parser.

Add new tsconfig.esbuild.json to include test sources to fix errors like:

packages/yaml/test/yaml-document.test.ts
  0:0  error  Parsing error: ESLint was configured to run on `<tsconfigRootDir>/test/yaml-document.test.ts` using `parserOptions.project`: <tsconfigRootDir>/packages/yaml/tsconfig.json
However, that TSConfig does not include this file. Either:
- Change ESLint's list of included files to not include this file
- Change that TSConfig to include this file
- Create a new TSConfig that includes this file and include it in your parserOptions.project
See the typescript-eslint docs for more info: https://typescript-eslint.io/linting/troubleshooting#i-get-errors-telling-me-eslint-was-configured-to-run--however-that-tsconfig-does-not--none-of-those-tsconfigs-include-this-file

Fix issues reported corresponding to the new rules

Thanks @tobiasqueck for catching the redundant file. I removed it.


Add CLS token pooling support.

This PR adding support for token pooling for models like BGE-M3.
model_id = "BAAI/bge-m3"
{:ok, model_info} = Bumblebee.load_model({:hf,  model_id})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, model_id})
serving =
      Bumblebee.Text.text_embedding(model_info, tokenizer,
        output_attribute: :hidden_state,
        output_pool: :cls_token_pooling,
        embedding_processor: :l2_norm
      )

Nx.Serving.run(serving, "A long text to test the embeddings")
%{
  embedding: #Nx.Tensor<
    f32[1024]
    [-0.04741977900266647, -0.02085975557565689, -0.028225498273968697, 0.02299957536160946, -0.011559668928384781, -0.07012897729873657, 0.03340381383895874, -0.01976216770708561, -0.03233117237687111, 0.006122056394815445, -0.010670339688658714, -0.013755269348621368, 0.022028403356671333, 0.01282929815351963, -0.011613521724939346, -0.03025030717253685, -0.0033455477096140385, -0.020186245441436768, -0.003963503520935774, -0.030614720657467842, -0.04809923842549324, -0.04979144409298897, -0.005613392218947411, 0.03222518414258957, 0.008074230514466763, 0.04548024386167526, -0.04274187982082367, -8.126439643092453e-4, 0.006089380942285061, -0.013495399616658688, 0.014120037667453289, 6.303095142357051e-4, 0.01951751671731472, -0.021895553916692734, 0.029148466885089874, -0.03433115407824516, 0.02577170729637146, 0.0065758670680224895, -0.023974351584911346, -0.032213930040597916, 6.08053058385849e-4, 0.011225073598325253, 0.00818843673914671, -0.044494032859802246, 0.009402443654835224, -0.02151455357670784, 0.02456141822040081, -0.04354599490761757, -0.03464091941714287, ...]

Bellow are the results from the python implementation (i am using only the "dense" output and not the sparse one)
In [16]: from FlagEmbedding import BGEM3FlagModel
In [17]: model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True)
In [18]: sentences_1 = "A long text to test the embeddings"

In [19]: embeddings_1 = model.encode(sentences_1, batch_size=12, max_length=8192 )['dense_vecs']

In [20]: embeddings_1
Out[20]:
array([-0.0474  , -0.02089 , -0.0282  , ..., -0.002155, -0.02812 ,
        0.05673 ], dtype=float16)

I believe the small differences are because of different implementation of floating point between python <> elixir.

@jonatanklosko let me know if this looks good. For testing i didn't saw anything regarding :mean_pooling so I didnt add one and i was thinking actually what is testable in that case.

@jonatanklosko thank you for the comments. I changed the code to reflect them. Let me know if this is good!


adding support to embedded fonts as byte array

I have added support to embedded fonts (from []byte). With this you can use "https://github.com/jteeuwen/go-bindata" to create the files from fonts.
I already it with success in my project:
https://github.com/prsolucoes/godiffexporter/blob/master/main.go
Advantages:

Everything already is on memory
The binary can be distributed without font files

Check my Makefile to see an example of how to generate the file with embedded fonts:
https://github.com/prsolucoes/godiffexporter/blob/master/Makefile
Thanks.

Thanks, @prsolucoes, for a useful contribution. I'll merge it in when I think of the best way to demonstrate it for end-users. Probably this will be in the form of a test with sufficient  documentation to show how a Go resource file is created.
Nice work!

Hi,
So i made the tests now to you merge it.
Thanks.

Thanks @prsolucoes
I isolated the rather large binary files to help with editing the test source file. Also, I used the calligra font to avoid copyright issues.

@prsolucoes If you would like to be acknowledged in the README, what name should I use?

Hi,
Thanks. You can use my name "Paulo Coutinho" if you want.