Make memory usage tracking thread-safe

Fixes #296. This keeps track of allocation sizes by adding a header to
every allocation holding the size. The difference between the memory
usage we've last reported to V8 and the current memory usage is tracked
in xml_memory_diff, and protected by xml_memory_mutex. It is
frequently synchronised by xml_memory_cb, which is called on the main
thread through xml_memory_handle.

Wrt code style, I prefer the inline functions with semantic names to encapsulate the raw pointer arithmetic. You might consider that a matter of taste, but if you aim for understandable code, I think names for these computations would help. I know that on some very exotic architectures the conversion between pointer and integer might be cause for concern as well, but I doubt libxml would compile on any such architecture in any case.
The uv_unref nicely avoids the problem of the stale async. Hadn't known about that before, and completely missed it while reading nathan7/libxmljs@2b0224a7ad4e845b1145f01c67a83932d580578c.
There is one thing which causes great concern for me: you are using the async for all memory adjustment reports, even on the main thread. While I have verified that it is indeed possible to send an async from the event dispatch thread, the handler will only be run once control returns to the main loop. So if you have some highly synchroneous work load, i.e. some script which spends hours composing complicated XML documents without ever returning control back to the event loop, then V8 won't be notifier of memory changes in all that time. I assume this might be a problem.
If there is a reason why this is not a problem, then I wonder whether we can drop the v8::Isolate::GetCurrent() == 0 and v8::V8::IsDead() case distinctions. If someone is alive to execute an async sent to the main event loop, then obviously the engine is not dead yet.
You should probably cherry-pick gagern/libxmljs@43d035271babb90039f3c0760be57db3ec1aec07 into your code, or simply reproduce its effect. After all, having thread-safe memory handling is one aspect, but making the rest of libxml thread-safe as well is important for #296, too.

I know that on some very exotic architectures the conversion between pointer and integer might be cause for concern as well, but I doubt libxml would compile on any such architecture in any case.

uintptr_t is guaranteed to be the same size as a pointer, and guaranteed to be valid to cast pointers to and from.

There is one thing which causes great concern for me: you are using the async for all memory adjustment reports, even on the main thread. While I have verified that it is indeed possible to send an async from the event dispatch thread, the handler will only be run once control returns to the main loop. So if you have some highly synchronous work load, i.e. some script which spends hours composing complicated XML documents without ever returning control back to the event loop, then V8 won't be notifier of memory changes in all that time. I assume this might be a problem.

Acknowledged — I had some test code that did take care of this, but it used pthreads directly, which wouldn't go over well on Windows.

If there is a reason why this is not a problem, then I wonder whether we can drop the v8::Isolate::GetCurrent() == 0 and v8::V8::IsDead() case distinctions. If someone is alive to execute an async sent to the main event loop, then obviously the engine is not dead yet.

Yep, I can't actually get that case to trigger. Tests run just fine with this, both libxmljs and node-libxslt:
diff --git a/src/libxmljs.cc b/src/libxmljs.cc
index c7a5d8a..a91eab1 100644
--- a/src/libxmljs.cc
+++ b/src/libxmljs.cc
@@ -37,12 +37,12 @@ NAUV_WORK_CB(xml_memory_cb) {
 #if (NODE_MODULE_VERSION > 0x000B)
     if (v8::Isolate::GetCurrent() == 0)
     {
-        return;
+        abort();
     }
 #endif
     if (v8::V8::IsDead())
     {
-        return;
+        abort();
     }

     uv_mutex_lock(&xml_memory_mutex);

The uv_unref nicely avoids the problem of the stale async. Hadn't known about that before, and completely missed it while reading nathan7/libxmljs@2b0224a.

Thanks!

You should probably cherry-pick gagern/libxmljs@43d0352 into your code, or simply reproduce its effect. After all, having thread-safe memory handling is one aspect, but making the rest of libxml thread-safe as well is important for #296, too.
Yep, I'll do that, though I'm somewhat wary of patching our vendored lib.

I've made some more progress, so, status report!

Wrt code style, I prefer the inline functions with semantic names to encapsulate the raw pointer arithmetic. You might consider that a matter of taste, but if you aim for understandable code, I think names for these computations would help.

Concern noted, and I'll take a look at doing that.

I've patched this up now, the code does look a lot more readable now, thank you!

You should probably cherry-pick gagern/libxmljs@43d0352 into your code, or simply reproduce its effect. After all, having thread-safe memory handling is one aspect, but making the rest of libxml thread-safe as well is important for #296, too.

Yep, I'll do that, though I'm somewhat wary of patching our vendored lib.

Done!

If there is a reason why this is not a problem, then I wonder whether we can drop the v8::Isolate::GetCurrent() == 0 and v8::V8::IsDead() case distinctions. If someone is alive to execute an async sent to the main event loop, then obviously the engine is not dead yet.

I'm still somewhat wary of taking that code out, even though it appears to do nothing. In particular, when I fix my patches to be aware of the synchronous case, this code might run while node is shutting down. It's harmless in the meanwhile.

There is one thing which causes great concern for me: you are using the async for all memory adjustment reports, even on the main thread. While I have verified that it is indeed possible to send an async from the event dispatch thread, the handler will only be run once control returns to the main loop. So if you have some highly synchronous work load, i.e. some script which spends hours composing complicated XML documents without ever returning control back to the event loop, then V8 won't be notifier of memory changes in all that time. I assume this might be a problem.

Acknowledged — I had some test code that did take care of this, but it used pthreads directly, which wouldn't go over well on Windows. Advice is definitely welcome on that.
I was considering having a thread-local "are we in V8" variable, but I'm not sure that's worth the complexity.

This remains a TODO, and I'm working on figuring out how to do this in the nicest way possible with the tools libuv gives us.

To special-case the V8 thread: you can use uv_thread_self when initializing the library to store a reference to the main thread. Then later on you can compare that against the current thread to see whether you are on the V8 thread. Note that in older versions of libuv, thta function returns some integer type.

To special-case the V8 thread: you can use uv_thread_self when initializing the library to store a reference to the main thread. Then later on you can compare that against the current thread to see whether you are on the V8 thread.

Done!

Note that in older versions of libuv, that function returns some integer type.

While pthread_t and HANDLE (on Windows) are both integer types, this appears to be wrapped in uv_thread_t down to the version of libuv that node 0.8 uses.

And uv_thread_equal was introduced in libuv/libuv@59658a8de7cc05a58327a164fd2ed4b050f8b4f4 and is contained in libuv versions starting from 1.0.0. Before that uv_thread_self used to return an unsigned long which it obtained by a brute force cast. You will notice that Travis CI fails on 0.10 as well for the current state of affairs here.

Personally I find it hard to review a sequence of changes if they are all squashed into a single commit again and again. It means I can't concentrate on what changed since the last iteration. But that's a matter of taste, I guess.

@gagern

Personally I find it hard to review a sequence of changes if they are all squashed into a single commit again and again. It means I can't concentrate on what changed since the last iteration. But that's a matter of taste, I guess.

If you want to review this as individual commits, check out https://github.com/polotek/libxmljs/compare/master...nathan7:memory-tracking-wip.
I try to keep my pull requests limited to semantic commits, but you I always keep the full history on the -wip branch. Sorry for not making that clear before.

If there is a reason why this is not a problem, then I wonder whether we can drop the v8::Isolate::GetCurrent() == 0 and v8::V8::IsDead() case distinctions. If someone is alive to execute an async sent to the main event loop, then obviously the engine is not dead yet.

I'm still somewhat wary of taking that code out, even though it appears to do nothing. In particular, when I fix my patches to be aware of the synchronous case, this code might run while node is shutting down. It's harmless in the meanwhile.

This code appears to be necessary in the synchronous case, so I'll get on that:
OK: 323 assertions (1792ms)
[Thread 0x7ffff4be0700 (LWP 15310) exited]
[Thread 0x7ffff53e1700 (LWP 15309) exited]
[Thread 0x7ffff5be2700 (LWP 15308) exited]
[Thread 0x7ffff63e3700 (LWP 15307) exited]
[Thread 0x7ffff6be4700 (LWP 15306) exited]

==== C stack trace ===============================

 1: V8_Fatal
 2: v8::internal::Isolate::Current()
 3: v8::Isolate::GetCurrent()
 4: NanAdjustExternalMemory(int)
 5: libxmljs::xml_memory_update(long)
 6: libxmljs::xmlMemFreeWrap(void*)
 7: xmlCleanupCharEncodingHandlers
 8: xmlCleanupParser
 9: libxmljs::LibXMLJS::~LibXMLJS()
10: ??
11: ??
12: __libc_start_main
13: _start

Program received signal SIGILL, Illegal instruction.
v8::base::OS::Abort () at ../../deps/v8/src/base/platform/platform-posix.cc:212
212	    V8_IMMEDIATE_CRASH();
=> 0x000000000126d41f <v8::base::OS::Abort()+15>:	0f 0b	ud2

@gagern
I've moved the V8 aliveness checks onto the synchronous case, and renamed and documented the new allocation approach.
I think we're pretty much there, but as usual, a second brain and pair of eyes is greatly appreciated :heart:

Note that in older versions of libuv, that function returns some integer type.

While pthread_t and HANDLE (on Windows) are both integer types, this appears to be wrapped in uv_thread_t down to the version of libuv that node 0.8 uses. I'm not sure what issue I should be seeing here. I'm using uv_thread_equal(const uv_thread_t*, const uv_thread_t*) for my comparison.

It seems this one has come back to bite me, breaking compilation on node 0.10 (UV_VERSION_MAJOR == 0)

@gagern
I've fixed the lack of uv_thread_equal on pre-1.0 libuv using your approach.

It appears I need to do some more libuv compatibility work, because uv_thread_self apparently didn't exist in node 0.8.
../src/libxmljs.cc: In function ‘void libxmljs::xml_memory_update(ssize_t)’:
../src/libxmljs.cc:55:49: error: ‘uv_thread_self’ was not declared in this scope
     uv_thread_t current_thread = uv_thread_self();
                                                 ^
../src/libxmljs.cc: In constructor ‘libxmljs::LibXMLJS::LibXMLJS()’:
../src/libxmljs.cc:159:33: error: ‘uv_thread_self’ was not declared in this scope
     xml_thread = uv_thread_self();
                                 ^

While uv_thread_self appears to have started existing in node 0.8.10, it appears to be broken on pre-1.0 libuv (it doesn't actually work with uv_thread_equals).
That's a pickle.

It appears that as long as we're comparing return values from uv_thread_self, we're in the clear.
The issue only shows up when you compare a "real" uv_thread_t (such as the ones returned by uv_thread_create) to one returned from uv_thread_self.
I think that's fine, we don't do that, and it only shows up on a rather narrow platform / node version combo.
The worst that could happen if we did that is that our memory usage is accidentally reported asynchronously in the synchronous case. But we don't do that.
Just a comment noting this would do here, really.

Looks good. The type uv_thread_t for xml_thread has me somewhat worried, since uv_thread_self will sometimes return a long instead which might make compilers balk. Not sure which versions are affected. Apparently not the current version of either node 0.10 or 0.12, but probably some past versions. I wonder whether you should be including a replacement for uv_thread_self as well in libuv pre 1.0. And of course that would be a candidate for nauv in nan as well.
Your header doesn't really have to include <nan.h> but doing so should cause no harm either. I'd really like to do a performance test soon, but apart from that things look good.

I've implemented uv_thread_self for pre-0.9.2 node versions, using the pre-1.0 behaviour. I'm not sure about the unsigned long matter — there's a node commit by @tjfontaine reverting the libuv change that switched the return type, so that might be worth checking on.

If someone on Windows, with node 0.8, could verify that this doesn't crash the libxmljs tests, and does crash the node-libxslt tests, that'd be splendid:
diff --git a/src/libxmljs.cc b/src/libxmljs.cc
index f306c24..3b3e136 100644
--- a/src/libxmljs.cc
+++ b/src/libxmljs.cc
@@ -73,6 +73,7 @@ NAN_INLINE void xml_memory_update(ssize_t diff) {
         NanAdjustExternalMemory(diff);
     }
     else {
+        abort();
         uv_mutex_lock(&xml_memory_mutex);

         xml_memory_diff += diff;

As far as I can tell, uv_thread_t is always unsigned long or larger to match the OS. I think we can always safely upcast OS thread IDs. I'd love to test this across every node version since 0.8.0, on every OS, but I can really only do that for Linux and OS X currently.

Results of the giant Travis test, across 102 node versions:

0.8.13: Ships with an npm so broken even npm test doesn't work, but, rest assured, that actually does work locally if I run it by hand.
0.9.{9..12}, 0.10.0: Mysteriously fails due to node::MakeCallback(v8::Handle<v8::Object>, v8::Handle<v8::Function>, int, v8::Handle<v8::Value>*) being AWOL, not our problem.
0.11.{0..12}: Everything breaks, C++ errors of Wagnerian fiercety are produced by Nan. That's not my department.
0.11.14: Yep, that's my fault: ../src/libxmljs.cc:56:53: error: ‘uv_thread_equal’ was not declared in this scope
iojs: Whoops. The node-gyp I bundled doesn't deal with iojs correctly. I can assure you that everything works just fine with the right node-gyp, however.

Meanwhile, my Mac Mini is busily installing the same set of node versions. I'll report back on that once testing kicks off.

I'm currently cooking up an asynchroneous version of Document.fromXml and reading a 30+M file on my system in parallel to test performance. At least that's the idea; right now I'm doing this on my code and the TLS variable mysteriously changes its value. So either I've got some memory corruption somewhere, or a bug in libuv or pthreads or so. I consider the former far more likely. Investigating…

OK, I believe we do have a performance problem with this approach. The biggest XML file on my system is /usr/share/foomatic/db/source/opt/gutenprint-ijs.5.2-pagesize.xml with 34M. I parsed 20 times in parallel:
var testInput = fs.readFileSync(testFile);
for (var i = 0; i < 20; ++i)
  libxmljs.Document.fromXmlAsync(testInput, function(err, doc) {
    if(err) throw err; console.log("Got document."); });

The result:
     my #300   your #302
real 0m23.811s 0m24.658s
user 0m24.483s 0m40.894s
sys  0m04.372s 0m16.335s

The #300 test case is in asyncTest1, the one for #302 in asyncTest2. So while the wall clock time appears to be the both for both, your branch will spend a lot more CPU resources, in user as well as in system space. I still prefer #300. Have I convinced you?

The way I read this, libxml seems to essentially serialize access to the parser, so there is only ever one thread accessing the parser, which is the bottleneck in my use case. But with your code, the event dispatch thread is almost as busy as that worker, while in mine it appears to be mostly idle, and could do other work. Of course, it is possible that if the V8 thread had something to do, your async events would simply coalesce, and therefore have almost no effect on the performance of the main thread either. I wonder how to test this theory.

Those results are indeed quite damning. I'll take a look later today, and see if I can work on that.
If I can be convinced that your approach is better, all the better! Less work for me [=

I've modified the tracking a little, to prevent overly frequent synchronisation: external memory is allowed to diverge up to 64k.
That might be enough to solve the CPU usage issues.
I'm also looking into hooking V8::AddGCPrologueCallback (accessible in Nan through NanAddGCPrologueCallback), though I'm not entirely sure that's useful at all.

Nope, that was pointless.
      #300     new #302  old #302
real  12.672s  18.229s   20.901s
user  27.730s  39.243s   45.397s
sys    5.827s  17.710s   22.030s

@nathan7 @gagern Ok, so after all this research and attempts, what is the patch I should be reviewing and looking to merge?

@defunctzombie If I understand things correctly, nathan is primarily concerned with the complexity of #300. I'd say please have a look at both, let us know whether you share that opinion, and how much performance you'd be willing to sacrifice in order to avoid it.

I doubt adding a gc prologue callback will be of any use: if it gets executed, the system has already decided to run a gc sweep, which as far as I understand things is the very same decision for which it requires memory adjustments.

@defunctzombie I'm concerned with #300 being more complex and requiring cooperation from the embedder, which is a breaking change at the C++ interface but not the JS interface.
I'm not sure I am qualified to review that code. You should probably find someone more well-versed in C++.

Well, up till now libxmljs made no attempt to support worker threads. Any third party code which used it in this fashion, like node-libxslt did, did so either out of ignorance or knowingly hoping for things not to break. The way I see it, this is the time you are first offering multi-threading support at all, and it's up to you to set down the guidelines for how you do it.
It's only after you start releasing this feature that you should stick to the API once chosen. Which is of course easier if you from the beginning require implementers to use sentinels or whatever, than if you guarantee that they won't have to worry about anything.

@gagern @nathan7 I took a look at both patches and I personally am leaning toward the one where the user doesn't have to do anything special. I think the Sentinel stuff in patch #300 could lead to more issues and opens up API surfaces for us to maintain.
For this patch, is the single thread perf much worse? Can we maybe detect single thread use and avoid dropping perf unless we know user needs the multi-thread stuff?
This patch also needs tests for this so that it is not broken in future updates.

I'll also add that I like the features thing from patch #300 and maybe that could be a separate PR? Dunno if anyone actually cares about them but it seemed potentially useful.

I took a look at both patches and I personally am leaning toward the one where the user doesn't have to do anything special. I think the Sentinel stuff in patch #300 could lead to more issues and opens up API surfaces for us to maintain.

OK, then let's aim for that.

For this patch, is the single thread perf much worse? Can we maybe detect single thread use and avoid dropping perf unless we know user needs the multi-thread stuff?

The uv_thread_equal(&xml_thread, &current_thread) case should do just that, as far as our own memory handling code is concerned. I fear that the biggest performance impact would likely come from libxml itself being compiled with threading support, since that means locks in many places. But that effect is independent of which approach we use to tie in memory management, and hopefully not enough to warrant compiling the library in two different versions. I haven't done any benchmarks yet.

This patch also needs tests for this so that it is not broken in future updates.

We could build on asyncTest2 for that. Either turn it into a compiled-code test case or add some proper error-storing code (without using v8 data structures) to make fromXmlAsync a proper library addition. I'd prefer the latter.

I'll also add that I like the features thing from patch #300 and maybe that could be a separate PR? Dunno if anyone actually cares about them but it seemed potentially useful.

Yes, I can extract that part of #300 and file it separately.

In my version of Nathan's memory-tracking branch, I included the list of features and implemented an asynchroneous parser including error collecting. It doesn't work yet, there seems to be some memory trouble somewhere, but there is an even more severe problem: up until now, we had not set HAVE_PTHREAD_H nor HAVE_WIN32_THREADS, thus essentially disabling threading support again. With that define added in (in gagern/libxmljs@fc66a1cc21e9d4bfab34ea712e94a10768566d8d) things changed dramatically: now many things are suddenly thread-local, see the global state object for comparison. In particular, the memory functions are now thread-local! This means that we have to call xmlMemSetup in every worker thread! With the approach from #300 that would be an easy thing to do: simply add some more calls to the constructor of the WorkerSentinel class. But I don't see how we can achieve this without cooperation from the calling code which set up the worker thread. So personally I'd see this as a strong indication that we should switch to #300 instead of this here, but if you can think of another approach, I'd be glad to hear that.

What happens if we just remove the code that tells v8 about the amount of externally allocated memory? Also, what is it that is starting worker threads? I am going through the original issue and trying to figure out why this is even a thing in the first place. Are worker threads being launched by some other lib? Are these worker threads uv workers or some other thread?
My main concern was with adding API around this when it seems to be a very niche and scoped use case? Maybe that is not the case but it certainly seems that way.
Can we just use global xmlMemSetup and memory tracking? What is the issue with keeping memory tracking global vs per thread?

What happens if we just remove the code that tells v8 about the amount of externally allocated memory?

We could end up with a bunch of stale documents which don't get garbage-collected because v8 doesn't know it's eating tons of memory.

Also, what is it that is starting worker threads? I am going through the original issue and trying to figure out why this is even a thing in the first place. Are worker threads being launched by some other lib? Are these worker threads uv workers or some other thread?

albanm/node-libxslt#9 was the starting point. node-libxslt depends on libxmljs to provide libxml bindings, but performs its operations asynchroneously. It's using the uv-based worker implementation from nan.
There might be other libraries doing something similar without my knowledge. After all, there is quite a number of packages depending on libxmljs, and I haven't had a closer look at what all of these do internally.

My main concern was with adding API around this when it seems to be a very niche and scoped use case? Maybe that is not the case but it certainly seems that way.

It might be a niche use case now, I don't know. It might become more common in the future, as more wrappers emergw around libs that depend on libxml2. The node way to do expensive things is asynchroneously, the way I understand it.
The way I see it, you have to decide whether you want to officially support multi-threading. If you do, I believe you need some API around this, no matter how uncommon the use case is at the moment. You won't have to worry about that new API requirement breaking any existing code, since with node 0.12 the current approach is broken in any case, and as long as you ensure that not using the new API breaks loudly enough, that should be fine. If it's only node-libxslt which is affected, then I can gladly take care of getting that adapted.
Or you decide not to support multi-threading, which means node-libxslt will either have to drop (or fake) its asynchroneous operations, or change its dependency to something which supports multi-threading. Seeing as I'm reasonably happy with #300, I'd suggest forking libxmljs to provide that functionality for node-libxslt. So the question is this: do you want a fork of this project to deal with the threading issues, or would you rather avoid the confusion such a fork brings with it?

Can we just use global xmlMemSetup and memory tracking? What is the issue with keeping memory tracking global vs per thread?

Looking at the configure.ac of libxml2, it seems as if WITH_THREADS can only get set if either HAVE_WIN32_THREADS or HAVE_PTHREAD_H gets set at the same time. So the current state of the config.h in this branch here, with WITH_THREADS set but neither of the other two, can't happen in a properly autotooled build, and is therefore unsupported. It won't result in reentrant code, since it won't have access to synchronization primitives like mutexes.


pushing bad PHPCS code

Invalid PHPCS sniff(s) specified in options or options file. Those have been ignored temporarily. Please update the options so that scanning can continue as expected.

Option name: --phpcs-sniffs-exclude
Invalid sniff(s): WordPress.Files.FileName
(commit-ID: 1177f1f4b21c05336d8221d3a6e39669271e3acf).

PHP Syntax Errors Found
Scan performed on the code at commit b2c5a7c86494a06b12cfed753a7621769780222b (view code).

Error: syntax error, unexpected |, expecting variable (T_VARIABLE)
https://github.com/firstandthird/wordpress-template-2022/blob/b2c5a7c86494a06b12cfed753a7621769780222b/vendor/symfony/polyfill-mbstring/bootstrap80.php#L15

PHP Syntax Errors Found
Scan performed on the code at commit 1c6a974548533b909361ed4996e4d6ab8b4af474 (view code).

Error: syntax error, unexpected |, expecting variable (T_VARIABLE)
https://github.com/firstandthird/wordpress-template-2022/blob/1c6a974548533b909361ed4996e4d6ab8b4af474/vendor/symfony/polyfill-mbstring/bootstrap80.php#L15


fixed alpha not work

Hi endroid:
I want a qr code with alpha but not success when i use the writeFile to save a PNG qr code.
Then I found an example in the following page.
http://php.net/manual/zh/function.imagefill.php
$new   = imagecreatetruecolor(320, 320);
$color = imagecolorallocatealpha($new, 0, 0, 0, 127);
imagefill($new, 0, 0, $color);
imagesavealpha($new, TRUE); // it took me a good 10 minutes to figure this part out
imagepng($new);

So, i try to edit the createInterpolatedImage function of PngWriter class, it looks like this
private function createInterpolatedImage($baseImage, array $data, QrCodeInterface $qrCode)
{
    $image = imagecreatetruecolor($data['outer_width'], $data['outer_height']);
    $backgroundColor = imagecolorallocatealpha($image, $qrCode->getBackgroundColor()['r'], $qrCode->getBackgroundColor()['g'], $qrCode->getBackgroundColor()['b'], $qrCode->getBackgroundColor()['a']);
    imagefill($image, 0, 0, $backgroundColor);
    imagecopyresampled($image, $baseImage, $data['margin_left'], $data['margin_left'], 0, 0, $data['inner_width'], $data['inner_height'], imagesx($baseImage), imagesy($baseImage));
    imagesavealpha($image, true);
    return $image;
}

then i got qr code with alpha, hope help you

Hi @mh1988, looks good, thank you!

I've created an exception for this, if background alpha channel is 0. Do you think, it's a good idea to do this like I did? Refer to https://github.com/endroid/qr-code/issues/264 and https://github.com/endroid/qr-code/pull/265

Hi @Huseriato thank you. I looked at it and I think there should be no problem with that approach. I will tag soon :)


Make it possible to run container unprivileged

I have pushed my changes.
This is how I configure the service now:
  doa:
    image: nbisweden/ega-doa:latest
    ports:
      - 8080:8080
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        window: 120s
    environment:
      - CERT_PASSWORD
      - POSTGRES_PASSWORD=${DB_LEGA_OUT_PASSWORD}
      - S3_ACCESS_KEY=${MINIO_ACCESS_KEY}
      - S3_SECRET_KEY=${MINIO_SECRET_KEY}
    configs:
      - source: rootCA.pem
        target: /etc/ega/ssl/CA.cert
      - source: server.p12
        target: /etc/ega/ssl/server.cert
      - source: client.pem
        target: /etc/ega/ssl/client.cert
      - source: client-key.der
        target: /etc/ega/ssl/client.key
      - source: jwt.pub.pem
        target: /etc/ega/jwt/key.pem
      - source: ega.sec.pem
        target: /etc/ega/crypt4gh/key.pem
      - source: ega.sec.pass
        target: /etc/ega/crypt4gh/key.pass


Removed unneeded image borders

Now that all item icons have borders built in, we don't need the styling to manually add the borders.

.deploy

.deploy


Crab mascot name suggestion

(he needs a name)
Suggestion Template
Suggestion: Crambon
Reason*: Cool name

*Not obligatory

Suggestion: Crambon
Reason: Cool name

Suggestion: Xerxes, the Cruel


README: contributing

Add Contributing section to README with link to developer guide.

Cool! We have a HACKING.md file in this project, so I think the link should go there. Small nit: I usual try do format the commit message a la: <area>: <message <= 50 chars>, so HACKING.md: add developer guide link.


Events is a blank page

https://kubernetes-contributor.netlify.com/events/ is a blank page

/remove-lifecycle stale
this is still true

/lifecycle frozen


IDVA6-1101 mobile view layout issues search section

Link to Jira:
https://companieshouse.atlassian.net/browse/IDVA6-1101
Changes:

A wrapper div adde to modify behaviour or the included elements
Additional styles added

CI: No security warnings found

CI: No security warnings found


default phrase behaviour

Hi,
The documentation for stars says:

phrase
Text string to search for. Defaults to returning all starred repositories from the specified user

But this doesn't seem to be the case.
library(stellar)
#>
#> Attaching package: 'stellar'
#> The following object is masked from 'package:graphics':
#>
#>     stars

stars(user = "MHenderson", phrase = "stars")
#> [1] ropenscilabs/stellar :
#> Search your github stars in R
#>
#> choose a repository to open, or anything else to exit:

stars(user = "MHenderson")
#> That text does not appear to describe any starred repositories

Best wishes,
Matthew

Thank you! That's my bad for the implementation of the default. Until I patch this (very soon) you can explicitly set phrase = NULL to achieve this (I think).

Hopefully working now. Thanks again!

Yes, it does. Thanks!


add command executor provider

ref https://github.com/treasure-data/digdag/pull/1315

related to https://github.com/st-tech/digdag/pull/14


In wireframes, include option to not convert to snake case for metadata extraction

When they upload the csv file to extract the metadata, the extract metadata columns should be converted to snakecase. So maybe we have two columns, one for the original column names and the other for the converted names? And have an option to not convert them?
Snake case is the way of naming variables/columns in data and in programming languages. It is in the form of snake_case or this_is_the_name, where everything is lowercase and a _ replaces spaces.
There should be two columns then:

Original Column Name
Converted (or new) Column Name

Age at visit
age_at_visit

Something along these lines @lwjohnst86 ? :)

Mmm, more like, a "global" option to say "convert or not". But, we're actively discussing how this should look like, so maybe hold off on doing any work on it.

@lwjohnst86 Thanks for reviewing. Alright, I will hold off a little bit on this. Let me know whenever you want me to chime in again :)

We decided to start the web app from scratch after the major design/architecture decisions and changes we made. No longer relevant.


Draw a grid/checker pattern on tile selection GUI

Help differentiate different grid settings from each other visually by drawing a grid and checker pattern in the tile selection UI.

Currently drawing an grid overlay on selection GUI. Not sure if enough, but closing for now.


Frontend modification

Ui-modification

Moved this PR to https://github.com/pondersource/nc-sciencemesh


feat: Create LoginViewController (template & unit test)

1- Add Login Template which contain (ViewController - viewModel - viewModelType - .xib)
2- Add Unit test file for LoginViewModel functions

Branches names should be in small letters

Commit type, feat should be in small letters.


Add flag to sct_image to print image headers

Checklist
GitHub

[X] I've given this PR a concise, self-descriptive, and meaningful title
[X] I've linked relevant issues in the PR body
[X] I've applied the relevant labels to this PR
[x] I've applied a release milestone (major, minor, patch) in line with Semantic Versioning guidelines
[X] I've assigned a reviewer

PR contents

[X] I've consulted SCT's internal developer documentation to ensure my contribution is in line with any relevant design decisions
[X] I've added relevant tests for my contribution
[X] I've updated the relevant documentation for my changes, including argparse descriptions, docstrings, and ReadTheDocs tutorial pages

Description

This PR:

Reproduces part of fslhd's C++ source code in a Python script (in accordance with their license).

The most relevant C++ source file is /fsl/src/NewNifti/NewNifti.cc
(see Fslinstallation/SourceCode.)

Wraps the script in an API function, and call API function via sct_image -show-header.
Deprecates sct_image -getorient since -header provides the same results (and more).

The usage of the CLI script is:
# fslhd formatting
sct_image -i t2.nii.gz -header fslhd

# fslhd formatting with tweaks (e.g. combining `dim` into a one-line field)
sct_image -i t2.nii.gz -header
# Alternative way to specify this
sct_image -i t2.nii.gz -header sct

# nibabel formatting (i.e. `print(Nifti1Image.header)`)
sct_image -i t2.nii.gz -header nibabel

The output of -header fslhd should be identical to fslhd: https://www.diffchecker.com/gzaVUfj9
(I've manually run this on a variety of images, both Nifti1 and Nifti2. But, I'm a bit stuck on how to write a good pytest test for this, considering we can't easily compare with fslhd in our test suite. We could compared against cached output, but there are ~60 fields and many possible images, so doing that feels a little awkward.)
Linked issues

Fixes #540.
Will replace -getorient, so it also fixes #3311.

Since this new functionality effectively replaces sct_image -getorient, I'm thinking of removing that option in this PR, which would fix https://github.com/neuropoly/spinalcordtoolbox/issues/3311 and possibly https://github.com/neuropoly/spinalcordtoolbox/issues/3281. But, I would need to investigate those issues further to be certain.

Thanks much! pip install -e . would do the job here.

actually that's what i did (under the sct_venv), but it did't create the launchers

Thanks much! pip install -e . would do the job here.

actually that's what i did (under the sct_venv), but it did't create the launchers

Interesting... I tried it just now and it worked OK. :thinking:

 Steps I tested

Add a dummy command to setup.py

    entry_points=dict(
        console_scripts=[
            '{}=spinalcordtoolbox.compat.launcher:main'.format(x) for x in \
            [
+               'test_command'
                'isct_convert_binary_to_trilinear',
                'isct_minc2volume-viewer',

Reinstall and try command

> test_command
zsh: command not found: test_command
> pip install -e .
Obtaining file:///home/joshua/repos/spinalcordtoolbox
Installing collected packages: spinalcordtoolbox
  Attempting uninstall: spinalcordtoolbox
    Found existing installation: spinalcordtoolbox dev
    Uninstalling spinalcordtoolbox-dev:
      Successfully uninstalled spinalcordtoolbox-dev
  Running setup.py develop for spinalcordtoolbox
Successfully installed spinalcordtoolbox
> test_command
Traceback (most recent call last):
  File "/home/joshua/repos/spinalcordtoolbox/python/envs/venv_sct/bin/test_command", line 33, in <module>
    sys.exit(load_entry_point('spinalcordtoolbox', 'console_scripts', 'test_command')())
  File "/home/joshua/repos/spinalcordtoolbox/spinalcordtoolbox/compat/launcher.py", line 34, in main
    assert os.path.exists(script)
AssertionError
# Tries to go through `compat/launcher.py`, so it's available

The failing RTD is confusing me, as I'm not able to reproduce the error message locally on this branch.
I took a look at the sphinxcontrib-programoutput repo and noticed that another user had recently opened an issue with similar failing output. I'm not sure if it's the same problem or just coincidence, but linking just in case: https://github.com/NextThought/sphinxcontrib-programoutput/issues/50#issuecomment-811091463

i agree caching would be the best approach here. Starting with the most important fields, I would say: dim, q/sform, q/sform_code.

I'm still not quite sure about this kind of test.
All that sct_image -header does is echo the values already stored in the image. So, if any problems were to arise, then the root cause is more likely to be because of the specific image (and not sct_image -header). Since most problems would be data-dependent, I'm not sure it would be useful to write a test that caches values for 1 image.

i agree caching would be the best approach here. Starting with the most important fields, I would say: dim, q/sform, q/sform_code.

I'm still not quite sure about this kind of test.
All that sct_image -header does is echo the values already stored in the image. So, if any problems were to arise, then the root cause is more likely to be because of the image itself (and not sct_image -header). Since most problems would be data-dependent, I'm not sure that a test using cached values would catch anything.

Note to self: We should also run sct_image -header on the relevant image files from #3311 and #3281 to confirm that the problem was with -getorient.


error G4127D1E8: The getter 'StrokeAlign' isn't defined for the class '_FocusedContainer'

Got these 2 error even after i create new project. i dont have another pc to test so this may only me but can you investigate?
`/C:/Users/ValkDokk/AppData/Local/Pub/Cache/hosted/pub.dev/pluto_layout-0.3.0/lib/src/ui/pluto_layout_container.dart(142,30): error G4127D1E8: The getter 'StrokeAlign' isn't defined for the class '_FocusedContainer'. [C:\Projects\flame_demo\build\windows\flutter\flutter_assemble.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(247,5): error MSB8066: Custom build for 'C:\Projects\flame_demo\build\windows\CMakeFiles\abbd1966738140111e8175bf1a2bb3aa\flutter_windows.dll.rule;C:\Projects\flame_demo\build\windows\CMakeFiles\0dfc9ba6e9d86f9ffbdb465a6b13bc3d\flutter_assemble.rule' exited with code 1. [C:\Projects\flame_demo\build\windows\flutter\flutter_assemble.vcxproj]
Exception: Build process failed.`

my setting:
environment:
sdk: ">=2.18.0 <3.0.0"
flutter: 3.3.8

The problem does not occur in 3.3.9.
Perhaps your flutter version is not 3.3.x.
Below is the problem with version 3.5.x.
This is planned to be fixed when updated to stable.
https://github.com/bosskmk/pluto_layout/issues/1


Gobble throwing 'build invalidated' errors when building on Windows

When running gobble build, I get this:
gobble: 02-babel done in 9753ms
gobble: 02-babel done in 9753ms
gobble: 02-babel done in 9753ms
gobble: 02-babel done in 9753ms
gobble: 02-babel done in 9753ms
gobble: 02-babel done in 9753ms
gobble: 03-esperantoBundle done in 768ms
gobble: 07-sorcery done in 541ms
gobble: 04-esperantoBundle done in 626ms
gobble: 08-sorcery done in 447ms
gobble: 05-esperantoBundle done in 583ms
gobble: 09-sorcery done in 387ms
gobble: 06-esperantoBundle done in 545ms
gobble: 10-sorcery done in 395ms
gobble: 24-merge done in 32ms
gobble: 11-moveTo done in 7ms
gobble: 11-moveTo done in 7ms
gobble: 12-merge done in 5ms
gobble: 12-merge done in 5ms
∙∙∙∙∙∙? 13-babel running...
gobble: build invalidated
>>>
Error
    at new GobbleError (C:\Users\Madgvox\Documents\ractive\node_modules\gobble\l
ib\utils\GobbleError.js:6:15)
    at C:\Users\Madgvox\Documents\ractive\node_modules\gobble\lib\nodes\Source.j
s:92:16
    at later [as _onTimeout] (C:\Users\Madgvox\Documents\ractive\node_modules\go
bble\node_modules\debounce\index.js:34:23)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
<<<

gobble: build invalidated
>>>
Error
    at new GobbleError (C:\Users\Madgvox\Documents\ractive\node_modules\gobble\l
ib\utils\GobbleError.js:6:15)
    at C:\Users\Madgvox\Documents\ractive\node_modules\gobble\lib\nodes\Source.j
s:92:16
    at later [as _onTimeout] (C:\Users\Madgvox\Documents\ractive\node_modules\go
bble\node_modules\debounce\index.js:34:23)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
<<<

gobble: build invalidated
>>>
Error
    at new GobbleError (C:\Users\Madgvox\Documents\ractive\node_modules\gobble\l
ib\utils\GobbleError.js:6:15)
    at C:\Users\Madgvox\Documents\ractive\node_modules\gobble\lib\nodes\Source.j
s:92:16
    at later [as _onTimeout] (C:\Users\Madgvox\Documents\ractive\node_modules\go
bble\node_modules\debounce\index.js:34:23)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
<<<

When running gobble, I get this:
gobble: server listening on port 4567
gobble: livereload server running
gobble: 02-babel done in 6213ms
gobble: 03-esperantoBundle done in 732ms
gobble: 04-sorcery done in 539ms
gobble: 01-moveTo done in 3ms
gobble: 05-sorcery done in 3ms
gobble: 19-merge done in 5ms
gobble: 06-moveTo done in 7ms
gobble: 07-merge done in 5ms
gobble: build invalidated (3 files changed). restarting
gobble: build invalidated (6 files changed). restarting

And it stays like that forever. If I go to localhost:4567 it just shows the 'gobbling' message.
I've tried doing a clean npm install, and completely resetting the repository through git. I'm going to try to do some more digging, but I'm not familiar with gobble so I'm not entirely sure what to look for.

Looks like this is probably the same issue as https://groups.google.com/forum/#!topic/ractive-js/5FPJUCzqLNg, so I'll close this.

That's the same issue that was killing the appveyor build. I got it to happen once on my machine, just after a fresh checkout and npm install. After it failed the first time, everything was OK. I came to the conclusion that something about watching windows filesystems is whacky. Is it repeatable for you? Running twice seems to be working for appveyor.

It's repeatable through restarts and reinstalls. I've yet to try completely nuking it, but that's my next step.
It's definitely something with the watcher. If I comment out the test section in the gobblefile it goes into an infinite loop just like in the link above.

I realized my node & npm versions were horribly out of date, but unfortunately bringing them up to date had no noticeable effect on the issue.

Alright, I've narrowed it down to exactly what is causing it to fail in the gobblefile:
The lib.push( sandbox ) on line 88, and the babel transform starting on line 117. If I comment those two out, gobble successfully gets to 'build completed' and I can access the localhost properly. I still can't use it, since the sandbox isn't there and the tests attempt to run in es6, but it's progress.


Our Buddy preparsecode

I was working on adding additional BBC testcases, from the ones that we used/created for the new BBC parser, and ran into a few problems.
Several of the new testcases run fine if not run through preparsecode(), for example https://github.com/joshuaadickerson/BBC-Parser/blob/master/Messages.php#L226 but if run through preparsecode() then we can have an unforced error created by the function.  There is also one that I have not figured out why its failing on Travis but parses fine on a real install pufff.
That lead me to preparsecode() and at first I simply fixed things with using both the pre and post areas on the sides of the code block created by the preg_split (so basically use [$i + 4] if it exists, to catch items on the trail edge of  a```[code]bla[/code]`` block.  This seemed to work OK in a few tests, at least it fixed the above issue without much fuss.
Then as I went back in history to see how that function developed, it really is a dumping ground to fix "things", outside of code tags while protecting code blocks as much as possible.
So as an experiment I decided to swap out all the code blocks with tokens, then run the bulk of preparsecode() on the token-ized message, and after restore the tokens with the original code blocks.   In doing that I also did a quick refactor to a basic class and stuffed it in the BBC namespace since its really preparseBBC these days.
Figuring I'm missing something obvious and catasrophic in this approach, but it really seems that is what the function was attempting to do all along.  However as additional checks/fixes were added, they did not support code in lists or tables and the like (really because of the above $i + 4).  Of course those are edge cases but the change looks like https://github.com/elkarte/Elkarte/compare/development...Spuds:preparse?expand=1  So better eyes  can take a look.

/me likes

The original $i + 4 thing I did was to replace the start of the loop with
		// It goes 0 = outside, 1 = begin tag, 2 = inside, 3 = close tag, repeat.
		if ($i === 0 || ($i % 4 === 0 && isset($parts[$i + 4])))
		{
			// Use text surrounding the code blocks for compliance.
			$code_holder = '\x%%' . $i . '%%\x';
			$parts[$i] .= isset($parts[$i + 4]) ? $code_holder . $parts[$i + 4] : '';

and then at the end
			// Update the pre and post text with any updates done
			$code_spot = strpos($parts[$i], $code_holder);
			if (isset($parts[$i + 4]))
				$parts[$i + 4] = Util::substr($parts[$i], $code_spot + strlen($code_holder));
			$parts[$i] = Util::substr($parts[$i], 0, $code_spot);
		}

		call_integration_hook('integrate_preparse_code', array(&$parts[$i], $i, $previewing));

which  takes pre[code]asdf[/code]post  combines pre+post, runs that through the checks and then puts pre and post back in their array slots.  It was after that I thought, why not do all of that just once in its own loop instead of inside the current loop.  Anyway just an idea.

For completeness .... the original $i + 4 that I refered to in my comments was
Use pre and post text blocks pre[code]bla[/code]post when doing the tests/corrections
		// It goes 0 = outside, 1 = begin tag, 2 = inside, 3 = close tag, repeat.
		if ($i === 0 || ($i % 4 === 0 && isset($parts[$i + 4])))
		{
			// Use text surrounding the code blocks for compliance.
			$code_holder = '\x{A9}%%' . $i . '%%\x{A9}';
			$parts[$i] .= isset($parts[$i + 4]) ? $code_holder . $parts[$i + 4] : '';

Now using that new $parts[$i] which is pre+placeholder+post for all of the tests/regex etc.  Then at the end of that loop, place the changes back into the respective pre and post areas, using the placeholder as the split/insertion point.
			// Update the pre and post text with any updates done
			$code_spot = strpos($parts[$i], $code_holder);
			if (isset($parts[$i + 4]))
				$parts[$i + 4] = Util::substr($parts[$i], $code_spot + strlen($code_holder));
			$parts[$i] = Util::substr($parts[$i], 0, $code_spot);

After that came the thought of changing all of the codes blocks, in a separate loop, with unique hash placeholders per block (or really split) and run the resulting entire multi hash/placeholder string through the preg's.  That should be more effective on a multi code post, but I'm not sure if I'm missing something in that approach vs the one in this comment.  Really the only difference will be in posts with multiple code blocks.

I had again a look at the code and it seems fine (just by reading it, since you talk about tests I assume you are doing them so it should work as well lol).
Is there anything left before being able to merge it? :smile_cat:

I just have to look over the code again and throw a few more fouled up tests at it to be sure which approach to PR .
I like the hash them all and process it once, but this also allows spanning multiple block elements which is not necessary DOM correct to do, the other approach allows just a single block nesting ... anyway I need to kick the tires a bit more before I choose.


Will vercel-php ever support the gd library?

My backend needs image compression and the most popular way to do it is with the gd library.
There is a workaround without using the gd library and that is by using the Imagick extension. The extension isn't readily available and the documentation does say something about being able to install the extension but I have no clue how to do it.

I have the same problem.
Fatal error: Uncaught Error: Call to undefined function imagecreatetruecolor() in /var/task/user/api/pixel8.php:76 Stack trace: #0 {main} thrown in /var/task/user/api/pixel8.php on line 76

Yeah i also need the gd extention to be present , it's a well known and popular extension to be added

Hi, I would really really love to support GD or ImageMagic, but it's really hard to do it, I am in dead end right now. So, I can tell you if or when vercel-php will support these extensions.


Improve multi_asset table

ma_tx_out is by far the most populated table (~165M entries on epoch 319). It also references the multi_asset table.
We can improve performance here. Two ideas:

We query the multiasset and only if it doesn't exist we insert it. Querying only requires policyId and name, which compose the unique key. Most probably the query will always succeed, since minting comes first.
We can cache a mapping of (policyId, name) -> MultiAssetId. There are 3.2M entries of multi_asset on epoch 319, so this seems like a good bargain: 3.2M entries in memory to avoid 165M queries.

Also multi_asset doesn't have any OnDeleteCascade fields, so on a rollback the db is left with unnecessary fields.

Done in https://github.com/input-output-hk/cardano-db-sync/pull/1094


XYZ and TMS tiles  coordinate converting in createFromTemplate function in ol/tileurlfunction.js

Describe the bug
I am recently working on vector tile in Openlayers-V6.0.0 and Geoserver TMS service.
According to the ol document, I use the '{-y}' placeholder in URL template to indicate TMS Grid.
But the url is incorrect as expected.
below is the configuration.
    const resolutions: number[] = new Array(29).fill(0);
    resolutions.forEach((v, i , a) => {
      a[i] =<PHONE_NUMBER>2804097 / Math.pow(2, i);
    });
    const origin = getTopLeft(getProjection('EPSG:3857').getExtent());
    const extent = [11742903.0, 3724884.75, 12383392.0, 4805854.0];
    const shaanxiCountyLayer = new VectorTileLayer({
      source: new VectorTile({
        url: 'http://<IP_ADDRESS>:8080/geoserver/gwc/service/tms/1.0.0/mygisdb%3Ashaanxi_county<EMAIL_ADDRESS>        format: new MVT(),
        tileGrid: new TileGrid({
          extent,
          resolutions,
          origin,
        }),
      }),
    });

I got these urls with nagative y coordinate.

createFromTemplate function

May be there is a problem when converting XYZ y to TMS y,
var y = range.getHeight() - tileCoord[2] - 1;

according to doc it should be
 var y = Math.pow(2, z) - tileCoord[2] - 1;

yes every thing works well after I do that.
To Reproduce
Configuring  VectorTileLayer as described in the above then check the url.

An EPSG3857/EPSG:900913 grid uses the projection extent and as it is the OpenLayers default it need not be specified.  To restrict the displayed extent specify a layer extent
const shaanxiCountyLayer = new VectorTileLayer({
  source: new VectorTile({
    url: 'http://<IP_ADDRESS>:8080/geoserver/gwc/service/tms/1.0.0/mygisdb%3Ashaanxi_county<EMAIL_ADDRESS>    format: new MVT()
  }),
  extent: extent
});

Thanks a lot, it works like a charm.


[Question] Unit Testing

I'm not sure where to post this, but I think it could be helpful here.  I am new to rust, so I am not 100% proficient with it, but I have not found a way to do unit testing.  I'd totally be willing to write up some documentation once I get it working. I haven't found any references to testing within this repo, nor within the wider rust community.
Any help would be appreciated, and I'd love to make a PR for this repo or the template one if I can figure it out.

I can do a bit of self-advertisement here: https://github.com/Patryk27/avr-tester/ 🙂
It's not as straightforward as cargo test, but it's very close!

you can use qemu, it works
https://github.com/Rahix/avr-hal/discussions/280
it works without no real hardware, yet will have timer issue
https://github.com/Rahix/avr-hal/pull/257


[openBrowser]: api should show correct error message

Expected behavior
openBrowser api should show the correct error message to the user.
Actual behavior
openBrowser api only accepts options as a parameter.
other than options we can't use anything, if user uses anything else except options
the error message should be correct and convenient for user to understand it.

openBrowser("https://taiko.gauge.org/#openbrowser")
✘ Error: Timed out after 30000 ms while trying to connect to Chrome!, run .trace for more info.

Steps to reproduce

open taiko in repl
openBrowser("https://taiko.gauge.org/#openbrowser")

Versions
Version: 1.0.1 (Chromium: 76.0.3803.0) f30e18e5b28f9d57b39d06038f075dd0ccc7d4e9

This issue has been verified and found fixed.
Tested version: ef43f50bfcd74ac6512454789151a726adc2b8b7


template group config not allowed under entities?

as posted in the community:
needed some extra template sensors again, so I tried:
sensor:

  - platform: powercalc
    entities:

      - create_group: All lights
        include:
          template: >
            {{expand('light.all_lights_only')|map(attribute='entity_id')|list}}

      - create_group: All switches
        include:
          template: >
            {{expand('group.all_switches')|map(attribute='entity_id')|list}}

      - entity_id: light.corridor_pool

ofc the templates are correct, and listing all entities in dev tools. All individual entities exist (and, already have individual powercalc sensors)
And yet, I get this error at startup:
Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.powercalc.sensor
Source: custom_components/powercalc/sensor.py:238
Integration: Powercalc (documentation, issues)
First occurred: 11:21:12 (1 occurrences)
Last logged: 11:21:12
Could not resolve any entities in group 'All lights'

and no sensors are created. the switch template does not show an error, but Powercalc does not create any sensors for that config either.
added to that, all other utility integration entities are now unavailable…
configuring the template entities individually like:
sensor:

  - platform: powercalc
    create_group: All lights
    include:
      template: >
        {{expand('light.all_lights_only')|map(attribute='entity_id')|list}}

  - platform: powercalc
    create_group: All switches
    include:
      template: >
        {{expand('group.all_switches')|map(attribute='entity_id')|list}}

  - platform: powercalc
    entities:

      - entity_id: light.corridor_pool
      - entity_id: etcetc
      - etcetc

does create the sensors.
is this a bug, or to be expected?

unfortunately I must retract the statement above the second config works alright.
Nowe see this:
Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.powercalc.sensor
Source: custom_components/powercalc/sensor.py:238
Integration: Powercalc (documentation, issues)
First occurred: 22:34:34 (2 occurrences)
Last logged: 22:34:34

Could not resolve any entities in group 'All lights'
Could not resolve any entities in group 'All switches'

also using that config
sensor:

  - platform: powercalc
    create_group: All lights
    include:
      template: >
        {{expand('light.all_lights_only')|map(attribute='entity_id')|list}}

  - platform: powercalc
    create_group: All switches
    include:
      template: >
        {{expand('group.all_switches')|map(attribute='entity_id')|list}}

somehow I suspect the template errors when a single entity inside that template list is unavailable? Ikea Trqdfri currently suffering from some serious issues in core, and maybe that is causing this too. not sure.
will add a test to not allow unavailable, maybe that helps

Did you create this light group with GUI helper or YAML?

only YAML.
Ah that would indeed be even better, if it auto discards unavailable lights. My group is a light.all_lights_only light group (and made in the UI Helper), so no YAML light group. SHouldnt make a difference but mentioning just to be sure.

I did ask what methodology you used to create the light group (not the powercalc sensor), but it's GUI helper as you already indicated.
That combination is problematic with the templating logic. probably because of loading/intializing order. Normally you only use templates when HA has fully started, so that would behave differently.
But the solution is to use the group option as I shared above. This is working for me with the GUI lightgroups.
Currently it will also include unavailable entities, and set the group to unavailable when any of the underlying entities are unavailable. However I think we can also look at the ignore_unavailable_state option for group entities.
Like this:
- create_group: All lights
  include:
    group: light.all_lights_only
  ignore_unavailable_state: true

Then all entities will still be included in the power group, but only the entities which are available will be summed into the total. When a power sensor become available again it will also be included in the group, which is not true for the approach you used. It depends on which entities were available during startup of powercalc.

sorry, didnt see this when I posted the earlier update above..
complicating factor is I also have some earlier powerful groups I want to keep:
      - create_group: Terrace outdoors
        entities:
          - entity_id: light.terrace_spot_1
          - entity_id: light.terrace_spot_2
          - entity_id: light.terrace_spot_3
          - entity_id: light.terrace_spot_4
        ignore_unavailable_state: true

      - create_group: Kerststerren
        entities:
          - entity_id: light.ster_links
          - entity_id: light.ster_rechts

      - create_group: Dining table
        entities:
          - entity_id: light.dining_table_lamp_1
          - entity_id: light.dining_table_lamp_2
          - entity_id: light.dining_table_lamp_3
        ignore_unavailable_state: true

Can we somehow include those inside the all_lights light group config?

Your second example already includes all entities of group light.all_lights_only, which also contains light.terrace_spot_1 etc. Does you already try this? You might add the ignore_unavailable_state: true same as you did for all the other groups.
I don't think you third example using the nesting has any benefits. Also this combination is not really suggested or supported (nested groups + using the include function). Is don't know if it has any drawbacks, unforseen issues etc.

I only tried the nesting because this was mentioned in the docs?
it would have the benefit of being able to group some, and add those groups to the All lights, which would then automagically not double those in the group, but contain them...
anyways, it does not work, because this is the result:

it loads all the individual entities if the nested groups, but it does not see all the others.
My end goal was merely to create daily and yearly entities for the power/energy of All light. I dont need the daily and yearly for the individual lights... I do need the power and energy for the individual lights though, so I will keep those in the config.
Have to figure out how to create the utility meter entities most efficiently for the grouped power entities

You can also easily create a new group with the powercalc gui and select sensor.lights_inside_total_device_power and sensor.lights_outside_total_device_power as additional power sensors, and the energy sensors on additional energy sensors.

I'll need to have a look into why not all light sensors are added when you combine include with entities, but only the entities are added.

Ook belangrijk, zoals hier vermeld:
https://github.com/bramstroker/homeassistant-powercalc#dynamically-including-entities

Met de include functie worden alleen automatisch entiteiten toegevoegd die in powercalc library staan (dus automatisch gemaakt kunnen worden).

Dus stel er is een light.test, maar die heeft geen profiel in powercalc library dan zal deze bijvoorbeeld niet include worden.
- platform: powercalc
  entity_id: light.test
  fixed:
    power: 50

Makkelijkste is toch denk ik gewoon een nested boom maken en de entities "hard" definieren, dat werkt gewoon en is ook het minst foutgevoelig.
Zoals uit het voorbeeld van de readme.
sensor:
  - platform: powercalc
    create_group: All lights
    entities:
      - entity_id: light.a
      - entity_id: light.b
      - create_group: Upstairs lights
        entities:
          - entity_id: light.c
          - create_group: Bedroom Bob lights
            entities:
              - entity_id: light.d
              - entity_id: light.e
                fixed:
                  power: 50
              - entity_id: light.f
                 manufacturer: signify
                 model: LCT010

Heb het voorbeeld van de readme nog uitgebreid met light_e en light_f zodat het ook duidelijk is dat je handmatig geconfigureerde sensors zo toe kan voegen. Die sensors worden dan dus ook aan de 3 groepen toegevoegd.

I don't know why I posted a whole reply in dutch lol. Anyway not going to translate now as we are both dutch anyway.

In order not to create a separate post, I will write with your permission here.
I have a kettle that has two switches. One is responsible for heating, the other is responsible for boiling. I wanted to combine them into one sensor, since they consume the same 1950W. But when I create a group, I have 3 sensors, one for boiling, the second for heating and one common. Can I somehow get only one shared?

hehe, nevermind, I got that ;-)
thing is, I dont use the GUI options , because that makes the startup slowdown (in the integration startup list, as we discussed some time ago)
Also, for Powercalc, I find it much easier to have all configs in Yaml in a single package. UI is ok, but when it comes to including more than a single entity there's nothing like Yaml....
In the end, I probably will (have to) look at that nested tree of all entities.
thanks for your support, as always, much appreciated.
if you add the ignore_unavailable to the templates, that would be a nice feature. (ofc we also need to guard in the template itself)
and find an answer to

I'll need to have a look into why not all light sensors are added when you combine include with entities, but only the entities are added.

In order not to create a separate post, I will write with your permission here. I have a kettle that has two switches. One is responsible for heating, the other is responsible for boiling. I wanted to combine them into one sensor, since they consume the same 1950W. But when I create a group, I have 3 sensors, one for boiling, the second for heating and one common. Can I somehow get only one shared?

Could you please make a new issue? that would also make it easier for others to find a solution for similar issues.
When done I will remove these comments to not pollute this issue too much with unrelated information.

I'll need to have a look into why not all light sensors are added when you combine include with entities, but only the entities are added.

I have added a test to the test suite of powercalc to test the combination of entities with include, but it's working fine.
Setup is as follows.
Have 3 lights, light.light_a, light.light_b and light.light_c.
light_a has manufacturer and model information in device registry and is supported by powercalc.
Created a light group light.lightgroup containing light.light_a, light.light_b and light.light_c.
Next I setup powercalc as follows:
sensor:
  - platform: powercalc
    create_group: PowercalcGroup
    include:
      group: light.lightgroup
    entities:
      - entity_id: light.light_b
         fixed:
           power: 50
      - entity_id: light.light_c
         fixed:
           power: 50

Now the newly created power group sensor.powercalcgroup_power contains all 3 power sensors from light a, b and c.
b and c are fixed 50W sensors, and a is controlled by powercalc library profile.
You should see something like this in the logs:
DEBUG:custom_components.powercalc.sensor:Including entities from group: light.light_group
DEBUG:custom_components.powercalc.sensor:Found include entities: [RegistryEntry(entity_id='light.light_a'.........
...
DEBUG:custom_components.powercalc.sensors.power:Creating power sensor (entity_id=light.light_a entity_category=None, sensor_name=light_a power strategy=LutStrategy manufacturer=lidl model=HG06462A standby_power=0.20 unique_id=99f899fefes)
DEBUG:custom_components.powercalc.sensors.power:Creating power sensor (entity_id=light.light_b entity_category=None, sensor_name=light_b power strategy=FixedStrategy manufacturer= model= standby_power=0.00 unique_id=fc7dbdca)
DEBUG:custom_components.powercalc.sensors.power:Creating power sensor (entity_id=light.light_c entity_category=None, sensor_name=light_c power strategy=FixedStrategy manufacturer= model= standby_power=0.00 unique_id=3db779de)
DEBUG:custom_components.powercalc.sensors.group:Creating grouped power sensor: Powercalc Group power (entity_id=sensor.powercalc_group_power)
DEBUG:custom_components.powercalc.sensors.group:Creating grouped energy sensor: Powercalc Group energy (entity_id=sensor.powercalc_group_energy)

@Mariusthvdb Did you see my last comment? I was not able to reproduce your issue. Maybe you can check with the additional information I provided?

still puzzling here.
I need the light.all_lights_only to be included, but also add some nested (powercalc) groups, which individual lights are already in the all_lights_only light group.
eg, I have these powercalc groups:
      - create_group: Terrace outdoors
        entities:
          - entity_id: light.terrace_spot_1
          - entity_id: light.terrace_spot_2
          - entity_id: light.terrace_spot_3
          - entity_id: light.terrace_spot_4
        ignore_unavailable_state: true

      - create_group: Dining table
        entities:
          - entity_id: light.dining_table_lamp_1
          - entity_id: light.dining_table_lamp_2
          - entity_id: light.dining_table_lamp_3
        ignore_unavailable_state: true

I need this to be grouped, because I also want those group power and energy sensors....
what will happen if I do:
  - platform: powercalc
    entities:
      - create_group: All lights powercalc
        include:
          group: light.all_lights_only
        entities:
          - create_group: Terrace outdoors
            entities:
              - entity_id: light.terrace_spot_1
              - entity_id: light.terrace_spot_2
              - entity_id: light.terrace_spot_3
              - entity_id: light.terrace_spot_4
            ignore_unavailable_state: true

          - create_group: Dining table
            entities:
              - entity_id: light.dining_table_lamp_1
              - entity_id: light.dining_table_lamp_2
              - entity_id: light.dining_table_lamp_3
            ignore_unavailable_state: true

ideally this config would create the groups and the grouped and individual sensors for the entities in those groups, and then check the light.all_lights_only entities to add the remaining individual light entities powercalc sensors, and ofc finally create the power/energy sensors for the All lights powercalc....

You second example should just work, as it is similar to the configuration I shared and is working for me and in the automatic test suite.
sensor:
  - platform: powercalc
    create_group: PowercalcGroup
    include:
      group: light.lightgroup
    entities:
      - entity_id: light.light_b
         fixed:
           power: 50
      - entity_id: light.light_c
         fixed:
           power: 50

restarting now.
btw, the last config with the multiple groups errors:
Invalid config for [sensor.powercalc]: value should be a string for dictionary value @ data['entities'][0]['include']['group']. Got ['light.all_lights_only', OrderedDict([('entity_id', 'light.dining_table'), ('igonore_unavailable_state', True)]), OrderedDict([('entity_id', 'light.terrace_outdoors'), ('igonore_unavailable_state', True)])]. (See ?, line ?). ```

apparently it wont accept the list.. (yet)

No you can't use the last config that way. The second example using nested groups is fully supported.

No you can't use the last config that way. The second example using nested groups is fully supported.

which 'last config' is that? I am confused now.

I didn't see your edit yet of the post.
I mean this configuration:
- platform: powercalc
    entities:
      - create_group: All lights powercalc
        include:
          group:
            - light.all_lights_only
            - entity_id: light.dining_table
              igonore_unavailable_state: true
            - entity_id: light.terrace_outdoors
              igonore_unavailable_state: true

You should not use that as it's not supported. As you already found out ;-)

You should not use that as it's not supported.

would be cool though. would you consider a FR for supporting the Hue light groups directly?

You should not use that as it's not supported.

would be cool though. would you consider a FR for supporting the Hue light groups directly?

Cannot implement that unfortunately as the information about which entities are in a hue group is not available to me. Did have a look into that before.

no, this config does not work, it simply shows all being unavailable.

Strange. Did you put igonore_unavailable_state: true in the root now?
Any clues in the logs?

yes, I tried this config:
sensor:

  - platform: powercalc
    entities:

      - create_group: All lights powercalc
        include:
          group: light.all_lights_only
        ignore_unavailable_state: true
        entities:
          - create_group: Terrace outdoors
            entities:
              - entity_id: light.terrace_spot_1
              - entity_id: light.terrace_spot_2
              - entity_id: light.terrace_spot_3
              - entity_id: light.terrace_spot_4
            ignore_unavailable_state: true

          - create_group: Dining table
            entities:
              - entity_id: light.dining_table_lamp_1
              - entity_id: light.dining_table_lamp_2
              - entity_id: light.dining_table_lamp_3
            ignore_unavailable_state: true

and see this in the attributes :

so all named entities in the nested create groups are taken into account (they are off power, so should be unavailable and hence ignored), but the light.all_lights_only (which is a compound HA light group containing other lights groups/lights, isnt take. into account:

maybe Powercalc can not iterate those like jinja can?

Might be because because the contained entity_id's in lights.all_lights_only are light groups as well? For example light.all_inside_light?

yes, that is what I said above. my suspicion too

I don't go recursively through that. Will have a look if I can implement that.

that would be very nice, and consistent with:
{{expand('light.all_lights_only')|map(attribute='entity_id')|list }}

which lists them all

Including multiple groups at once is not implemented and supported yet, as previously indicated. So you can try, but that won't work in any way. Maybe I will add support for that in the future.
I have worked since my last comment on supporting recursively expanding of the groups. See #1433.
You can give that a test run by installing the master branch. So your example with include: group: light.all_lights_only should work now hopefully.
The reason templates are not working for that, and I can't reuse the expand myself is because during startup the entities attribute of the light group is not available yet. This attribute is only populated after startup, so that makes things very complicated.
Global ignore_unavailable_state: true was already implemented long time ago on your request, that should work. It's also listed as global configuration option in the readme.

You can give that a test run by installing the master branch.

installing as I type this, thanks!

The reason templates are not working for that, and I can't reuse the expand myself is because during startup the entities attribute of the light group is not available yet. This attribute is only populated after startup, so that makes things very complicated.

Yes, I am aware. my remark about the
{{expand('light.all_lights_only')|map(attribute='entity_id')|unique|list }}
was merely directed at using some technique to filter out unique entities. not asking you to implement the templates as jinja literally.

Global ignore_unavailable_state: true was already implemented long time ago on your request, that should work.

Nice, missed that, will change right away, saving me again 3 lines of yaml. yeah.

That's very odd, do you have some other configuration for these lights somewhere?
You might put on debug logging and then search for occurances light.dining_table_lamp_1, that should give some pointers are to where power entity for this light is created duplicately.

Btw very strange some things were working for you already after updating to master, because I just merged to master 1 minute ago. I saw I hadn't merged the PR yet.

yeah, I saw that and downloaded the branch ;-)

Can you sent me the full log with debug enabled after startup. You can sent it on discord if you want. Might have a look later today or the weekend, have to go now.

will do.
Ive just reordered the sensor to:
      - create_group: All lights powercalc
        include:
          group: light.all_lights_only
        entities:

          - entity_id: light.boekenkast_marte
            standby_power: 0.18
            linear:
              attribute: brightness
              calibrate:
                - 1 -> 0.3
                - 3 -> 0.7
                - 13 -> 0.8
                - 26 -> 0.9
                - 38 -> 1.0
                - 51 -> 1.1
                - 64 -> 1.3
                - 76 -> 1.4
                - 89 -> 1.6
                - 102 -> 1.8
                - 115 -> 2.3
                - 128 -> 2.5
                - 140 -> 2.9
                - 153 -> 3.3
                - 166 -> 3.8
                - 178 -> 4.1
                - 191 -> 5.1
                - 204 -> 5.9
                - 217 -> 6.4
                - 230 -> 8.1
                - 242 -> 9.4
                - 255 -> 10.8

          - entity_id: light.licht_1
    #         model: TRADFRIbulbT120E27WSopal470lm

# Hue outdoor post
          - entity_id: light.backdoor_outdoors
            model: LWB010 #LWF002

          - create_group: Terrace outdoors
            entities:
              - entity_id: light.terrace_spot_1
              - entity_id: light.terrace_spot_2
              - entity_id: light.terrace_spot_3
              - entity_id: light.terrace_spot_4

          - create_group: Dining table
            entities:
              - entity_id: light.dining_table_lamp_1
              - entity_id: light.dining_table_lamp_2
              - entity_id: light.dining_table_lamp_3

and restarted, to see all individual entities listed in that group. so dining table group is blocking anything below that in the group.
all sensor entities now are available, the main group, and the embededd group, and all individual entities.
the error on the dining table remains, and yes, I'll pop the log over I discord.

Btw one idea to try.
What happens if you switch around Dining table and Terrace outdoors?

no difference. still the same error.
except the entities for terrace outdoors, which now come after the dining table config are no longer created.
Let me do this:
comment the dining table completely, restart, check whether the entities still are created. (which they shouldn't).
If yes, Ill try and delete them manually, and next add them back to the config

Closing this isuse as we did some further discussion and resolution on discord.


samples: add sample for proprietary library integration

A simple example that demonstrates how to include a proprietary static library into the Zephyr build system.
The library is in an out of tree location.
This sample executes some mathematical functions that can be used for audio
processing like filtering (Fast Fourier Transform (FFT)) or echo cancellation
(Least Mean Square (LMS) filter algorithm).
To use this sample, with an out of tree library, one needs to define
an environment variable LIB_LOCATION.
If that is not defined, a default Zephyr API is used instead.
By default it uses CMSIS-DSP backend for math operations.
In README one can find more information.

 I want this to be an example :)

@sylvioalves can you, please, help me a bit here?
Why do I get this error on all esp32 platforms:
[594/599] Linking C executable zephyr/zephyr_pre0.elf
FAILED: zephyr/zephyr_pre0.elf zephyr/zephyr_pre0.map /__w/zephyr/zephyr/twister-out/heltec_wireless_stick_lite_v3_esp32s3_appcpu/samples/application_development/number_crunching/sample.app_dev.number_crunching/zephyr/zephyr_pre0.map
: && ccache /opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/xtensa-espressif_esp32s3_zephyr-elf-gcc  -gdwarf-4 zephyr/CMakeFiles/zephyr_pre0.dir/misc/empty_file.c.obj -o zephyr/zephyr_pre0.elf  zephyr/CMakeFiles/offsets.dir/./arch/xtensa/core/offsets/offsets.c.obj  -fuse-ld=bfd  -T  zephyr/linker_zephyr_pre0.cmd  -Wl,-Map=/__w/zephyr/zephyr/twister-out/heltec_wireless_stick_lite_v3_esp32s3_appcpu/samples/application_development/number_crunching/sample.app_dev.number_crunching/zephyr/zephyr_pre0.map  -Wl,--whole-archive  app/libapp.a  zephyr/libzephyr.a  zephyr/arch/common/libarch__common.a  zephyr/arch/arch/xtensa/core/libarch__xtensa__core.a  zephyr/lib/libc/picolibc/liblib__libc__picolibc.a  zephyr/lib/libc/common/liblib__libc__common.a  zephyr/drivers/interrupt_controller/libdrivers__interrupt_controller.a  zephyr/drivers/clock_control/libdrivers__clock_control.a  zephyr/drivers/pinctrl/libdrivers__pinctrl.a  zephyr/drivers/timer/libdrivers__timer.a  modules/cmsis-dsp/libmodules__cmsis-dsp.a  -Wl,--no-whole-archive  zephyr/kernel/libkernel.a  -L"/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0"  -L/__w/zephyr/zephyr/twister-out/heltec_wireless_stick_lite_v3_esp32s3_appcpu/samples/application_development/number_crunching/sample.app_dev.number_crunching/zephyr  -lgcc  zephyr/arch/common/libisr_tables.a  -lgcc  -Wl,--fatal-warnings  -Wl,--gc-sections  -Wl,--build-id=none  -Wl,--sort-common=descending  -Wl,--sort-section=alignment  -Wl,-u,_OffsetAbsSyms  -Wl,-u,_ConfigAbsSyms  -nostdlib  -static  -Wl,-X  -Wl,-N  -Wl,--orphan-handling=warn  -Wl,-no-pie  -DPICOLIBC_DOUBLE_PRINTF_SCANF  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/src/linker/esp32s3.rom.alias.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/esp_rom/esp32s3/ld/esp32s3.rom.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/esp_rom/esp32s3/ld/esp32s3.rom.api.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/esp_rom/esp32s3/ld/esp32s3.rom.libgcc.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/esp_rom/esp32s3/ld/esp32s3.rom.newlib.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/esp_rom/esp32s3/ld/esp32s3.rom.version.ld  -T/__w/zephyr/modules/hal/espressif/zephyr/esp32s3/../../components/soc/esp32s3/ld/esp32s3.peripherals.ld  -Wl,--wrap=longjmp  --specs=picolibc.specs  -lc  -lgcc && cd /__w/zephyr/zephyr/twister-out/heltec_wireless_stick_lite_v3_esp32s3_appcpu/samples/application_development/number_crunching/sample.app_dev.number_crunching/zephyr && /usr/bin/cmake -E true
/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0/../../../../xtensa-espressif_esp32s3_zephyr-elf/bin/ld.bfd: zephyr/zephyr_pre0.elf section `rodata' will not fit in region `dram0_0_seg'
/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0/../../../../xtensa-espressif_esp32s3_zephyr-elf/bin/ld.bfd: zephyr/zephyr_pre0.elf section `rodata' will not fit in region `iram0_0_seg'
/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0/../../../../xtensa-espressif_esp32s3_zephyr-elf/bin/ld.bfd: DRAM segment data does not fit.
/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0/../../../../xtensa-espressif_esp32s3_zephyr-elf/bin/ld.bfd: region `iram0_0_seg' overflowed by 49112 bytes
/opt/toolchains/zephyr-sdk-0.16.5-1/xtensa-espressif_esp32s3_zephyr-elf/bin/../lib/gcc/xtensa-espressif_esp32s3_zephyr-elf/12.2.0/../../../../xtensa-espressif_esp32s3_zephyr-elf/bin/ld.bfd: region `dram0_0_seg' overflowed by 90064 bytes
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Should I disable this sample for esp32 or do you have any hint on how to fix this?
Thank you!

@iuliana-prodan, Hi, let me check.

@iuliana-prodan, I guess there is something that can be done regarding esp32xx appcpu to fix this. However, I think the easiest way right now is for you to add  arch_exclude: xtensa from this sample? It seems the code is intended to ARM devices only due to #include "arm_math.h", correct?

@iuliana-prodan, I guess there is something that can be done regarding esp32xx appcpu to fix this. However, I think the easiest way right now is for you to add  arch_exclude: xtensa from this sample? It seems the code is intended to ARM devices only due to #include "arm_math.h", correct?

This runs also on HiFi4 DSP from i.MX8MP, which is also Xtensa.
Actually I developed this sample for that target.
By default is using CMSIS-DSP but I also link with NatureDSP from Cadence.
I have an output of the console in the README.
So, no, I don't want to disable for Xtensa.

@iuliana-prodan, I guess there is something that can be done regarding esp32xx appcpu to fix this. However, I think the easiest way right now is for you to add  arch_exclude: xtensa from this sample? It seems the code is intended to ARM devices only due to #include "arm_math.h", correct?

This runs also on HiFi4 DSP from i.MX8MP, which is also Xtensa. Actually I developed this sample for that target. By default is using CMSIS-DSP but I also link with NatureDSP from Cadence. I have an output of the console in the README. So, no, I don't want to disable for Xtensa.

Got it.

Weird, it SKIPS some test due to RAM overlfow:
 yd_esp32/esp32/appcpu     samples/application_development/number_crunching/sample.app_dev.number_crunching SKIPPED (dram0_1_seg overflow)
and fails in others with same the reason. Still checking it.

@iuliana-prodan This fixes it: #72013 CI is not properly detecting the overflow scenario in there.

Thank you @sylvioalves for the quick fix!

Are we going to try to merge this PR?

@kartben @nashif please review. Thanks!

@kartben need the assignee to review/approve

@kartben can you please have a look at this?
I want to catch 3.7.

@iuliana-prodan I am really sorry for taking so long to finally get to this. I think this needs more work as right now the sample is IMO trying to accomplish two very different things, and as such is not doing the best job at either.
Please let me know if I got it wrong or if you disagree but it feels like trying to both provide CMake sample for how to link to external lib and a showcase of "advanced DSP magic with multiple backends, including proprietary ones", is making things unnecessarily complex.
If the sample is meant to be the former, it does indeed belong to "application_development" samples but should really be minimal, and easier to reproduce (i.e. I believe the sample's test should even be able to exercise both options of LIB_LOCATION provided / not provided), and should be named something like "3rd party library integration" or sth like that. But this might be what "external_library" sample does already?
If the sample is meant to be a demonstrator for a generic "number crunching" / "advanced DSP" I'd say maybe put it under modules/cmsis_dsp as an advanced sample?

Hi @kartben,
Thanks for the review.
I would like to keep the sample as is, so I will move it in /samples/modules/cmsis_dsp/number_crunching.

This is a lot of code that's really far from a "minimal" sample. In fact, it feels like it might be designed as a test rather than a sample based on the terminology used in the code? Have you considered making it a test instead?
Also, given we already have the external lib sample and a basic CMSIS-DSP sample, I am not sure this additional sample is needed but I will let the CMSIS-DSP maintainers also chime in.

Yes, is a complex sample because it not just for ARM.
The scope is to be used for other architectures also, like Xtensa.
This is an example for NatureDSP from Cadence, for Xtensa arch.
CMSIS-DSP is used by default since it's available in Zephyr.
With this sample we can demonstrate how to use equivalent or other APIs that are not in Zephyr.
I don't want to be a test, we already have zdsp tests.
For example, I can't even run the zdsp tests on my DSP from i.MX8MP, which is a HiFi4 Xtensa arch (+ getting errors when trying to compile it with other toolchain - see here)
Is a complex sample since, on a DSP, we usually do complex match calculations.
Also, the API is not always the same for ARM, Xtensa, or others.

Bulk changing the milestone from everything which did not make it to the v3.7.0 freeze deadline
and which is not labeled as a bug or documentation (or does not appear clearly as only containing
a bug fix or a documentation change).
If you disagree about this change please add the milestone again.
Please note that until rc2 only PRs containing bug fixes, docs and tests can be merged.
After that and until release only bug fixes and docs changes.
Exceptions require approval by the release team and a vote by the TSC.
https://docs.zephyrproject.org/latest/project/release_process.html#stabilization-phase

@stephanosio @kartben can you, please, have a look at this?
Since Zephyr 3.7 was release, this can be merged now.
Thanks!

This "CMSIS-DSP sample" has its own DSP library abstraction layer and is actually a sample for multiple DSP libraries, including CMSIS-DSP and NatureDSP.
A better (and more correct) way of doing this would be to add the relevant APIs to the Zephyr DSP subsystem modelled after the CMSIS-DSP API, implement the CMSIS-DSP and NatureDSP backends for it, and write a sample for the Zephyr DSP subsystem.
Nevertheless, this still technically is a "CMSIS-DSP sample" and is only a sample; so, I will not leave a -1 if you prefer to not implement the more complete solution described above.

Hi @stephanosio,
I would prefer this to be merged as is.
Your solution can be done later since the backends for CMSIS-DSP and Nature DSP are not quite 1:1 match.
Thanks!

I agree with @stephanosio that if the goal is showing generic DSP API it would be better to work towards making this an actual generic front-end in Zephyr DSP subsystem.
I don't think having "advanced" CMSIS-DSP sample is really needed: there's nothing Zephyr specific in the sample as it's "just" showing users how to use CMSIS-DSP API. There's already https://docs.zephyrproject.org/latest/samples/modules/cmsis_dsp/moving_average/README.html showing how to get going with CMSIS-DSP (i.e. how to pull in the module, etc.).
The way forward to make it possible for end users to actually leverage multiple DSP backends is as @stephanosio suggests, not by creating an ad-hoc code sample.

As I previously mentioned here this is not just a sample for CMSIS-DSP.
This is an example to demonstrate that one can start developing a sample from the existing code in Zephyr (like CMSIS-DSP) and then port it or integrate it with a 3rt party library or a proprietary code. I know we have the application_development/external_lib, but that doesn’t do much, is just a dummy code.
If I were to add a sample with 2 backends I would keep only simple operations (like vector addition) since the other math ops are no exact match between CMSIS-DSP and NatureDSP.
I also tried ZDSP tests – see my comment here.
The frontend of ZDSP does not match NatureDSP, only a few vector operations.
So, to make a generic frontend is a complicated task, that cannot be addressed by this PR.
With this sample I want to demonstrate that you don’t always have a 1:1 match, you have to make some tweaks in order to have your code integrated with an existing sample.
Since, you don’t consider this a well defined application I will move it in samples/boards/nxp_adsp.

Changes since last version:

move the sample from samples/modules/cmsis_dsp/number_crunching/ to samples/boards/nxp/adsp/number_crunching/.

@kartben we would like to get this sample merged into the upstream. What is needed to address your change request?

Changes since last version:

move the sample from samples/modules/cmsis_dsp/number_crunching/ to samples/boards/nxp/adsp/number_crunching/.

@stephanosio can you, please, have a look at this.
As seen above I've moved the sample in samples/boards/nxp/adsp/number_crunching/.
Thanks!


There's so many crashes on Android with release

I saw on the third-party crash report that there are a lot of crashes inside Flutter on Android devices, and I have switched many different versions of Flutter without getting better.

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, v1.7.8+hotfix.4-pre.4, on Mac OS X 10.14.5 18F132,
    locale zh-Hans-CN)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 10.2.1)
[✓] iOS tools - develop for iOS devices
[✓] Android Studio (version 3.4)
[!] IntelliJ IDEA Community Edition (version 2018.2.1)
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
[✓] Connected device (1 available)

! Doctor found issues in 1 category.

This is my current version of Flutter. The new version of the app released yesterday also has these crashes.
I think these crashes are a very serious problem that needs to be resolved as soon as possible.

There's not enough information here to track down the crashes. Feel free to reopen if you can provide more information about the crashes.


Database is often lost after a macOS crash

I found already some database crash related issues here. I generelly have no torubles with it.
Except when macOS crashes, my db is gone. Does anybody else have this problem? Maybe I havae something misconfigured (I have zsh and nothing special configured so far)

+1.. This actually happens to me quite a lot.. Not sure why. It happens randomly even when I have no apparently crash.. but some changes in the configuration...
On ubuntu 16.04..
I have been thinking of putting my .z file into a git version control (with some kind of cron job based automated commit) to reduce the risk of losing the history.
Is there a better way?

Did you try this pull request? https://github.com/rupa/z/pull/199

yeah.. I replied to your PR but haven't tried it out. Will try it out..


Custom babel settings do not work

I am making an app with Create-React-App, electron, rescripts. The setup has previously been working, but now I need a custom babel config to add decorators and I can't get it to work.
.rescriptsrc.js:
module.exports = [   require.resolve("./webpack.config.js"),   ["use-babel-config", ".babelrc"] ]
Was previously only the webpack.
webpack.config.js:
module.exports = config => {   config.target = "electron-renderer"   return config }
.babelrc:
{   "presets": ["react-app"],   "plugins": [     ["@babel/plugin-proposal-decorators", { "legacy": true }],     ["@babel/plugin-proposal-class-properties", { "loose": true }]   ] }
I want to keep the CRA preset but add decorator support.
package.json:
"devDependencies": {     "@babel/core": "^7.6.4",     "@babel/plugin-proposal-class-properties": "^7.7.4",     "@babel/plugin-proposal-decorators": "^7.7.4",     "@rescripts/cli": "^0.0.11",     "@rescripts/rescript-env": "^0.0.10",     "@rescripts/rescript-use-babel-config": "^0.0.9",     "@storybook/addon-actions": "^5.2.5",     "@storybook/addon-links": "^5.2.5",     "@storybook/addons": "^5.2.5",     "@storybook/react": "^5.2.5",     "babel-loader": "^8.0.6",     "babel-preset-react-app": "^9.1.0",     "concurrently": "^4.1.2",     "cross-env": "^5.2.1",     "electron": "^6.0.3",     "electron-builder": "^21.2.0",     "eslint-config-airbnb": "^18.0.1",     "eslint-config-prettier": "^6.5.0",     "eslint-plugin-jsx-a11y": "^6.2.3",     "eslint-plugin-prettier": "^3.1.1",     "prettier": "^1.19.1",     "prettier-eslint": "^9.0.1",     "prettier-eslint-cli": "^5.0.0",     "prettier-stylelint": "^0.4.2",     "stylelint": "^12.0.0",     "stylelint-config-standard": "^19.0.0",     "wait-on": "^3.3.0"   }
Error output in electron:
Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.
Check the render method of Icon.
If I remove the ["use-babel-config", ".babelrc"], then it works as normal but I cannot use decorators.
I think it is reading the .babelrc config because if I misstype something there I can crash the app. I have tried removing the "plugin-proposal-class-properties" but it does not make a difference.
Any ideas on something I might have missed?

In using it, try wrapping your custom rescript with braces:
webpack.config.js
module.exports = [
  ["use-babel-config", ".babelrc"] ,
  [require.resolve("./webpack.config.js")]
];

Also, above you wrote your webpack rescript as module.exports = config => { config.target = "electron-renderer" return config }. I want to make sure you're appropriately distinguishing between the two expressions in the function body. Can you try this:
webpack.config.js
module.exports = config => {
  config.target = "electron-renderer";
  return config;
};

If all else fails, try reversing the order:
.rescriptsrc.js
module.exports = [
  ["use-babel-config", ".babelrc"] ,
  require.resolve("./webpack.config.js")
];

If it's still broken, please share the code & I'll debug :)

@m-brantheim closing this out since I haven't heard back from you.

Hello again,
Sorry I was away for the holiday, just got back to work.
The braces was not the problem, I created a project from scratch using this config and it worked so I figured it must come from something in my code. I was able to track it down.
I have a function for importing .svg files to render them with react...something like this.
import React from "react"
import { ReactComponent as AddIcon } from "./svg/Add.svg"
...
import IconEnum from "./IconEnum"

const getIcon = code => {
  switch (code) {
    case IconEnum.ADD:
      return <AddIcon />
    ...
  }
}

export default getIcon

Any idea why this breaks when I add ["use-babel-config", ".babelrc"]?

I found the problem was already solved here: https://github.com/harrysolovay/rescripts/issues/13


Ambiguous use of 'init(callback:)'

I'm trying to use Stevia and ws and am getting the following error in this included pod:
.../Pods/thenPromise/Source/Promise+Delay.swift:30:16: Ambiguous use of 'init(callback:)'
I've tried the usual update/install of pods but no joy. Any advise would be appreciated as I've not used these libraries before and was just starting to like it, but couldn't compile.
When I tried to use the freshOS starter project I got: Module compiled with Swift 4.0 cannot be imported in Swift 4.1
Thanks

Check: Xcode 9.3 & Swift 4.1
Promises calls like this:
func foo() -> Promise<Void> {
    return Promise { resolve, _ in
      // code...
  }
}

Will now need to be disambiguated like so:
func foo() -> Promise<Void> {
    return Promise { (resolve: @escaping () -> Void, _) in
      // code...
  }
}

Thanks, I solved the problem as described in [https://github.com/freshOS/ws/issues/52](WS issues) by vimask.

@sdrpa Thanks for chiming in on this one :)
Closing this since it looks like it has been solved, feel free to reopen if needed.


Support ESM

@alexnault was this ever released?

@borislit Not on npm since the current maintainer does't have publishing access: https://github.com/eduolalo/moment-business-days/issues/99
We've managed to get ESM working by installing this specific commit straight from github. Hackish but it works.
npm i https://github.com/kalmecak/moment-business-days.git#9146190b053ca0b06b75f0a525fa5404e0b02b1b


Style rb-side-nav status badges

TP: https://rockabox.tpondemand.com/entity/9830
Follow-on from #227.

Add rb-side-nav invalid status badge
Count
Invalid

Waiting on valid attribute to be added

@nwhite89 in regards to commit history quality- for us to achieve a readable commit history, we should try and mention the component in the commit message to give it some context.
I'm going to squash the history on this since @ohskylab's #1c99428 commit message is wrong. Your commit will be merged into that, and mine into the #6a3cc36.

Sure not a problem


Post deploy hooks

Context
A convenient run a bash script should be added when as a successful post deploy hook. Specific actions being up to me to implement, maybe a fun "success" echo as the default script contents to draw attention to it's existence

I'd normally opt for something simple like a scripts/post_deploy.sh, but I am not 100% if that fits with the vision for this repo. Perhaps a yarn postdeploy command?


How the file structure of train sets looks like?

I wanna use my own datasets .thanks

I got it at:
https://github.com/yjxiong/temporal-segment-networks/wiki/Working-on-custom-datasets.