Error when uploading metas to IPFS

I am still experiencing issues with my issue request, however I have managed to get 10 test JSONs linked to valid IPFS urls, so I decided to proceed as a test run
When I attempt to proceed to uploading metadata to IPFS via the command node utils/nftport/uploadMetas.js
, I get the error:
PS C:\Users\OneDrive\Documents\Komodo_Joe\NFTPortUpload\video-source-code-create-nft-collection-main> node utils/nftport/uploadMetas.js
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
undefined metadata uploaded & added to _ipfsMetas.json
As can be seen in my attached code the _ipfsMetas.json file content is not matching my seperated JSON files or _metadata.json file.
NOTE: I did not use the hashlips generator to create my images and metadata, however my JSON is valid. Any idea of fix?
Thank you for the great content :)

fixed with latest code:
https://github.com/codeSTACKr/video-source-code-create-nft-collection/tree/fix-uploads-mints


Dauntless Shield not calculated properly

From what I can tell, the ability Dauntless Shield doesn't actually apply the +1 Defense boost to Zamazenta even though it displays one in the calculation text. I plan on looking into the bug, but I wanted to make it known.
Using a Zamazenta and putting its Defense to -1 is no different than keeping it neutral on the calc as well, but I assume the bugs are related.


Keeping compilation artifacts of each run

Is there a way for me to make cargo-hack keep the compiled binary after each run? I'm thinking of something like this:
# Assume crate foo has features alpha and bravo
$ cargo hack build --feature-powerset --keep-artifacts
$ ls target/debug/cargo-hack-artifacts
foo
foo-alpha
foo-bravo
foo-alpha-bravo

This would be useful if for example, you would like to upload them as CI artifacts.
If this is currently not possible, I would like to volunteer to work on a PR, thanks.

It is probably possible to write a script that does this using --print-command-list (https://github.com/taiki-e/cargo-hack/pull/175), but adding a flag to do this in a single command seems reasonable to me.

I'm in the middle of implementing this, and it turns out it's quite complicated with a lot to add.
First of all, cargo can produce a lot of different types of binary objects, and they are all placed under different paths. The behaviour is well-documented here, but still it's a lot of work to implement.
Second, cargo allows target selection (not to be confused with the target triple) using flags like --lib, --bins, --bin, etc. See cargo build -h. Right now these flags are of "passthrough" type, but they need to be changed to "propagated" type instead so that we can know which objects to copy.
This is all to say that it's more work than I expected. Not too difficult; just tedious. Give me some time.


posix spawn, take 2

This is a commit series to re-enable the usage of posix_spawn on macOS.
CI: https://ci.nodejs.org/job/libuv-test-commit-osx-cmake/359/
CI+node: https://ci.nodejs.org/job/libuv-in-node/191/

darn, of course we would now run into what looks like another pre-existing nodejs/kernel bug more frequently now 😂

@vtjnash there is also an issue with posix_spawnp code path when used with posix_spawn_file_actions_addchdir_np which turns out to be a bug in the implementation. Originally caught in https://github.com/microsoft/vscode/issues/118267#issuecomment-797016213. I am using pthread_chdir_np as a workaround.
diff --git a/src/unix/process.c b/src/unix/process.c
index 0fb9fda1..ef290d0e 100644
--- a/src/unix/process.c
+++ b/src/unix/process.c
@@ -366,6 +366,28 @@ static uv_once_t posix_spawn_init_once = UV_ONCE_INIT;
 static uv__posix_spawn_fncs_t posix_spawn_fncs;
 static int posix_spawn_can_use_setsid;

+static int uv__change_current_thread_directory(const char* path) {
+  // Changes the current thread's directory to a path or directory file
+  // descriptor. libpthread only exposes a syscall wrapper starting in
+  // macOS 10.12, but the system call dates back to macOS 10.5. On older OSes,
+  // the syscall is issued directly.
+  if (__builtin_available(macOS 10.12, *)) {
+    return pthread_chdir_np(path);
+  } else {
+    return syscall(SYS___pthread_chdir, path);
+  }
+}
+
+// The recommended way to unset a per-thread cwd is to set a new value to an
+// invalid file descriptor, per libpthread-218.1.3/private/private.h.
+static int uv__reset_current_thread_directory() {
+  if (__builtin_available(macOS 10.12, *)) {
+    return pthread_fchdir_np(-1);
+  } else {
+    return syscall(SYS___pthread_fchdir, -1);
+  }
+}
+

 void uv__spawn_init_posix_spawn_fncs(void) {
   /* Try to locate all non-portable functions at runtime */
@@ -501,12 +523,23 @@ int uv__spawn_set_posix_spawn_file_actions(posix_spawn_file_actions_t* actions,

   /* Set the current working directory if requested */
   if (options->cwd != NULL) {
-    if (posix_spawn_fncs->file_actions.addchdir_np == NULL) {
-      err = ENOSYS;
-      goto error;
-    }
+    if (strchr(options->file, '/') != NULL) {
+      /* There is a bug in macOS where a relative executable
+       * path like "../myprogram" will cause `posix_spawnp` to
+       * successfully launch the program, but erroneously return
+       * ENOENT when used with posix_spawn_file_actions_addchdir_np
+       * because of the stat call performed on the executable path
+       * which was introduced in macOS 10.15.
+       * Ref: https://opensource.apple.com/source/Libc/Libc-1439.40.11/sys/posix_spawn.c.auto.html */
+      err = uv__change_current_thread_directory(options->cwd);
+    } else {
+      if (posix_spawn_fncs->file_actions.addchdir_np == NULL) {
+        err = ENOSYS;
+        goto error;
+      }

-    err = posix_spawn_fncs->file_actions.addchdir_np(actions, options->cwd);
+      err = posix_spawn_fncs->file_actions.addchdir_np(actions, options->cwd);
+    }
     if (err != 0)
       goto error;
   }
@@ -628,6 +661,9 @@ int uv__spawn_resolve_and_spawn(const uv_process_options_t* options,
     do
       err = posix_spawnp(pid, options->file, actions, attrs, options->args, env);
     while (err == EINTR);
+    /* Restore the thread's working directory if it was changed. */
+    if (options->cwd != NULL)
+      uv__reset_current_thread_directory();
     return err;
   }

The bug here sounds almost identical to this reported problem from 2006: https://gcc.gnu.org/legacy-ml/gcc/2006-09/msg00631.html. Is there anyone here who can tell me if rdar://problem/4736203 is still a known issue in the mach kernel? I can provide code to reproduce, but currently involves re-building all of nodejs with this PR and then running one particular unit test a small number of times.

I can reproduce this kernel bug without libuv also simply by calling posix_spawn a few times:
https://gist.github.com/vtjnash/679174286cf282c298e5d11752343b8a
The specific conditions under which it occurs seems to be if posix_spawn returns an error, it will revoke any pending SIGCHLD, so we might be able to work around it by calling if (posix_spawn() == 0) kill(getpid(), SIGCHLD);, but it is awful and incomplete, and really seems to require an Apple kernel developer to correct this.

I hope this time the code lands. Ping me if I can help somehow, I may manage to squeeze some time.

Adding here for visibility, FB9578526 tracks https://github.com/libuv/libuv/pull/3257#issuecomment-891374523

What's blocking this – how can we help?

https://github.com/libuv/libuv/pull/3064#issuecomment-778495619 says "we had to revert this because it was causing failures in the Node.js test suite."
Can we use posix_spawn when it's working fine (not triggering the bug / failure in test suite) and the original function in all other cases as a stopgap? This should be what most people want until we have the proper fix.

It is a show-stopper-type bug that breaks all usages.

@vtjnash
Can you please provide some context/idea around what that old kernel bug in xnu bug actually is, if it isn't resolved by the patch #3257 (comment)?

mach will sometimes fail to deliver signals as required, and as needed for proper function of uv_spawn. This seems to be greatly exacerbated by using posix_spawn. A partial workaround would be to switch to kevent notifications of process exit, but that is possibly a non-trivial amount of work, and only reduces the scope of the issue, but doesn't solve it.

Just FYI: This patch is also the best path forward on linux-glibc, not just on macos, however further changes are needed. namely:
POSIX_SPAWN_CLOEXEC_DEFAULT --> is not implemented, however the same practical effect is calling posix_spawn_file_actions_addclosefrom_np closing fd =>3  if that function is available,
using posix spawn on recent glibc's will give you the best performing way while using significantly less memory, it is also not affected  by all the vfork() problems because it is implemented using clone CLONE_VFORK  with custom stack..

It looks like Chromium uses posix_spawn on macOS to launch processes: https://source.chromium.org/chromium/chromium/src/+/main:base/process/launch_mac.cc;l=295;drc=7319bbdb7d6ea97ef3237381e4a88b7a465dd0da
Is there something that libuv needs that Chromium's code doesn't address? Can we take inspiration from there?

As long as you don't care what happens if the process fails to launch, it works fine already

Chromium uses EVFILT_PROC (instead of SIGCHLD), which also solves the issue. We could use that too, if someone was willing to put in the work to implement it.

Chromium uses EVFILT_PROC (instead of SIGCHLD), which also solves the issue. We could use that too, if someone was willing to put in the work to implement it.

I have some time to spend on this, but I don't have a lot of background context here. Would you be able to give me some pointers on what would need to change to use EVFILT_PROC as opposed to SIGCHLD?

We need to change the way libuv registers for the NOTE_EXITSTATUS for these objects: by calling EV_ADD here for them, and then handling them directly when kqueue notifies us of them, instead of waiting for the SIGCHLD event and calling waitpid on them (and doing it for *BSD's only, including macOS, but not changing linux or others).

@vtjnash Gotcha, thanks. How have you been reproducing the failure? The test suite seems to pass reliably on my machine.

The code is above in a gist https://gist.github.com/vtjnash/679174286cf282c298e5d11752343b8a

@vtjnash sorry, I meant for triggering it in libuv's usage 😅  I'd like to be able to run some sort of test on libuv to check (a) that the current usage in this patch triggers the bug and (b) whether the code I end up writing has, in fact, fixed the issue. It looks like the gist you posted is a standalone example of triggering the bug.

Oh, I do not have that version written. I have only tested with the nodejs test and that snippet.

Ah, the nodejs test suite triggers it? Okay, that’s a good lead, I can
hopefully distill a test for libuv from that.
On Thu, Jan 20, 2022 at 17:21 Jameson Nash @.***> wrote:

Oh, I do not have that version written. I have only tested with the nodejs
test and that snippet.
—
Reply to this email directly, view it on GitHub
https://github.com/libuv/libuv/pull/3257#issuecomment-1018075718, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABKGAB5SRE4E27V3FW47VLUXCYK5ANCNFSM5BAUTZUQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID:
@.***>

--
j

Okay, I've narrowed it down to the parallel/test-child-process-exec-error test from the Node.js test suite. Without this patch applied, that test passes consistently (albeit slowly). With the patch applied, it sometimes hangs, i.e. doesn't exit. Seems to be fairly unreliable, fails maybe 1% of the time.
Next step is to extract this test into a test in libuv, and hopefully produce a more reliably-failing version.

Here's a libuv test that exhibits the behavior:
diff --git a/test/test-spawn.c b/test/test-spawn.c
index 9f2eb24b..50022bce 100644
--- a/test/test-spawn.c
+++ b/test/test-spawn.c
@@ -1891,6 +1891,43 @@ TEST_IMPL(spawn_quoted_path) {
 #endif
 }

+TEST_IMPL(spawn_exercise_sigchld_issue) {
+  int r;
+
+  init_process_options("spawn_helper1", exit_cb);
+
+  r = uv_spawn(uv_default_loop(), &process, &options);
+  ASSERT(r == 0);
+
+  // This test exercises a bug in the darwin kernel that causes SIGCHLD not to
+  // be delivered sometimes. Calling posix_spawn many times increases the
+  // likelihood of encountering this issue, so spin a few times to make this
+  // test more reliable.
+  for (int i = 0; i < 100; i++) {
+    uv_process_options_t options = {0};
+    uv_process_t process;
+    options.file = args[0] = "program-that-had-better-not-exist";
+    args[1] = NULL;
+    args[2] = NULL;
+    args[3] = NULL;
+    args[4] = NULL;
+    options.args = args;
+    options.exit_cb = fail_cb;
+    options.flags = 0;
+    r = uv_spawn(uv_default_loop(), &process, &options);
+    ASSERT(r == UV_ENOENT || r == UV_EACCES);
+  }
+
+  r = uv_run(uv_default_loop(), UV_RUN_DEFAULT);
+  ASSERT(r == 0);
+
+  ASSERT(exit_cb_called == 1);
+  ASSERT(close_cb_called == 1);
+
+  MAKE_VALGRIND_HAPPY();
+  return 0;
+}
+
 /* Helper for child process of spawn_inherit_streams */
 #ifndef _WIN32
 void spawn_stdin_stdout(void) {

I've opened #3441 as a continuation of this PR, with the new EVFILT_PROC / NOTE_EXIT code. It's a bit hacky as I'm not all that familiar with the libuv codebase, but it at least passes my new test :)

CI: https://ci.nodejs.org/job/libuv-test-commit/2219/
I have rebased, addressed Apple's review comment, and fixed an unrelated resource management issue (with zombies) on an unlikely error path. Should be good to merge now, if we are passing all CI.

libuv CI is green, so let's check nodejs: https://ci.nodejs.org/job/libuv-in-node/199/

Ah, that did not actually go well. I have created https://github.com/libuv/libuv/pull/3451 to fix that failure (a small problem with https://github.com/libuv/libuv/pull/3441 @nornagon can you review it?)

Trying this again:
CI+node: https://ci.nodejs.org/job/libuv-in-node/200/
CI: https://ci.nodejs.org/job/libuv-test-commit-freebsd/2105/
CI: https://ci.nodejs.org/job/libuv-test-commit-osx-cmake/397/

Reopenning, since GitHub internal state was confused. The only failures above look like unrelated CI problems with nodejs and libuv. This is ready to merge, but would be good to get a review first of the new commits (relative to #3064).

@nornagon if you have more time and interest in helping with this, the last remaining blocker now to merging this is to alter the posix_spawn_file_actions_adddup2 to be more careful to pick a dup2 target fd which does not conflict with an existing fd that is needed by another action. (https://github.com/libuv/libuv/pull/3257#discussion_r797553011)

@vtjnash I'm happy to, but would you be able to help with a little more context? e.g. would it be sufficient to sort the pipes list somehow?

There is not a sort order that will work (for example, the mapping 0=>0, 2=>1, 1=>2, 2=>3). The code can look similar to the non-posix_spawn case, and first needs to dup2 any low numbered fd (less than stdio_count) to a free fd number, then to the final position. Most of that code even is already written, but it fails to check that the number is actually free, which it can do by scanning the input array to check for any conflicts. Relatedly, it can also avoid calling dup2 on any fd > stdio_count, which may help simplify this code. It is always a very small array, so any n^2 behavior is fine here.

@santigimeno what do you think now? I think I have improved the code to also avoid a couple redundant syscalls (close is not needed because CLOEXEC is set).

Really eager to see this merged (and not have to maintain an Electron fork) anymore. Huge thanks to everyone contributing to this! 😄

https://ci.nodejs.org/job/libuv-in-node/208/
https://ci.nodejs.org/job/libuv-test-commit-osx-cmake/407/

This has broken AIX, IBM i and z/OS CI (fyi @libuv/aix @libuv/ibmi @libuv/zos),
e.g. https://ci.nodejs.org/view/libuv/job/libuv-test-commit/2228/
https://ci.nodejs.org/job/libuv-test-commit-ibmi-cmake/935/BUILDTYPE=Release,nodes=ibmi73-ppc64/console
/home/IOJS/build/workspace/libuv-test-commit-ibmi-cmake/BUILDTYPE/Release/nodes/ibmi73-ppc64/src/unix/process.c: In function 'uv__process_child_init':
/home/IOJS/build/workspace/libuv-test-commit-ibmi-cmake/BUILDTYPE/Release/nodes/ibmi73-ppc64/src/unix/process.c:278:34: error: 'F_DUPFD_CLOEXEC' undeclared (first use in this function)
  278 |     pipes[fd][1] = fcntl(use_fd, F_DUPFD_CLOEXEC, stdio_count);
      |                                  ^~~~~~~~~~~~~~~
/home/IOJS/build/workspace/libuv-test-commit-ibmi-cmake/BUILDTYPE/Release/nodes/ibmi73-ppc64/src/unix/process.c:278:34: note: each undeclared identifier is reported only once for each function it appears in

According to the latest documentation, that has been a part of POSIX since 2001

F_DUPFD_CLOEXEC is defined in POSIX.2008 but I can see why you'd think it's from the previous edition: it was approved in Austin Group Interpretation 1003.1-2001 # 171 from 2007.
@richardlau What might work is compiling with -D_POSIX_C_SOURCE=200809, maybe -D_XOPEN_SOURCE=600 or -D_XOPEN_SOURCE=700 too.

The situation on AIX is more subtle -- the builds on AIX 7.1 are failing, but on 7.2 they are passing. I found https://www.ibm.com/support/pages/apar/IJ05201 for AIX 7.2 which must have been applied to our AIX 7.2 instances -- I don't know if there is an equivalent for AIX 7.1 or the other IBM platforms (@libuv/aix @libuv/ibmi @libuv/zos ). This also explains why the AIX bit of libuv-in-node CI in https://github.com/libuv/libuv/pull/3257#issuecomment-1054600676 passed as that ran on AIX 7.2.

The situation on AIX is more subtle -- the builds on AIX 7.1 are failing, but on 7.2 they are passing. I found https://www.ibm.com/support/pages/apar/IJ05201 for AIX 7.2 which must have been applied to our AIX 7.2 instances -- I don't know if there is an equivalent for AIX 7.1 or the other IBM platforms (@libuv/aix @libuv/ibmi @libuv/zos ). This also explains why the AIX bit of libuv-in-node CI in #3257 (comment) passed as that ran on AIX 7.2.

z/OS does not support F_DUPFD_CLOEXEC, so this would break z/OS.

This is not supported on IBM i either.


Exec tachikoma update<PHONE_NUMBER>4709

:hamster::hamster::hamster:   Strategy none
Powered by Interval Pull Request as a Service - Tachikoma.io Announcement: 2015-06-28 Updated permissions - Tachikoma.io

Coverage remained the same at 67.647% when pulling 5c147db5239552f3acca3a336dd1638b4b1cbd90 on pazjacket:tachikoma/update-20171024214709 into ec251e61adc687ae0589e5283f1773ece07521c0 on sanemat:master.


build static libraries or shared libraries for windows

How to build static libraries or shared libraries for windows?

https://k2-fsa.github.io/sherpa/onnx/install/windows.html
Please see our documentation.
You need at least vs >= 2019 to build static libraries.

thanks

Does it fix your issue?

yes, succeed a few times for 1.8.9 version 。1.8.10 version has not been successful yet。

yes, succeed a few times for 1.8.9 version 。1.8.10 version has not been successful yet。

could you post the error logs?


Migrate from 4.0.0 to 4.4.0 was broke my build with Cannot invoke method getClientConfig() on null object

Gradle version - 2.14
With build-extractor 4.0.0 version:
gw wrapper
[buildinfo] Not using buildInfo properties file for this build.
:wrapper UP-TO-DATE

BUILD SUCCESSFUL

Total time: 5.748 secs

After migrate to build-extractor 4.4.0:
gw wrapper

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'sense-support'.
> Failed to notify project evaluation listener.
   > Cannot invoke method getClientConfig() on null object
   > Cannot invoke method getClientConfig() on null object
   > Cannot invoke method getClientConfig() on null object
   > Cannot invoke method getClientConfig() on null object

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 3.1 secs

What was happened and what i do wrong?

@lavcraft,
Thanks for reporting this. We will check and get back to you soon.

@lavcraft,
We are looking at your issue, could you provide us a simple example and stack trace of this error?
It is hard to understand the issue from the error massage.
For your convenience you are welcome to use our Artifactory gradle plugin examples.
Thanks.

I have a similar issue with
gradle.version = 2.11
build-extractor = 4.+

But the mentioned above error has appeared in gradle update dependencies phase (simple multi-module project)

I am also getting this error, here is a stacktrace:
FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':LD3'.
> Cannot invoke method getClientConfig() on null object

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.ProjectConfigurationException: A problem occurred configuring project ':LD3'.
	at org.gradle.configuration.project.LifecycleProjectEvaluator.addConfigurationFailure(LifecycleProjectEvaluator.java:79)
	at org.gradle.configuration.project.LifecycleProjectEvaluator.notifyAfterEvaluate(LifecycleProjectEvaluator.java:74)
	at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:61)
	at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:529)
	at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:90)
	at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:47)
	at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:35)
	at org.gradle.initialization.DefaultGradleLauncher$2.run(DefaultGradleLauncher.java:125)
	at org.gradle.internal.Factories$1.create(Factories.java:22)
	at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
	at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
	at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:122)
	at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:32)
	at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:99)
	at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:93)
	at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
	at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
	at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:93)
	at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:82)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:94)
	at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:43)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:28)
	at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:81)
	at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:46)
	at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:51)
	at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:28)
	at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:43)
	at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:173)
	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:241)
	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:214)
	at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
	at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:207)
	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
	at org.gradle.launcher.Main.doAction(Main.java:33)
	at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
	at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:55)
	at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:36)
	at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
Caused by: java.lang.NullPointerException: Cannot invoke method getClientConfig() on null object
	at org.jfrog.gradle.plugin.artifactory.extractor.listener.ProjectsEvaluatedBuildListener.afterEvaluate(ProjectsEvaluatedBuildListener.groovy:38)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:249)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:229)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:44)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:221)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:209)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:44)
	at org.gradle.internal.event.BroadcastDispatch.dispatch(BroadcastDispatch.java:79)
	at org.gradle.internal.event.BroadcastDispatch.dispatch(BroadcastDispatch.java:30)
	at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy12.afterEvaluate(Unknown Source)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:44)
	at org.gradle.internal.event.BroadcastDispatch.dispatch(BroadcastDispatch.java:79)
	at org.gradle.internal.event.BroadcastDispatch.dispatch(BroadcastDispatch.java:30)
	at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy12.afterEvaluate(Unknown Source)
	at org.gradle.configuration.project.LifecycleProjectEvaluator.notifyAfterEvaluate(LifecycleProjectEvaluator.java:67)
	... 41 more

BUILD FAILED

Total time: 7.107 secs

The project :LD3 is being included via multi-project dependency in this project, it is at ../../ld3 relative to this project. Notably, this library is able to be compiled on its own without any issue, only when being included as a project dependency in another gradle script is this error thrown.

@chrissphinx, @yevtsy and @lavcraft,
Thanks for reporting this issue. Can you please try installing the below snapshot version and let us know if it resolve the issue? You can do that by adding the following to your build script instead of the plugin version currently defined there.
buildscript {
    repositories {
        maven {
            url "https://oss.jfrog.org/oss-snapshot-local"
        }
        jcenter()
    }
    dependencies {
        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.4.x-SNAPSHOT"
    }
}

@eyalbe4 thank you for this, however, I ran into the problem when using the artifactory plugin. Will this snapshot affect that?

@chrissphinx, if you ran into this issue with version 4.4.0 of the Artifactory Plugin, but version 4.3.0 works fine, then this fix aims to fix this problem. Please let us know if indeed it resolves the issue.

@eyalbe4 thank you for the response, it actually occurred on 4.2.0 and 4.4.0, but I did not check on 4.3.0. I will try to do so and let you know.

@eyalbe4: To follow up on this: it looks like I'm getting Error: Could not find any convention object of type ArtifactoryPluginConvention for both versions 4.2.0 and 4.3.0. I think I had a circular dependency with what I was doing on Friday, but this time I made sure to set it up correctly and that's the error I get. Moving to 4.4.0 gives the same error as above, so that is at least correct.
However, your 4.4.x-SNAPSHOT build does indeed remedy the problem and both the main project and its dependency project (that has the Artifactory build directives) configure and build correctly. Thank you very much for the fix!

@chrissphinx, thanks for feedback and we're happy to hear the snapshot version resolves the issue. We improved the code further to make sure it is compatible with all project structures and we'd appreciate your feedback for the latest version as well. To test it, all you need to do is to remove the plugin code from your local maven repository (~/.m2/repository/org/jfrog/buildinfo/) and run the build again.
@yevtsy and @lavcraft, we'd really appreciate feedback from e

@eyalbe4, no problem. I actually had to remove the directory from under ~/.gradle/caches/... but I did so and executed the build again without any issues. Let me know if I can be of further assistance!

I have the same problem using 4.4.0 and I could fix using snapshot version 4.4.x-SNAPSHOT.


Accessing sessionStorage may throw errors if a browser is configured to not support it

Bug Report
Accessing sessionStorage may throw errors if a browser is configured to not support it. See this article for more information on how this might happen and how to detect it: https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API We should handle this scenario more gracefully where we access session storage in dev site

Let's treat this as a catastrophic failure and fail to load dev site in general if storage isn't available.


Change auto paren behaviour

Is your feature request related to a problem? Please describe.
I'm always frustrated when I need to confirm an autocompletion item, in order to have the parens inserted. I can understand this behaviour for snippets where if you complete it without confirmation, it would be disastrous.
Describe the solution you'd like
<c-n> & <c-p> should insert the parens automatically without needing to press the confirmation key. But this behaviour is only applicable for LSP completion items.
Describe alternatives you've considered
VSCode has this behaviour:

The vscode example you showed is a great example how it should work. Currently what I am finding quite annoying is that it does insert parenthesis but the cursor is moved to the end of a closing paren So I then have to switch to normal mode, move the cursor back and enter insert mode agian.

Unfortunately, it cannot be implemented.  It is Vim/neovim completion's feature.

VSCode has this behaviour:

You have confirmed the candidate.  So it is not without needing to press the confirmation key.


[New Port Request] xtensor-zarr

Library name: xtensor-zarr
Library description: ZARR implementation by/for Xtensor
ZARR is a storage format for multi-dimensional data.
Source repository URL: https://github.com/xtensor-stack/xtensor-zarr
Project homepage (if different from the source repository):
Anything else that is useful to know when adding (such as optional features the library may have that should be included):
There are already a bunch of xtensor-* libs in vcpkg.

If you wish to add this port in the future, you can reopen this issue, but we're closing it for now.


Tests that link with SwiftShaders libvulkan.so can only be run from the out directory.

This is a hassle when running specific test binaries. We should fix the RPATHs in the executables to lookup the dylib at a path relative to the binary at the very least.
The error when running from some other directory is:
dyld[74768]: Library not loaded: './libvulkan.dylib'
  Referenced from: '/Users/foo/Development/engine/src/out/host_debug_unopt_arm64/display_list_rendertests'
  Reason: tried: './libvulkan.dylib' (no such file), '/usr/local/lib/libvulkan.dylib' (no such file), '/usr/lib/libvulkan.dylib' (no such file), '/Users/foo/Development/engine/src/libvulkan.dylib' (no such file), '/usr/local/lib/libvulkan.dylib' (no such file), '/usr/lib/libvulkan.dylib' (no such file)
zsh: abort      out/host_debug_unopt_arm64/display_list_rendertests

cc @iskakaushik or perhaps @jason-simmons: I though we already set the RPATHs to look at locations relative to the binary already. Did this change recently?

cc @flar

When I look at the external dependencies for various test executables in out/foo, libvulkan.dylib is the only one listed with a relative path. All of the others have absolute paths either from /System/Library/Frameworks or '/usr/lib'.
Should we be linking this internal library statically instead of dynamically?

I think we should be linking it dynamically still.

I wasn't able to reproduce this but I think it may be because I had the Vulkan SDK installed. Something to investigate.

I get this on M1 Mac. No Vulkan SDK installed.


Referencing other fields

    $collection->updateMany(
        ['game_id' => $game_id],
        ['$min' => ['low_score' => '$score'], '$max' => ['high_score' => '$score']]
    );

The above code doesn't actually compare the values currently in low_score with the value of score. If I replace $score with a numeric value it works as it should.
    $collection->updateMany(
        ['user' => $this->user['id'], 'low_score' => ['$exists' => false], 'high_score' => ['$exists' => false]],
        ['$set' => ['low_score' => '$score', 'high_score' => '$score']]
    );

This code doesn't set low_score and high_score to the value of score if they don't exist. It sets them to the string "$score". I want it to set them to the value of score

The MongoDB query syntax only allows comparing fields with literal values, not with other fields from the same document. Similarly, updates using this query language can only refer to literal values, not other fields in the document either. The $min and $max query operators also work differently. I'll make the first example where we're storing the score in the document, simultaneously updating the low_score and high_score fields:
$collection->updateOne(
    ['game_id' => $game_id],
    [
        '$set' => ['score' => $score],
        '$min' => ['low_score' => $score],
        '$max' => ['high_score' => $score],
    ]
);

Note that the $min and $max operator set the field to the specified value if the field previously did not exist (which is what you attempted to do in the second document). This all works well because we're using literal values. If you want to update the documents after the fact, referencing another field in your document, you have to use aggregation pipeline to do so.
With MongoDB 4.2, you can use an aggregation pipeline as update specification, but you may only use the $addFields, $project, and $replaceRoot pipeline stages. You can also use their aliases $set, $unset, and $replaceWith. Note that you must be using the latest versions of MongoDB (4.2.0 or newer), the mongodb PHP extension (1.6.0 or newer), and this library (1.5.0 or newer) for the following examples to work.
The query above would have to look like this:
$collection->updateOne(
    ['game_id' => $game_id],
    [
        ['$set' => [
            'low_score' => ['$min' => ['$score', '$low_score'],
            'high_score' => ['$max' => ['$score', '$max_score'],
        ]],
    ]
);

In this example, we're using the $set aggregation pipeline stage (also known as $addFields) to update the low_score and high_score fields. Note that the $set operator is no longer in the top-level array of the update specification but nested once - this is because we can have multiple aggregation pipeline stages in any order, even repeating previous stages (e.g., $set followed by $replaceWith, followed by another $set stage). The $min and $max aggregation pipeline operators have a different syntax compared to the query operators of the same name, but the behaviour is the same. You can find a list of aggregation pipeline operators in the MongoDB Documentation.
One more point about aggregation pipeline updates: while the entire update operation is atomic, each aggregation pipeline stage is evaluated and executed in sequence. Thus, when using the '$score' expression in a stage, it will refer to the value of the score field before applying the current pipeline stage. Take the following example:
$collection->updateOne(
    ['game_id' => $game_id],
    [
        ['$set' => [
            'score' => ['$sum' => '$score', $score],
            'low_score' => ['$min' => ['$score', '$low_score'],
            'high_score' => ['$max' => ['$score', '$max_score'],
        ]],
    ]
);

You would think that this aggregation pipeline stage would increase the stored score by the value in $score, then update the low_score and high_score fields accordingly. However, the entire $set stage is evaluated at once, so the $min and $max operators use the value of score before the $sum operator is applied. You have to split this into two separate aggregation pipeline stages for this to work as expected:
$collection->updateOne(
    ['game_id' => $game_id],
    [
        ['$set' => [
            'score' => ['$sum' => '$score', $score],
            'low_score' => ['$min' => ['$score', '$low_score']],
            'high_score' => ['$max' => ['$score', '$max_score']],
        ]],
    ]
);

Of course, if you wanted to combine them you could use the literal value and make it a single $set stage:
$collection->updateOne(
    ['game_id' => $game_id],
    [
        ['$set' => [
            'score' => ['$sum' => '$score', $score],
        ]],
        ['$set' => [
            'low_score' => ['$min' => ['$score', '$low_score']],
            'high_score' => ['$max' => ['$score', '$max_score']],
        ]],
    ]
);

Now, the first stage increases the score, while the second stage updates the low_score and high_score fields. However, the entire operation is atomic, so you don't have to worry about a second update query coming in to mess things up - they will be executed in sequence.
One last point I'd like to make is that you can only reference fields from the document being updated, not any other documents. The latter would require an aggregation command using the $out or $merge stage instead of an update command as shown above.
Please let me know if you have any other questions.

Ok, I think I see what you're saying. I tried it but got an error.
        $collection->updateOne(
            ['project' => 12],
            [
                ['$set' => [
                    'low_score' => ['$min' => ['$score', '$low_score']],
                    'high_score' => ['$max' => ['$score', '$max_score']],
                ]]
            ]
        );

Error determining if update will go over space quota: Error estimating update space usage: Error finding 'u' part of update: Expected 'u' to be BSON (or equivalent), but got []interface {} instead. Doc = [{q [{game_id 12}]} {u [[{$set [{low_score [{$min [$score $low_score]}]} {high_score [{$max [$score $high_score]}]}]}]]} {upsert false} {multi true}]

I tried by dropping the first set of [] on the 2nd parameter, I did some commands like that and they work, but wasn't with $min and $max.
        $collection->updateOne(
            ['project' => 12],
            ['$set' => [
                    'low_score' => ['$min' => ['$score', '$low_score']],
                    'high_score' => ['$max' => ['$score', '$max_score']],
            ]]
        );

The dollar ($) prefixed field '$max' in 'high_score.$max' is not valid for storage.

@sevenpixels: I attempted to reproduce this on a standalone mongod instance (v4.2.1) using extension version 1.6.1 and library version 1.5.1.
$ rm -rf /tmp/db1;
$ mkdir /tmp/db1;
$ mongod --dbpath /tmp/db1 --port 27017

I then ran the following script, based on your example above:
<?php

require_once 'vendor/autoload.php';

$client = new MongoDB\Client('mongodb://<IP_ADDRESS>/');

$collection = $client->selectCollection('test', 'gh701');

$collection->drop();

$collection->insertOne([
    'project' => 12,
    'score' => 10,
    'low_score' => 15,
    'high_score' => 5,
]);

$collection->updateOne(
    ['project' => 12],
    [
        ['$set' => [
            'low_score' => ['$min' => ['$score', '$low_score']],
            'high_score' => ['$max' => ['$score', '$max_score']],
        ]]
    ]
);

var_dump($collection->findOne());

This resulted in the following output, which is what I would have expected:
object(MongoDB\Model\BSONDocument)#12 (1) {
  ["storage":"ArrayObject":private]=>
  array(5) {
    ["_id"]=>
    object(MongoDB\BSON\ObjectId)#11 (1) {
      ["oid"]=>
      string(24) "5dfa57a6280de609357968d2"
    }
    ["project"]=>
    int(12)
    ["score"]=>
    int(10)
    ["low_score"]=>
    int(10)
    ["high_score"]=>
    int(10)
  }
}

PHP 7.3.12
MongoDB extension version => 1.6.1
libbson bundled version => 1.15.2
libmongoc bundled version => 1.15.2
mongodb/mongodb 1.5.1
MongoDB is in Atlas and shows Version 4.0.13
Not sure if it matters, but I'm using updateMany vs updateOne. I edited my comment to reflect that. I'm trying to update all the scores for a particular game at once.

Actually I think I see the problem ... I'm on MongoDB 4.0.13, not 4.2. I'm going to see if I can get the server on 4.2 and try again.

MongoDB is in Atlas and shows Version 4.0.13

Update pipelines definitely require MongoDB 4.2, as noted in both the server release notes and the library's documentation (see: Collection::updateMany()). Do let us know if you find a place where we missed noting the server requirement.
As for the server error your received, it was quite different from what I saw when running MongoDB 4.0.x locally. Are you using the free tier of Atlas by any chance? AFAIK, that has an additional API layer in front of the backing database, which could explain why the error message was so different.

It was a M2 cluster, $9/mo. I'm in the middle of trying to get it switched over to M10 which allows for 4.2.

It was a M2 cluster, $9/mo.

Thanks for clarifying. I should have specified "shared tier" instead of "free" in my last post. I'll follow up with Atlas folks via an internal channel to let them know about the error message inconsistency.

Thank you! That worked. So it was just the fact that I wasn't on 4.2, which was my fault for not checking that
Another question in relation to this. Is there a way to store the timestamp new MongoDB\BSON\UTCDateTime when a new min or max is updated in a min_updated and max_updated field? Using the $min and $max just does it on it's own but I don't know if there is a way to tell if it actually updated the field or not.
So that you could track the date and time that the high or low was recorded.

For non-pipeline updates, MongoDB historically provided a $currentDate operator to return the current date (the same BSON type as PHP's UTCDateTime). An equivalent feature for the aggregation framework was introduced by SERVER-23656 in server version 4.2: the $$NOW variable.
I don't have any examples to refer you to, but I expect you can use a combination of $$NOW with conditional operators to accomplish your goal. I expect the solution may be a bit convoluted since you'll need to determine if either $min or $max actually changed the document by preferring $score over the existing $low_score or $high_score field value, respectively. That said, don't feel pressured to try to put all of this logic into a single $set stage if you think it may be easier to use a series of stages to keep the code more readable. For example, you could compare $low_score and $score in one stage and conditionally set an updatedAt field to $$NOW (likewise for $high_score) and then do the $min and $max stuff in a subsequent, separate stage.

Ok thanks. Appreciate the help.


save file as SVG on mobile devices

Hello,
I have problem with saving my signature as svg on mobile devices.

write something on signature pad
save file using signaturePad.toDataURL('image/svg+xml') and then save it somewhere
load this file using signaturePad.fromDataURL(previousFile)

What is the current behavior?
4. signature pad display only 25% area of svg loaded in left upper corner
What is the expected behavior?
4. signature pad display full file
#440  this solution works for me but for some reason this PR is closed.

See #411

@abraxxa Hi! I don't want to discuss details of this issue in #465, so I'll follow up here.
I created a demo that loads an SVG file (created with this library) via SignaturePad#fromDataURL and as <img> tag: https://jsfiddle.net/szimek/r3qfg4vm/. Both images are displayed correctly, even if I zoom in/out, which changes the value of window.devicePixelRatio.
Could you update this demo to replicate this issue?
I reviewed the code and it looks like it should be correct as it is, i.e.:

canvas height and width is taken from CSS, using canvas.offsetHeight and canvas.offsetWidth;
if canvas size is e.g. 300 x 200px, then its internal coordinates will match this values, i.e. top left corner will have coordinates 0,0 and bottom right one will have 300, 200
the same values are used in viewBox attribute of generated <svg> tag; it uses canvas.height / ratio, but it should be the same as canvas.offsetHeight, because canvas.height is defined as canvas.offsetHeight * ratio, so it's simply undoing this multiplication
given that viewBox value ("0 0 300 200") matches canvas internal coordinates, it should work when using SignaturePad#fromDataURL and if you show it using <img> tag you should be able to adjust its sizes simply via CSS (as in the demo linked in this comment)

I guess I'm missing something, given the number of people who reported this issue, but it would be great to be able to replicate it.

@szimek sorry for the delay but I was able to reproduce it now by disabling your resizeCanvas function: https://jsfiddle.net/gcr2u0e3/
I'm not sure if the https://github.com/szimek/signature_pad#handling-high-dpi-screens section was already in the docs when I started using your library. Can you add the function and auto-resize feature to the library so not every user needs to implement it on her/his own? Maybe behind a new option disabled by default to not cause a breakage on update?

I've played with my Quasar app and found out the following:

you need to call resizeCanvas before fromDataURL
setting the height and width attributes on the canvas tag directly in the html instead of css, the drawn line is multiplied by the devicePixelRatio, for example on my 27" 4k screen with a 1.5 ratio, a 200 pixel mouse movement would draw a 300 pixel line
the wrapper class and position, left and top canvas attributes aren't needed

@abraxxa Thanks for the feedback!
I don't really want to provide a default resize function, because it depends on specific requirements, e.g.

some people might want to store image before resize and then restore it after, * or they might want to listen to onorientationchange instead,
or might want to calculate canvas width/height differently (e.g. element.offsetWidth/Height is the simplest way, but it includes border and padding as well; one would need to use window.getComputedStyle to get more accurate dimensions)
...

But I agree that it's not explained in the README at all that custom logic is needed to properly display things on different screen sizes. I'll try to come up with a better documentation for that, some kind of "Getting started" section.
If you have any suggestions what you'd like to see in this section, let me know.


Add theme: No Gaps

Add theme: No Gaps
This PR adds a new theme to the theme library.
Theme Details

Name: No Gaps
Description: Removes the gaps when the view isn't split.
Homepage:
Author: @Peleret

Thansk!


readme(example) Don't make the user manually launching osv

With this new line the example config works right off the bait just copy pasting it.

Thank you for the PR @Zeioth .
Though I think something is wrong. Correct me but two instances are required to debug with osv. The program field as far as I understand, will just connect back to the same instance which probably results in undesirable behaviour.

@jbyuki let me double check when I'm back home. IDK about possible border cases but I was able to debug successfuly with that just by running
DapContinue
As usual

Maybe I'm missing something as well. I will have a test later.


Task2

issue: #3

Hey @nubEA
Thanks for opening this PR 🚀. Mentor will review your pull request soon and till then, keep contributing and stay calm.
Thanks for contributing in OpenCode'24 ✨✨!

Hey @nubEA
Your PR has been merged 🥳🥳 and you have earned 10 points.
Thanks for contributing in OpenCode'24 ✨✨


Check mEmail for empty String to prevent "credential identifier cannot be empty" error

made code more robust, should fix #141

@iainmcgin please review.

This PR can be closed now: fixed in #357.

#357 merged.


Bootstrap 4 and new styling

This builds on @fabridamazio's work in #363
I've reimplemented the layout and styling rules to make it look shinier and adapt to different size screens better. This PR also implements a working "hamburger" button for toggling the nav UI on mobile screens.

@danroth27 Are you happy with these changes to the third-party notices?

@danroth27 Actually it's possible that the correct license for the Iconic icons is their "font license" (also included in this PR). We are redistributing their font files here. Do you think this is OK?

Actually it's possible that the correct license for the Iconic icons is their "font license" (also included in this PR). We are redistributing their font files here.

@SteveSandersonMS I think you're right. We're not distributing the icons at all, correct?

This PR only uses the font files and theu

This PR also implements a working "hamburger" button for toggling the nav UI on mobile screens.

Can you do this as an externally-visible Property?  I've got some screens where having it around would be preferable, and others where I'd love to be able to hide the nav with a hamburger menu, even on a desktop.

@danroth27 I'm not sure it's only the font license, since it sounds like that applies only to the font files, and we are also using one of their CSS files. I've included both licenses in the notices now. If you still think this is wrong please let me know and we can change it.
@masonwheeler The default template doesn't need any feature like that, and it's important to keep it streamlined. Of course, it's only a starting point, and developers are expected to modify it in whatever way is applicable to their projects.

@SteveSandersonMS 👍


[QUESTION] How to avoid overwriting extracted audio files using ffmpeg

Checklist

[x] I'm asking a question
[x] I've looked through the README and FAQ for similar questions
[x] I've searched the bugtracker for similar questions including closed ones

Question
I'm trying to extract all audio from a Youtube channel using ffmpeg, keeping only the audio file and skipping the audio extraction if the resulting file already exists.
Roughly my command line looks like:
youtube-dl  --audio-format mp3 -x -ciw -o "%(title)s.%(ext)s" -v https://www.youtube.com/channel/<some channel>
I've tried using --no-post-overwrites but this will keep the video files as well.
Using --postprocessor-args -n fails with youtube_dl.postprocessor.ffmpeg.FFmpegPostProcessorError: Error, both -y and -n supplied. Exiting. as -y is added unconditionally.
Is there any way to achieve this, and if so, can you point me out to the right arguments?
Thank you!

I'm facing the same problem. If ID were used in the filename, the answer to https://askubuntu.com/questions/673442/downloading-youtube-playlist-with-youtube-dl-skipping-existing-files would work it seems.

Let's make this a feature request please. Add some option to not overwrite a destination file if it already exists.


Beta schematic rest api

This PR adds functionality for both schematic's reticulate and REST APIs.
This is a version of DCA running schematic's REST API
https://sagebio.shinyapps.io/data_curator-staging/
This is a version of DCA running the reticulate API
https://sagebio.shinyapps.io/data_curator-testing1/


L515 host perf mode refactoring

Follow on L515 stability workaround PR refactoring host performance mode option into customized class. Also allows change TRB settings before, at, and after sensor start. This enables usage changing settings with pipeline on android.
Tracked on: RS5-8011

@maloel Can this PR be closed?


Mast: Watch abiertos.es

Mast requests the watch of the watch_keyword abiertos\.es. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

abiertos\.es has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ


chore: release  service 0.1.7

:package: Staging a new release
0.1.7 (2023-07-27)
Bug Fixes

service: provide labels to ko command as separate flags (#61) (74b9a9f)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/carbynestack/ephemeral/releases/tag/service-v0.1.7 :sunflower:


ui-lib: most peerdeps should be a dependency

in OCM (currently trying to switch to npm):
ERROR in<EMAIL_ADDRESS>8:50-76

Module not found: Error: Can't resolve 'react-tagsinput' in '/container_workspace/node_modules/@openshift-assisted/ui-lib/build/cjs/common/components/ui/formik'

Same for @sentry/browser, 'axios', monaco-editor, etc...
Looking at the package.json file
"peerDependencies": {
    "@reduxjs/toolkit": "^1.9.1",
    "@sentry/browser": "^5.9 || ^6",
    "axios": ">=0.22.0 <1.0.0",
    "i18next": "^20.4 || ^21",
    "monaco-editor": "0.45.0",
    "react": "^17 || ^18",
    "react-dom": "^17 || ^18",
    "react-i18next": ">11.7.3",
    "react-monaco-editor": "^0.55.0",
    "react-redux": "^8.0.5",
    "react-router-dom": "^5.3.0",
    "react-tagsinput": "^3.20",
    "redux": "^4",
    "uuid": "^8.1",
    "yup": "^1.4.0"
  },

Most of these should be dependencies apart from react and react-dom

Now I'm working in a MR to solve problems wit dependencies: https://github.com/openshift-assisted/assisted-installer-ui/pull/2621
I can include these changes also.

@ammont82 I have a branch here https://gitlab.cee.redhat.com/service/uhc-portal/-/merge_requests/5699 if you remove the  overrides in package.json and run npm install you will see them

i'll try it on Monday. Thanks!


numpy prod overflow during creating tensor

I am trying to allocate super large tensor using tensorflow, but failed.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/framework/tensor_util.py#L417
Above code uses numpy.prod to calculate shape size, and for numpy everything is typed, say if the shape is [500000000, 5], then numpy.prod returns -1794967296, it's very easy to reproduce it.
So how about use int64 instead? int64 should be large enough for any tensor.
Changing shape_size = np.prod(shape) to shape_size = np.prod(shape, dtype=np.int64) should fix it. Also about 100 lines of code using np.prod, could we change them all to int64?

@fesun Are you on a 32 bit machine?  I get
>>> np.prod([500000000, 5])
2500000000
>>> np.prod([500000000, 5]).dtype
dtype('int64')

Explicitly setting dtype=np.int64 does seem like the right solution, but I'd like to understand why it's breaking for you first.

Maybe it's a Windows issue? https://docs.scipy.org/doc/numpy/user/basics.types.html#id3 says that the default integer type is the same as a C long, which is 32 bits in MSVC.

@girving I am on 64 bit windows machine, I think @mrry is right, numpy default int type inside windows is 32 bit.

@fesun Would you be up for a pull request?  Adding dtype=np.int64 sounds good.

@girving sure, thanks.


Need to add a timeout when VPN gets stuck in a reconnect

I've got a situation where when the receiving side of my VPN has a connectivity issue for an extended period of time, the calling VPN (on the Synology) ends up getting stuck on the synovpnc connect command. To the point that I end up in an endless loop of task scheduler skipping the scheduled Reconnect VPN task because it is "already running."
I think line 161 of the script should be amended as follows:
https://github.com/ianharrier/synology-scripts/blob/23b01227bd7777748bf56a42264d907504eb5c8f/reconnect-vpn.sh#L161C1-L161C48
/bin/timeout --kill-after=60 /usr/syno/bin/synovpnc connect --id=$PROFILE_ID`

The kill-after argument could be added as a CONST above as well. I'm going to create a pull request on this one. Docs on the standard GNU coreutils timeout command below.

`$ timeout --help
Usage: timeout [OPTION] DURATION COMMAND [ARG]...
  or:  timeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.

Mandatory arguments to long options are mandatory for short options too.
      --preserve-status
                 exit with the same status as COMMAND, even when the
                   command times out
      --foreground
                 when not running timeout directly from a shell prompt,
                   allow COMMAND to read from the TTY and get TTY signals;
                   in this mode, children of COMMAND will not be timed out
  -k, --kill-after=DURATION
                 also send a KILL signal if COMMAND is still running
                   this long after the initial signal was sent
  -s, --signal=SIGNAL
                 specify the signal to be sent on timeout;
                   SIGNAL may be a name like 'HUP' or a number;
                   see 'kill -l' for a list of signals
  -v, --verbose  diagnose to stderr any signal sent upon timeout
      --help     display this help and exit
      --version  output version information and exit

DURATION is a floating point number with an optional suffix:
's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.
A duration of 0 disables the associated timeout.

Well not a good idea to merge this. Looks like the GNU Timeout is broken on the Synology:
/bin/timeout: invalid time interval ‘/usr/syno/bin/synovpnc’

Fixed it.  I will fix the PR.


switch conmon-rs.spec.rpkg to conmon-rs.spec

What happened?
rpkg tool is not supported anymore AFAIK. We should switch to conmon-rs.spec. This would also help in the effort to get conmon-rs into the official distros.
What did you expect to happen?
NA
How can we reproduce it (as minimally and precisely as possible)?
NA
Anything else we need to know?
NA
conmon-rs version
NA
OS version
NA
Additional environment details (AWS, VirtualBox, physical, etc.)
NA

@lsm5 thank you for the report. Do you consider to work on this as well?

@lsm5 thank you for the report. Do you consider to work on this as well?

Sure, I can get something for packit copr and podman-next builds. But I'd rather not be on the hook for the official fedora package whenever we get there :)


About the WER and CER

Sorry to bother you, in your paper you mentioned that Word error rate (WER) and character error rate (CER) between source and converted speech are obtained by an ASR model. about the code for wer and cer calculation. Are you able to open source it?

Yes. You can find it here.


Documentation issue

It's a Fashion MNIST dataset Tesorflow tutorial.What are digits doing here?

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

I think we're talking about the classification of images of numeric arabic digits.

Hi @abdurrehman1998
Yes, this tutorial is using fashion-mnist, this comment is left-over from an earlier version.
The source file for this tutorial is here: https://github.com/tensorflow/models/blob/master/samples/core/tutorials/keras/basic_classification.ipynb
Can you send me a PR with a fix?
The easiest way to do it is to edit in Colab and use the "save a copy to github" option.
Thanks!

Hey guys, I sent a PR to fix this #5032.

Merged. Thanks!


Add secondary platforms workflow

I don't know why some fuser tests are failing on arch64-unknown-linux-gnu.
@Wandalen

Never mind, I set something up wrong.
Note that this PR does not actually run the tests on these other platforms, it just checks that the project compiles.

I don't know why some fuser tests are failing on arch64-unknown-linux-gnu.
@Wandalen

"fuser is the new grep".  Unfortunately, having solved the grep test issues, fuser now intermittently fails in its udp or tcp tests.

Never mind, I set something up wrong.

OK.
As a general comment, it is preferred to add new platform CIs after someone gets the posixutils working on that platform with no exclusions.


Add web archive as a final fallback [WIP]

This implements the web archive as a potential fallback for when the archive is no longer available during a batch download.  This does not allow for downloading removed archives if you do not already have the image metadata in your local database unless you add that data yourself.
You may want to wait to merge (if at all) until the web archive is actually stable.  It seems to go up and down.  I added a warn line right before it downloads the image that indicates it is getting the copy from the internet archive.  Mostly for myself since I'll be using this code and want to make sure the images are pulled properly.
This is for https://github.com/9-FS/nhentai_archivist/issues/39

I'm wrapping the code in a config option in order to not add any extra stress to the server right now.  The code exists but it is disabled by default.  Checks for option ARCHIVE_ORG to be set to true.  There might be more graceful ways to implement this but this is how I've done it.

Moved the web archive code out of the main download loop.  Fixes the following items.

Download failure warnings for downloading from the main servers are restored (they were broken with my original code and I didn't notice at first)
No longer tries to pull from the web archive 5 times.  I didn't notice this before and it was spamming the server pretty badly I guess.

Current patch duplicates the download_image function with an archive_image function and modified it a little to point to the web archive.  It's only called if the main download loop fails (it uses image_download_success to check like the full fail message does then resets it to true for it's own loop) and if webarchive is set to true in config.  Otherwise it should get skipped.

Ah okay, I should've read all of your comments first before making my review comments. :D It's good to see that we're on the same page here that the current duplicated code would not be ideal. But again, I don't really want to download images from the web archive because they're under enough stress as it is, I would only like to download metadata that has been purged from nhentai.

For me the point would be to grab what I can from the IA but at the moment I've been just running it in a different set of configuration on galleries that fail to grab images from the main servers directly then bailing when it throws an error.  At this point I'd already have metadata because it is stored in the DB from the original API pulls.  This isn't setup to get any metadata from IA.


Support MultiWorkerMirroredStrategy distributed training strategy for dynamic embeddings

I tried to explore available approaches for distributed training of large-scale recommendation models with huge embedding tables and tried to use TFRA DynamicEmbedding combined with MultiWorkerMirroredStrategy.

Target task is to train simple two-tower model over online stream of events on multiple CPU workers (model is pretty simple - so no need to train on GPU).
On the first sight, MultiWorkerMirroredStrategy can suite my needs because model will have very small volume of parameters apart from the embeddings - so we can replicate them across all workers

It seems like current implementation struggle with MultiWorkerMirroredStrategy. My attempts to make it works failed with following error:
    ValueError: `colocate_vars_with` must only be passed a variable created in this tf.distribute.Strategy.scope(), not: <tf.Variable 'DynamicEmbedding/user-embedding-shadow:0' shape=(0, 64) dtype=float32, numpy=array([], shape=(0, 64), dtype=float32)>

I tried to launch following training code on 2 workers with following commands:
TF_CONFIG='{"cluster": {"worker": ["localhost:12345", "localhost:23456"]}, "task": {"type": "worker", "index": 0} }' python3 main.py &
TF_CONFIG='{"cluster": {"worker": ["localhost:12345", "localhost:23456"]}, "task": {"type": "worker", "index": 1} }' python3 main.py &

Source code
import dataclasses
from typing import Dict

import tensorflow as tf
import tensorflow_datasets as tfds
# tensorflow_recommenders_addons does some patching on TensorFlow, so it MUST be imported after importing TF
import tensorflow_recommenders as tfrs
import tensorflow_recommenders_addons as tfra
from tensorflow_recommenders_addons import dynamic_embedding as de

redis_config = tfra.dynamic_embedding.RedisTableConfig(redis_config_abs_dir="redis.config")
redis_creator = tfra.dynamic_embedding.RedisTableCreator(redis_config)
batch_size = 4096
seed = 2023

@dataclasses.dataclass(frozen=True)
class TrainingDatasets:
    train_ds: tf.data.Dataset
    validation_ds: tf.data.Dataset

@dataclasses.dataclass(frozen=True)
class RetrievalDatasets:
    training_datasets: TrainingDatasets
    candidate_dataset: tf.data.Dataset

def create_datasets():
    def split_train_validation_datasets(ratings_dataset: tf.data.Dataset) -> TrainingDatasets:
        train_size = int(len(ratings_dataset) * 0.9)
        validation_size = len(ratings_dataset) - train_size
        print(f"Train size: {train_size}")
        print(f"Validation size: {validation_size}")

        shuffled_dataset = ratings_dataset.shuffle(buffer_size=5 * batch_size, seed=seed)
        train_ds = shuffled_dataset.skip(validation_size).shuffle(buffer_size=10 * batch_size).apply(lambda dataset: dataset.padded_batch(batch_size))
        validation_ds = shuffled_dataset.take(validation_size).apply(lambda dataset: dataset.padded_batch(batch_size))

        return TrainingDatasets(train_ds=train_ds, validation_ds=validation_ds)

    ratings_dataset = tfds.load("movielens/1m-ratings", split="train")
    movies_dataset = tfds.load("movielens/1m-movies", split="train").map(lambda x: x["movie_title"])

    for item in ratings_dataset.take(3):
        print(item)

    for item in movies_dataset.take(3):
        print(item)

    training_datasets = split_train_validation_datasets(ratings_dataset)
    return RetrievalDatasets(training_datasets=training_datasets, candidate_dataset=movies_dataset.padded_batch(batch_size))

def train_multi_worker():
    strategy = tf.distribute.MultiWorkerMirroredStrategy()
    datasets = create_datasets()
    train_ds = strategy.experimental_distribute_dataset(datasets.training_datasets.train_ds)

    with strategy.scope() as scope:
        class TwoTowerModel(tfrs.Model):
            def __init__(self, user_model: tf.keras.Model, item_model: tf.keras.Model, task: tfrs.tasks.Retrieval):
                super().__init__()
                self.user_model = user_model
                self.item_model = item_model
                self.task = task

            def compute_loss(self, features: Dict[str, tf.Tensor], training=False) -> tf.Tensor:
                user_embeddings = self.user_model(features["user_id"])
                movie_embeddings = self.item_model(features["movie_title"])
                return self.task(user_embeddings, movie_embeddings)

        def create_de_two_tower_model(candidate_dataset: tf.data.Dataset) -> tf.keras.Model:
            user_model = tf.keras.Sequential([
                de.keras.layers.Embedding(
                    embedding_size=64,
                    key_dtype=tf.string,
                    initializer=tf.random_uniform_initializer(),
                    init_capacity=100_000,
                    restrict_policy=de.FrequencyRestrictPolicy,
                    name="user-embedding",
                    kv_creator=redis_creator,
                    distribute_strategy=strategy
                ),
                tf.keras.layers.Dense(64, activation="gelu"),
                tf.keras.layers.Dense(32),
                tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1))
            ], name='user_model')

            item_model = tf.keras.models.Sequential([
                de.keras.layers.Embedding(
                    embedding_size=64,
                    key_dtype=tf.string,
                    initializer=tf.random_uniform_initializer(),
                    init_capacity=100_000,
                    restrict_policy=de.FrequencyRestrictPolicy,
                    name="movie-embedding",
                    kv_creator=redis_creator,
                    distribute_strategy=strategy
                ),
                tf.keras.layers.Dense(64, activation="gelu"),
                tf.keras.layers.Dense(32),
                tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1))
            ], name='movie_model')

            current_model = TwoTowerModel(user_model, item_model, task=tfrs.tasks.Retrieval(
                metrics=tfrs.metrics.FactorizedTopK(candidate_dataset.map(item_model))
            ))
            current_optimizer = de.DynamicEmbeddingOptimizer(tf.keras.optimizers.Adam())
            return current_model, current_optimizer

        model, optimizer = create_de_two_tower_model(datasets.candidate_dataset)
        model.compile()
    history = model.fit(train_ds, epochs=1, steps_per_epoch=10)
    print(history)

if __name__ == '__main__':
    train_multi_worker()

Redis configuration
{
  "redis_connection_mode": 2,
  "redis_master_name": "master",
  "redis_host_ip": [
    "<IP_ADDRESS>"
  ],
  "redis_host_port": [
    6379
  ],
  "redis_user": "default",
  "redis_password": "",
  "redis_db": 0,
  "redis_read_access_slave": false,
  "redis_connect_keep_alive": false,
  "redis_connect_timeout": 1000,
  "redis_socket_timeout": 1000,
  "redis_conn_pool_size": 20,
  "redis_wait_timeout": 100000000,
  "redis_connection_lifetime": 100,
  "redis_sentinel_user": "default",
  "redis_sentinel_password": "",
  "redis_sentinel_connect_timeout": 1000,
  "redis_sentinel_socket_timeout": 1000,
  "storage_slice_import": 2,
  "storage_slice": 2,
  "using_hash_storage_slice": false,
  "keys_sending_size": 1024,
  "using_md5_prefix_name": false,
  "redis_hash_tags_hypodispersion": true,
  "model_tag_import": "test",
  "redis_hash_tags_import": [
    "{1}",
    "{2}"
  ],
  "model_tag_runtime": "movielens.v6",
  "redis_hash_tags_runtime": [
    "{1}",
    "{2}"
  ],
  "expire_model_tag_in_seconds": 604800,
  "table_store_mode": 2,
  "model_lib_abs_dir": "/tmp/"
}

Relevant information

Are you willing to contribute it not sure (it's must be hard to add it if this is not well supported yet)
Are you willing to maintain it going forward? no
Is there a relevant academic paper? (if so, where): no
Is there already an implementation in another framework? (if so, where): no
Was it part of tf.contrib? (if so, where): no

Which API type would this fall under (layer, metric, optimizer, etc.)

model.fit

Who will benefit with this feature?

This will allow to launch distributed training over multiple workers with large external dynamic embedding table

Hi @sivukhin, thank you for the feedback!  Thank you!

Hi @sivukhin, because of resource lock of TF, the MirroredStrategy for TFRA multi-table is not efficient. We recommend using Horovod for distributed training.
https://github.com/tensorflow/recommenders-addons/blob/master/docs/api_docs/tfra/dynamic_embedding/keras/layers/HvdAllToAllEmbedding.md
https://github.com/tensorflow/recommenders-addons/blob/6f7bbb86a03bf17ee7a8c4b8d36415a2ca1cf693/tensorflow_recommenders_addons/dynamic_embedding/python/keras/layers/embedding.py#L528
https://github.com/tensorflow/recommenders-addons/blob/master/demo/dynamic_embedding/movielens-1m-keras-with-horovod/movielens-1m-keras-with-horovod.py

@MoFHeka, thanks for quick reply!
I will try to use Horovod, sure (I'm not familiar with it but looks like this is mature library which everyone used :-) )
For now I only encountered the problem that latest version of TFRA (0.6.0 in PyPi) doesn't have HvdAllToAllEmbedding. Do you plan to release fresh version of library (it seems like horovod support was added only recently)

@MoFHeka, thanks for quick reply! I will try to use Horovod, sure (I'm not familiar with it but looks like this is mature library which everyone used :-) )
For now I only encountered the problem that latest version of TFRA (0.6.0 on PyPI) doesn't have HvdAllToAllEmbedding. Do you plan to release fresh version of library (it seems like horovod support was added only recently)

Yes, it is not a released feature, but you can try to install from source by following the guidance:  https://github.com/tensorflow/recommenders-addons#installing-from-source. It is easy to do. Any problem can seek help here.
BTW, the next release also will be published soon.

Yes, thanks!
I managed to install fresh TFRA version from sources. I created simple Dockerfile to automate these actions (maybe it will be helpful for someone): https://gist.github.com/sivukhin/da17615df0628a58e4680f7ab48ad8a2
Did HvdAllToAllEmbedding supports training on CPU?
I tried to replace Embedding with HvdAllToAllEmbedding & run training in horovod (horovodrun -np 2 python train.py) in my simple example, but got following error:
[1,1]<stderr>: Expected shape [1682,64] for value, got [1001,64]
[1,1]<stderr>: 	 [[{{node Adam/Adam/update_5/None_lookup_table_insert_2/TFRA>RedisTableInsert}}]] [Op:__inference_train_function_2884]

Error stack trace

[1,0]<stderr>: 2023-11-01 16:22:45.510991: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at redis_table_op.cc:1502 : INVALID_ARGUMENT: Expected shape [1682,64] for value, got [998,64]
[1,0]<stderr>: 2023-11-01 16:22:45.511084: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at redis_table_op.cc:1502 : INVALID_ARGUMENT: Expected shape [1682,64] for value, got [998,64]
[1,0]<stderr>: 2023-11-01 16:22:45.511114: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at redis_table_op.cc:1502 : INVALID_ARGUMENT: Expected shape [1682,64] for value, got [998,64]
  1/100 [..............................] - ETA: 9:31 - factorized_top_k/top_1_categorical_accuracy: 7.3242e-04 - factorized_top_k/top_5_categorical_accuracy: 0.0046 - factorized_top_k/top_10_categorical_accuracy: 0.0059 - factorized_top_k/top_50_categorical_accuracy: 0.0320 - factorized_top_k/top_100_categorical_accuracy: 0.0649 - loss: 34113.6  1/100 [..............................] - ETA: 9:32 - factorized_top_k/top_1_categorical_accuracy: 2.4414e-04 - factorized_top_k/top_5_categorical_accuracy: 0.0029 - factorized_top_k/top_10_categorical_accuracy: 0.0054 - factorized_top_k/top_50_categorical_accuracy: 0.0337 - factorized_top_k/top_100_categorical_accuracy: 0.0637 - loss: 34138.2383 - regularization_loss: 0.0000e+00 - total_loss: 34138.23832023-11-01 16:22:45.511462: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at redis_table_op.cc:1502 : INVALID_ARGUMENT: Expected shape [1682,64] for value, got [1001,64]
[1,1]<stderr>: Traceback (most recent call last):
[1,1]<stderr>:   File "/work/train.py", line 135, in <module>
[1,0]<stderr>: Traceback (most recent call last):
[1,0]<stderr>:   File "/work/train.py", line 135, in <module>
[1,1]<stderr>:     train_multi_worker()
[1,1]<stderr>:   File "/work/train.py", line 130, in train_multi_worker
[1,1]<stderr>:     history = model.fit(datasets.training_datasets.train_ds, epochs=1, steps_per_epoch=100, callbacks=callbacks_list, verbose=1)
[1,1]<stderr>:   File "/usr/local/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
[1,1]<stderr>:     raise e.with_traceback(filtered_tb) from None
[1,1]<stderr>:   File "/usr/local/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
[1,0]<stderr>:     train_multi_worker()
[1,0]<stderr>:   File "/work/train.py", line 130, in train_multi_worker
[1,0]<stderr>:     history = model.fit(datasets.training_datasets.train_ds, epochs=1, steps_per_epoch=100, callbacks=callbacks_list, verbose=1)
[1,0]<stderr>:   File "/usr/local/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
[1,0]<stderr>:     raise e.with_traceback(filtered_tb) from None
[1,0]<stderr>:   File "/usr/local/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
[1,0]<stderr>:     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
[1,1]<stderr>:     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
[1,0]<stderr>: tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
[1,0]<stderr>:
[1,0]<stderr>: Expected shape [1682,64] for value, got [998,64]
[1,0]<stderr>: 	 [[{{node Adam/Adam/update_5/None_lookup_table_insert_3/TFRA>RedisTableInsert}}]] [Op:__inference_train_function_2884]
[1,1]<stderr>: tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
[1,1]<stderr>:
[1,1]<stderr>: Expected shape [1682,64] for value, got [1001,64]
[1,1]<stderr>: 	 [[{{node Adam/Adam/update_5/None_lookup_table_insert_2/TFRA>RedisTableInsert}}]] [Op:__inference_train_function_2884]

Training source code

import dataclasses
import os
from typing import Dict

import horovod.tensorflow as hvd
import tensorflow as tf
import tensorflow_datasets as tfds
# tensorflow_recommenders_addons does some patching on TensorFlow, so it MUST be imported after importing TF
import tensorflow_recommenders as tfrs
import tensorflow_recommenders_addons as tfra
from tensorflow_recommenders_addons import dynamic_embedding as de

hvd.init()

redis_config = tfra.dynamic_embedding.RedisTableConfig(redis_config_abs_dir="redis.config")
redis_creator = tfra.dynamic_embedding.RedisTableCreator(redis_config)
cuckoo_creator = de.CuckooHashTableCreator(saver=de.FileSystemSaver(proc_size=1, proc_rank=0))

batch_size = 4096
seed = 2023

@dataclasses.dataclass(frozen=True)
class TrainingDatasets:
    train_ds: tf.data.Dataset
    validation_ds: tf.data.Dataset

@dataclasses.dataclass(frozen=True)
class RetrievalDatasets:
    training_datasets: TrainingDatasets
    candidate_dataset: tf.data.Dataset

def create_datasets():
    def split_train_validation_datasets(ratings_dataset: tf.data.Dataset) -> TrainingDatasets:
        train_size = int(len(ratings_dataset) * 0.9)
        validation_size = len(ratings_dataset) - train_size
        print(f"Train size: {train_size}")
        print(f"Validation size: {validation_size}")

        shuffled_dataset = ratings_dataset.shuffle(buffer_size=5 * batch_size, seed=seed)
        train_ds = shuffled_dataset.skip(validation_size).shuffle(buffer_size=10 * batch_size).apply(lambda dataset: dataset.padded_batch(batch_size))
        validation_ds = shuffled_dataset.take(validation_size).apply(lambda dataset: dataset.padded_batch(batch_size))

        return TrainingDatasets(train_ds=train_ds, validation_ds=validation_ds)

    ratings_dataset = tfds.load("movielens/100k-ratings", split="train").map(lambda x: {
        'user_id': tf.strings.to_number(x["user_id"], tf.int64),
        'movie_id': tf.strings.to_number(x["movie_id"], tf.int64)
    })
    movies_dataset = tfds.load("movielens/100k-movies", split="train").map(lambda x: tf.strings.to_number(x["movie_id"], tf.int64))

    for item in ratings_dataset.take(3):
        print(item)

    for item in movies_dataset.take(3):
        print(item)

    training_datasets = split_train_validation_datasets(ratings_dataset)
    return RetrievalDatasets(training_datasets=training_datasets, candidate_dataset=movies_dataset.padded_batch(batch_size))

class TwoTowerModel(tfrs.Model):
    def __init__(self, user_model: tf.keras.Model, item_model: tf.keras.Model, task: tfrs.tasks.Retrieval):
        super().__init__()
        self.user_model = user_model
        self.item_model = item_model
        self.task = task

    def compute_loss(self, features: Dict[str, tf.Tensor], training=False) -> tf.Tensor:
        user_embeddings = self.user_model(features["user_id"])
        movie_embeddings = self.item_model(features["movie_id"])
        return self.task(user_embeddings, movie_embeddings)

def build_two_tower_model(candidate_dataset: tf.data.Dataset) -> tf.keras.Model:
    user_model = tf.keras.Sequential([
        de.keras.layers.HvdAllToAllEmbedding(
            embedding_size=64,
            key_dtype=tf.int64,
            value_dtype=tf.float32,
            initializer=tf.random_uniform_initializer(),
            init_capacity=100_000,
            restrict_policy=de.FrequencyRestrictPolicy,
            name="user-embedding",
            kv_creator=redis_creator,
        ),
        tf.keras.layers.Dense(64, activation="gelu"),
        tf.keras.layers.Dense(32),
        tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1))
    ], name='user_model')

    item_model = tf.keras.models.Sequential([
        de.keras.layers.HvdAllToAllEmbedding(
            embedding_size=64,
            key_dtype=tf.int64,
            value_dtype=tf.float32,
            initializer=tf.random_uniform_initializer(),
            init_capacity=100_000,
            restrict_policy=de.FrequencyRestrictPolicy,
            name="movie-embedding",
            kv_creator=redis_creator,
        ),
        tf.keras.layers.Dense(64, activation="gelu"),
        tf.keras.layers.Dense(32),
        tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1))
    ], name='movie_model')

    model = TwoTowerModel(user_model, item_model, task=tfrs.tasks.Retrieval(
        metrics=tfrs.metrics.FactorizedTopK(candidate_dataset.map(item_model))
    ))
    optimize = de.DynamicEmbeddingOptimizer(tf.keras.optimizers.Adam())
    model.compile(optimizer=optimize)
    return model

def train_multi_worker():
    datasets = create_datasets()

    model_dir = f'model_dir_{hvd.rank()}'
    print(f'model_dir: {model_dir}')
    tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=model_dir)
    broadcast_callback = de.keras.callbacks.DEHvdBroadcastGlobalVariablesCallback(root_rank=0)
    checkpoint_callback = de.keras.callbacks.DEHvdModelCheckpoint(
        filepath=model_dir + '/weights_epoch{epoch:03d}_loss{loss:.4f}',
        options=tf.saved_model.SaveOptions(namespace_whitelist=['TFRA'])
    )
    callbacks_list = [tensorboard_callback, broadcast_callback, checkpoint_callback]
    model = build_two_tower_model(datasets.candidate_dataset)
    history = model.fit(datasets.training_datasets.train_ds, epochs=1, steps_per_epoch=100, callbacks=callbacks_list, verbose=1)
    print(history)

if __name__ == '__main__':
    train_multi_worker()

I also tried to launch demo from movielens-1m-keras-with-horovod but didn't succeed at first time (the demo is bit complicated - I don't need it's full functionality + I didn't setup GPU properly - so more likely the major problem on my side).

Hm..ok. I still got this weird error about inconsistent shapes (with Redis & Cuckoo) - but maybe need to dig more into it...

One more thing, if you want to train on CPU, parameter server is your best choice. MirrorStrategy is much more efficient on a single multi-GPU machine.
Why parameter server is better for CPU? We have very simple model with very few weights (apart from embedding table). I thought that multi-worker strategy will be more efficient as it will require only rare communication between workers in order to accumulate updated gradients.
With parameter server I just not sure what will be stored on them... If all dense weights will be there - it seems like this can create huge communication overhead, no?

The lower communication time overhead of multi-worker strategy is based on synchronous training. If many CPU nodes are trained asynchronously with a small batch size, parameter server can complete the training of all samples faster under a specific cluster size.
Another method is semi-synchronous training, the parameters of the dense layer are synchronized by horovod, but the parameters of the embedding are trained asynchronously by PS. You can refer to: semi-synchronous training in TF1 API. Although this demo uses the TF1 API, the principles used in TF2 are similar.
Redis is used as a serving, although you can definitely use it as a Photoshop for training purposes. If you want to use Redis Embedding in horovod synchronization training, use the normal Embedding layer instead of HvdAllToAllEmbedding. In addition, enabling bp_v2 will improve the model convergence effect, and the bp_v2 function of redis requires another compilation of Redis module.

Thanks @MoFHeka, got it!
One last question from my side - does recommenders-addons plans to support more fresh versions of tensorflow in future library releases?

@sivukhin
For now, it will continue to integrate and be compatible with the latest version of Tensorflow, but this is a lot of work. So it would be great if you could also contribute to the TFRA code.

Yes, thanks! I managed to install fresh TFRA version from sources. I created simple Dockerfile to automate these actions (maybe it will be helpful for someone): https://gist.github.com/sivukhin/da17615df0628a58e4680f7ab48ad8a2
Can you provide the compiled pip whl file?


fixing the service provider code

so that it builds Elasticsearch\Client correctly for elasticsearch/elasticsearch version ~2.0.

Oy, sorry I missed that they changed their API completely...I'll tag a new release momentarily.

Thanks!


Conditional processing in country page

Somewhere we need to let the author control which content goes into the export.
The users will have a set of partner suppliers:

teledyne
axiomatic
banchio

We have to let the author specify if a class can be included in a custom export for a particular partner.
We can't do this at the class level, since we must also control if the summary data in the country summary is included.
So, we need to modify the row(s) in the country summary to include a series of 3 checkboxes.  These will be titled teledyne, axiomatic and banchio.
At the publishing phase, users will be able customize the transform to specify the (optional) partner this data is going to.  The export will then only include country summary elements with the relevant tick.  Note: we also can't include the relevant .dita file in the export.
Hmm, that last section seems problematic. I don't really see how the content of france.dita can control which elements of france.dita are published.
From doing some googling the audience parameter may be right solution for this. So, instead of a series of checkboxes, the audience element includes one or more of the above id values.

Could you please tell is it ok to have as list of attributes as above

Yes, but we will need a way for the author to edit which items are in the list.
Aah, hold on. We need the author to specify the items on each row of the table of class items.

Update: here is an example of excluding a group of content:
https://docs.oasis-open.org/dita/dita/v1.3/os/part2-tech-content/langRef/ditaval/ditaval-val.html
More detail on doing it here:
https://www.oxygenxml.com/doc/versions/25.1/ug-editor/topics/profiling-conditional-text.html
So, to "prove" this, I'd like us to define these audience values:

deny-france
deny-spain
deny-england
deny-wales

I'd then like us to:

add deny-france to all data for Spain
add deny-spain to unit_a in `France
add deny-wales to all of the Composite classes in France

Then we can do test exports and verify the correct content is omitted for each deny category.

Re-open this.  We have re-created our mock content, and we need to re-apply the logic to our current data document set.


feat(axllent/mailpit): scaffold axllent/mailpit

axllent/mailpit: An email and SMTP testing tool with API for developers
$ aqua g -i axllent/mailpit

Check List

[x] Read CONTRIBUTING.md

[x] Read OSS Contribution Guide

[x] All commits are signed

This repository enables Require signed commits, so all commits must be signed

[x] Avoid force push
[x] Do only one thing in one Pull Request
[x] Execute cmdx s to scaffold code
[x] Install and execute the package and confirm if the package works well

How to confirm if this package works well
Reviewers aren't necessarily familiar with this package, so please describe how to confirm if this package works well.
Please confirm if this package works well yourself as much as possible.
Command and output
$ mailpit &
$ echo -e "To<EMAIL_ADDRESS>Test Email\n\nThis is a test email."  | mailpit sendmail -f<EMAIL_ADDRESS>$ curl http://localhost:8025/view/latest.txt
This is a test email.

If files such as configuration file are needed, please share them.
Reference

Thank you for your contribution!

v4.203.0 is out 🎉
https://github.com/aquaproj/aqua-registry/releases/tag/v4.203.0


Unable to run CI example (permission denied)

I am trying to run the DataKitCI example under ci/skeleton. When I try to build the container using the command from the readme, I get the following error:
$ docker build -t my-ci --build-arg CONFIG=exampleCI .
Sending build context to Docker daemon  11.26kB
Step 1/9 : FROM datakit/ci
 ---> f16cddfd6f02
Step 2/9 : ARG CONFIG=exampleCI
 ---> Using cache
 ---> c96660dbfaef
Step 3/9 : ADD . /datakit-ci
 ---> Using cache
 ---> 835d5519c636
Step 4/9 : WORKDIR /datakit-ci
 ---> Using cache
 ---> 709a5c6fd874
Step 5/9 : RUN sudo chown opam .
 ---> Using cache
 ---> c083e3152b38
Step 6/9 : RUN opam config exec make $CONFIG && ln _build/skeleton/$CONFIG.native /datakit-ci/datakit-ci && rm -rf _build
 ---> Running in edcdb5c45eaf
ocamlbuild -use-ocamlfind exampleCI.native
Exception Sys_error("/datakit-ci/_build/_log: Permission denied").
make: *** [Makefile:6: exampleCI] Error 100
The command '/bin/sh -c opam config exec make $CONFIG && ln _build/skeleton/$CONFIG.native /datakit-ci/datakit-ci && rm -rf _build' returned a non-zero code: 2

What did I miss? Any help is appreciated.

Actually, I think this example is just broken. Could you try the fix in https://github.com/moby/datakit/pull/619?

Yep, it works now. Thanks for the quick fix @talex5!

The way I'm trying to run the CI example:

Run DataKit server

$ docker run -it --net datakit-net --name datakit -v [path_to_git_repo]:/data -p 5640:5640 datakit/db

Check server by running DataKit client

$ docker run -it --privileged --net datakit-net datakit/client
bash-4.3$ ls -l /db/branch/master/ro/
[git_repos_contents]

From this, and the subsequent server logs, I assume that the server is running and reachable on tcp://<IP_ADDRESS>:5640.

Run CI

$ docker run --name my-ci --net datakit-net -p 8443:8443 my-ci --metadata-store tcp:<IP_ADDRESS>:5640
2018-03-02 13:12.13 INF [datakit-ci] Generating new private key...
2018-03-02 13:12.13 APP [datakit-ci] >>> Configure the CI by visiting
                                     https://<IP_ADDRESS>:8443/auth/intro/SFEq6uYvHCgKJjg1iOrHy4BwIG2heYGc
2018-03-02 13:12.13 INF [datakit-ci] Connecting to DataKit server on tcp:<IP_ADDRESS>:5640
2018-03-02 13:12.13 INF [datakit-ci] Starting monitor loop
2018-03-02 13:12.13 WRN [datakit-ci] Failed to connect to DataKit: (Failure
   "Failed to connect to DataKit server at proto=\"tcp\" addr=\"<IP_ADDRESS>:5640\": (Failure\
  \n  \"Error connecting socket to 9p endpoint tcp:<IP_ADDRESS>:5640: Unix.Unix_error(Unix.ECONNREFUSED, \\\"connect\\\", \\\"\\\")\")") (will retry in 10s)

Could you please point out what I'm doing wrong? (I would create a new issue but I assume this is due to some misconfiguration on my side.)

Okay, it seems like the CI can connect if I use <IP_ADDRESS>:5640 (taken from the output of docker network inspect datakit-net).

You have to give the actual address of the server. <IP_ADDRESS> is a special address that means "listen on any interface", but you can't use it as a destination. You should be able to use --metadata-store tcp:datakit:5640.

Yeah, it works with tcp:datakit:5640, thanks!


feat(EE): exposing worker health endpoint

PR Type
enhancement, configuration changes

Description

Added a new ingress path for the worker health endpoint in the multiwoven-ingress.yaml file.
Configured the worker health endpoint to use a specific host and port.
Updated the chart and application versions to 0.30.0 in the Chart.yaml file.
Introduced a new workerHost configuration in the values.yaml file to support the health endpoint.

Changes walkthrough 📝
Relevant filesConfiguration changes

Chart.yamlBump chart and application version to 0.30.0

charts/multiwoven/Chart.yaml
Updated chart version to 0.30.0.  Updated application version to 0.30.0.

+2/-2

values.yamlAdd workerHost configuration for health endpoint

charts/multiwoven/values.yaml

Added workerHost configuration for worker health endpoint.

+1/-0

Enhancement

multiwoven-ingress.yamlAdd ingress path for worker health endpoint

charts/multiwoven/templates/multiwoven-ingress.yaml
Added new ingress path for worker health endpoint.  Configured path to use worker host and health port.

+10/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

you don't typically need to expose the pod health check endpoints outside of the cluster (unless something else external to the cluster is pinging that endpoint).

We are exposing it so we can configure an uptime monitor in AppSignal for the worker as well.


Update: Known array lengths

As pointed out in https://github.com/ramda/ramda/issues/3364, V8 cares if you instantiate an Array with a known length is far faster to fill than to start with an empty array
This MR updates the following areas of ramda to do exactly that

_clone :: https://jsperf.app/jopaqu - seeing a ~10% gain
intersperse :: https://jsperf.app/dibehu - ~40%
mapAccum :: https://jsperf.app/gozalu - 2% (small)
mapAccumRight, same as ^^
range :: https://jsperf.app/yaxihu - ~45%
scan :: https://jsperf.app/tokefo - ~50%
times :: https://jsperf.app/matoqu - ~50% gain!
xprod :: https://jsperf.app/cezeko - ~75%
zip :: https://jsperf.app/foxowi - seeing a ~15% gain
zipWith :: https://jsperf.app/yasibi - ~ 20%

These seem like some useful improvements, ramda is in dire need of performance. Maybe sync with master and declare it ready for review?

These seem like some useful improvements, ramda is in dire need of performance. Maybe sync with master and declare it ready for review?

ty for the ping on this one @Berndy , I will try to review #3417 in the next day or two

@kedashoe made the updates you requested

@kedashoe I added an optimization for arrays in _clone, but that broke unit tests so I reverted it. Will revisit that in another MR. Need re-approval because of the update

@kedashoe all good again, read for re-approval


Update ItMiiSample* test classes to use v9 cluster resource

https://build.weblogick8s.org:8443/job/weblogic-kubernetes-operator-kind-new/12424/
https://build.weblogick8s.org:8443/job/weblogic-kubernetes-operator-kind-new/12425/
For all 4 test classes ItMiiSampleFmwMain ItMiiSampleFmwAux, ItMiiSampleWlsMain, ItMiiSampleWlsAux, except initial test case (verifying that the domain with 2 managed server starts up successfully ) all the other update cases involving domain rolling restart failed with the same reason as OWLS-101782

Two general comments:

The templates are used to generate files that then should be added to git in the corresponding locations under ./kubernetes/samples/scripts/create-weblogic-domain/model-in-image/domain-resources (the integration tests are expected to detect the discrepancy and fail when this happens).
The MII sample documentation contains copies of the generated domain YAML. The corresponding locations should be replace their text with the new YAML. grep "kind.*Domain" *.md in the documentation/staging/content/samples/domains/model-in-image .


[SMALLFIX] Handle null value for Get Worker

GetWorker should not be communicating with any other components so IOExceptions do not make as much sense as null. @apc999 will consider null annotation after we decide how to enforce.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15817/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/15822/
Test PASSed.