C++ extension incompatibility with linux

What's the reason for C++ extensions not being compatible with linux, but working on alpine? Surely this is something we can fix? My C knowledge may be rather limited but I'm fairly familiar with C++ build systems.

Currently, we use musl-libc and musl-gcc in glibc-based Linux distributions to compile C programs. For the C++ part, we still use g++ and do not use musl because there is currently no suitable musl wrapper for c++.

Is g++ targeting libstdc++ not able to produce statically linked binaries on glibc based systems? I'm trying to understand why it's possible on Alpine, but not on Debian.

It's also possible to compile both c and c++ with musl by building from source or using a prebuilt toolchain.

Is g++ targeting libstdc++ not able to produce statically linked binaries on glibc based systems?

Exactly. And it also cannot be linked statically using glibc with dns things. Using musl-gcc is a compromise.

I see, I was aware that gcc glibc wasn't too happy with static linking, but didn't think that also translated into the C++ world. Things are simpler on Windows with vc.
I'll see if I can set up a cross compiler chain. We already have to compile musl from source for rpm based systems anyway, wouldn't hurt to have spc doctor produce a cross compiler that we can then use to compile c++ extensions targeting musl.

BTW, I have an even weirder idea about compiling musl.

Treat musl as a dependent library and write it in library/linux/musl.php.
When downloading and compiling, the system environment can be detected. If the musl tool chain is required, compile it, otherwise skip it.
If self-compiled musl is used, dynamically specify the cc of the current Builder as path/to/musl-gcc.

Doing so will result in a better user experience, but will greatly increase the complexity of the framework.

Or do you think it's simpler to just offer a alpine docker image to let users create php with the extensions they want?

I don't think it's appropriate to change this now. Because the upcoming official version will provide spc binaries for multiple platforms, and this binary theory can run on any supported distribution, or you can call --docker on unsupported distributions to build a Docker Mirror and run.

The libphp.a/php binary could still be used on all linux distros, I suppose?

I feel that if libphp wants to be used truly smoothly, it may be possible to treat spc itself as part of the compilation process rather than as an independent build program. In addition, currently we can rely on buildroot/bin/php-config to handle these parameters, but I also plan to add a command that can output the compilation parameters, such as ./spc export-embed-vars --libs etc. (Maybe this Do it better? I'm not very clear about the usage of libphp).

I feel that if libphp wants to be used truly smoothly, it may be possible to treat spc itself as part of the compilation process rather than as an independent build program.

I think that goes a bit too far. It's not too much to ask to spin up a VM and install php to run spc. The process is simple enough for the target audience this project has. I expect we would run into too many issues trying to make it a pre-compiled binary that does everything for the user.

In addition, currently we can rely on buildroot/bin/php-config to handle these parameters, but I also plan to add a command that can output the compilation parameters, such as ./spc export-embed-vars --libs etc. (Maybe this Do it better? I'm not very clear about the usage of libphp).

I'm not sure I understand. What is bin/php-config? Libphp.a can be used to statically link the php runtime into other programs (such as web servers like frankenphp).

I'm not sure I understand. What is bin/php-config? Libphp.a can be used to statically link the php runtime into other programs (such as web servers like frankenphp).

The first mention of it comes from here: https://github.com/crazywhalecc/static-php-cli/pull/153#issuecomment-1713959846
But, it's also just an idea, it's not in my current plans, and maybe it might be implemented later than Windows support. There is still a lot of work to be done before the official binary release.

I think that goes a bit too far. It's not too much to ask to spin up a VM and install php to run spc.

For this part of the content, I initially planned to build a GitHub Actions repository for self-configurable compilation in other systems and projects: https://github.com/crazywhalecc/setup-static-php
For local builds, everything will be based on spc binary, which will greatly simplify the installation environment of spc itself. You're right: programmers who are involved in compilation, or even programmers who are involved in libphp, may never be troubled by the compilation environment. But what we can do is to simplify this process as much as possible. Many people around me will give up using a project because the installation and deployment of it are too complicated.

I think that goes a bit too far. It's not too much to ask to spin up a VM and install php to run spc.

For automated build, I initially planned to build a GitHub Actions repository for self-configurable compilation in other systems and projects: https://github.com/crazywhalecc/setup-static-php

Yes, in my opinion this would be enough. It's reasonable to spin up an alpine image for development purposes on c++ extensions. If they can be built by a Github action that is simple to use in the end, I'm sure no user would object.
I will add the c++/alpine disclaimer in the readme, though, so new contributors can understand it better. Maybe I'll set up a cross-compiler chain for non-alpine when I'm bored, but it's not a priority then.

Good news, getting self compiled cross-musl running was very easy. I will create a pr with a full cross compile install.
[19:45:08] [INFO] Deploying cli file
[19:45:08] [INFO] [EXEC] cp '/opt/static-php-cli/source/php-src/sapi/cli/php' '/opt/static-php-cli/buildroot/bin/'
[19:45:08] [DEBU] Running command with direct output: cp '/opt/static-php-cli/source/php-src/sapi/cli/php' '/opt/static-php-cli/buildroot/bin/'
[19:45:08] [INFO] running cli sanity check
[19:45:08] [INFO] [EXEC] /opt/static-php-cli/buildroot/bin/php -r "echo \"hello\";"
[19:45:08] [DEBU] testing ext: snappy
[19:45:08] [DEBU] Running command with result: /opt/static-php-cli/buildroot/bin/php --ri "snappy"
[19:45:08] [INFO] Build complete, used 63.543 s !
[19:45:08] [INFO] Static php binary path: /opt/static-php-cli/buildroot/bin/php
[19:45:08] [DEBU] Removing path recursively: "/opt/static-php-cli/buildroot/license"
[19:45:08] [DEBU] scanning directory /opt/static-php-cli/buildroot/license
[19:45:08] [DEBU] Making new directory recursive: /opt/static-php-cli/buildroot/license
[19:45:08] [INFO] License path: /opt/static-php-cli/buildroot/license/

It does take a little while, though. ~5 minutes on my 5950x system, so it might definitely be 30+ minutes on a single core.

Treat musl as a dependent library and write it in library/linux/musl.php

I'm thinking about this, but I'm not sure it's a great idea. Notably, at times it can make sense to rebuild all libraries, such as when extensions change, but you wouldn't want to recompile musl.


Container Instance Resource implementation

Features:

Now implements a basic model of Container Instance Resources (providing defaults based on C4.Xlarges - future work to base this on the EC2 instance type to follow) - existing tests updated and new ones added for this
Check for ability to place a task on each instance - refuse to place on a task without the MEMORY, CPU or PORTS reminding to place the task on the instance - new tests for this
Reserve resources on start of a task, free up on removal - new tests added for this

Bug Fixes:

Now able to use container instance ARNs for describe_container_instances - one new test covers this case

I'm not sure I've followed your preferred practice in all the cases, creating a few private helper methods for calculating resources given the AWS implementation as a list of dicts rather than duplicate code, happy to rework if need be.

Coverage increased (+0.02%) to 93.511% when pulling 880f3fb9506c9dd70c50e75a53a78148b402cba4 on gjtempleton:ECSContainerResources into bba197e29f7fdd635fcbcb2cd18993d839f93f39 on spulec:master.

This is great! Thanks for the contribution.


Adds minSearchLength & maxResults to MentionsInput properties.

For larger data sets, the potential suggestion list is very large.
This pull request adds two properties to help with this:

minSearchLength: The minimum number of characters required to initiate a search.  This is to prevent the list from triggering after a single character has been input.
maxResults: The maximum number of results to display.

README.md has also been updated to correspond to changes.

This seems like something that should be implemented in user land. You can already provide a function for the data prop, which allows you to only return results under certain conditions and to filter and restrict the results however you want.
Following the principle of minimal API surface, we will only add new props when there's no way to achieve something in another way.
Thank you anyways for your contribution. For the next time to avoid frustration, it might be better to first create an issue before starting implementation.


Find in Explorer can throw in some scenarios when rerendering the match highlights

Verified. Tree view had the same error and the same fix: https://github.com/microsoft/vscode/pull/170680


Add intersphinx feature

The Intersphinx extension allows to reference an object from another code in a documentation generated with Sphinx.
The doc of MorphIO is generated using Doxygen so it can not be cross linked with intersphinx by default. Maybe using this method would be nice to overcome this:
https://docs.typo3.org/typo3cms/extensions/sphinx/2.5.1/AdvancedUsersManual/AdvancedCrossLinks/Index.html#doxygen-documentation
So we could reference the morphio types in other documentations.

The objects.inv file required by Intersphinx is actually built, it's just it is very incomplete. #439 should fix this.


Add support for multiple key::value pairs per line

I want to use DataView fields in my notes and have their values autocompleted. However, autocomplete does not work if you include the brackets (or parentheses).
This works:

This does not:

Here is the field as defined in my fileClass file:

Originally posted by @zackdn in https://github.com/mdelobelle/metadatamenu/issues/10#issuecomment-1192681471

closed with #47


Fix(Provider/Matrix): CertSecretRef

This PR fixes #316 and allows using custom CA certificates with the Matrix provider.

@Nold360 can you please signoff your commit, instructions here: https://github.com/fluxcd/notification-controller/pull/318/checks?check_run_id=4949662575

@Nold360 can you please signoff your commit, instructions here: https://github.com/fluxcd/notification-controller/pull/318/checks?check_run_id=4949662575

Should be fixed now


onPollWindow_: use startTime when currentTime is not ready yet (fixes #2848)

There is a race condition: it could happen that onPollWindow_ checks falling outside the availability window before videoElement returns the currentTime correctly (it returns 0 instead) so the following calculation leads to incorrect jump forward. Now it will use this.videoWrapper_.getTime() which returns startTime in that case instead.

@googlebot I fixed it.

@JakubRimal, thanks!  Is there a particular issue associated with this fix?  Do all the tests still pass?

Doh, it's there in the title.  I overlooked it!

Do all the tests still pass?

./build/all.py – yes
./build/test.py – actually it does not pass because it does not pass even without my commit but the result (please see the following output) is completely the same for the last commit from master (67a49d45e37ecf5d11d78aead6cc76d3f7782c4d) and for my comit (ebdea3ede7bf506ca82a42898679e69f8a9fe5f4) which means my commit did not break it.
[WARNING] Using default browsers: ['Chrome', 'Firefox']
[INFO] Generating Closure dependencies...
[INFO] Compiling the library (ui, release)...
[WARNING] No changes detected, skipping. Use --force to override.
[INFO] Running test (1 / 1, 0 failed so far)...
Chrome 85.0.4183.83 (Fedora 0.0.0) CastUtils serialize/deserialize TimeRanges deserialize into equivalent objects FAILED
    Error: Failed: Type negotiation should happen before MediaSourceEngine.init!
        at <Jasmine>
        at Function.jasmineAssert (test/test/boot.js:38:7 <- test/test/boot.js:41:7)
        at _loop (lib/media/media_source_engine.js:296:20 <- lib/media/media_source_engine.js:7145:32)
        at _class._callee2$ (lib/media/media_source_engine.js:294:53 <- lib/media/media_source_engine.js:7197:19)
        at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
    NotSupportedError: Failed to execute 'addSourceBuffer' on 'MediaSource': The type provided ('video/mp4; codecs="avc1.42c01e"') is unsupported.
    error properties: Object({ INDEX_SIZE_ERR: 1, DOMSTRING_SIZE_ERR: 2, HIERARCHY_REQUEST_ERR: 3, WRONG_DOCUMENT_ERR: 4, INVALID_CHARACTER_ERR: 5, NO_DATA_ALLOWED_ERR: 6, NO_MODIFICATION_ALLOWED_ERR: 7, NOT_FOUND_ERR: 8, NOT_SUPPORTED_ERR: 9, INUSE_ATTRIBUTE_ERR: 10, INVALID_STATE_ERR: 11, SYNTAX_ERR: 12, INVALID_MODIFICATION_ERR: 13, NAMESPACE_ERR: 14, INVALID_ACCESS_ERR: 15, VALIDATION_ERR: 16, TYPE_MISMATCH_ERR: 17, SECURITY_ERR: 18, NETWORK_ERR: 19, ABORT_ERR: 20, URL_MISMATCH_ERR: 21, QUOTA_EXCEEDED_ERR: 22, TIMEOUT_ERR: 23, INVALID_NODE_TYPE_ERR: 24, DATA_CLONE_ERR: 25, code: 9 })
    Error: Failed to execute 'addSourceBuffer' on 'MediaSource': The type provided ('video/mp4; codecs="avc1.42c01e"') is unsupported.
        at _loop (lib/media/media_source_engine.js:311:48 <- lib/media/media_source_engine.js:7168:85)
        at _class._callee2$ (lib/media/media_source_engine.js:294:53 <- lib/media/media_source_engine.js:7197:19)
        at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
        at Generator.invoke [as _invoke] (node_modules/babel-polyfill/dist/polyfill.js:7138:22)
        at Generator.prototype.<computed> [as next] (node_modules/babel-polyfill/dist/polyfill.js:6952:21)
        at step (lib/media/media_source_engine.js:6705:191)
        at lib/media/media_source_engine.js:6705:361
    Error: Expected 0 to be greater than 0.
        at <Jasmine>
        at UserContext.<anonymous> (test/cast/cast_utils_unit.js:251:33 <- test/cast/cast_utils_unit.js:363:33)
        at <Jasmine>
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls overflow menu allows picture-in-picture only when the content has video FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls resolutions menu clears the buffer when changing resolutions FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls resolutions menu displays resolutions based on current stream FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) CastReceiver sends update messages every stage of loading FAILED
        [object Object] thrown
Chrome 85.0.4183.83 (Fedora 0.0.0) ERROR
  An error was thrown in afterAll
  Failed: Unhandled rejection in Promise: shaka.util.Error {
    "severity": 2,
    "category": 4,
    "code": 4032,
    "data": [],
    "handled": false,
    "message": "Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()"
  }
  Error: Failed: Unhandled rejection in Promise: shaka.util.Error {
    "severity": 2,
    "category": 4,
    "code": 4032,
    "data": [],
    "handled": false,
    "message": "Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()"
  }
      at <Jasmine>
      at test/test/boot.js:56:5 <- test/test/boot.js:59:5
Chrome 85.0.4183.83 (Fedora 0.0.0) ERROR
  Disconnected, because no message in 300000 ms.
Chrome 85.0.4183.83 (Fedora 0.0.0) CastUtils serialize/deserialize TimeRanges deserialize into equivalent objects FAILED
        Error: Failed: Type negotiation should happen before MediaSourceEngine.init!
            at <Jasmine>
        at Function.jasmineAssert (test/test/boot.js:38:7 <- test/test/boot.js:41:7)
        at _loop (lib/media/media_source_engine.js:296:20 <- lib/media/media_source_engine.js:7145:32)
        at _class._callee2$ (lib/media/media_source_engine.js:294:53 <- lib/media/media_source_engine.js:7197:19)
        at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
    NotSupportedError: Failed to execute 'addSourceBuffer' on 'MediaSource': The type provided ('video/mp4; codecs="avc1.42c01e"') is unsupported.
    error properties: Object({ INDEX_SIZE_ERR: 1, DOMSTRING_SIZE_ERR: 2, HIERARCHY_REQUEST_ERR: 3, WRONG_DOCUMENT_ERR: 4, INVALID_CHARACTER_ERR: 5, NO_DATA_ALLOWED_ERR: 6, NO_MODIFICATION_ALLOWED_ERR: 7, NOT_FOUND_ERR: 8, NOT_SUPPORTED_ERR: 9, INUSE_ATTRIBUTE_ERR: 10, INVALID_STATE_ERR: 11, SYNTAX_ERR: 12, INVALID_MODIFICATION_ERR: 13, NAMESPACE_ERR: 14, INVALID_ACCESS_ERR: 15, VALIDATION_ERR: 16, TYPE_MISMATCH_ERR: 17, SECURITY_ERR: 18, NETWORK_ERR: 19, ABORT_ERR: 20, URL_MISMATCH_ERR: 21, QUOTA_EXCEEDED_ERR: 22, TIMEOUT_ERR: 23, INVALID_NODE_TYPE_ERR: 24, DATA_CLONE_ERR: 25, code: 9 })
    Error: Failed to execute 'addSourceBuffer' on 'MediaSource': The type provided ('video/mp4; codecs="avc1.42c01e"') is unsupported.
        at _loop (lib/media/media_source_engine.js:311:48 <- lib/media/media_source_engine.js:7168:85)
        at _class._callee2$ (lib/media/media_source_engine.js:294:53 <- lib/media/media_source_engine.js:7197:19)
        at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
        at Generator.invoke [as _invoke] (node_modules/babel-polyfill/dist/polyfill.js:7138:22)
        at Generator.prototype.<computed> [as next] (node_modules/babel-polyfill/dist/polyfill.js:6952:21)
        at step (lib/media/media_source_engine.js:6705:191)
        at lib/media/media_source_engine.js:6705:361
    Error: Expected 0 to be greater than 0.
        at <Jasmine>
        at UserContext.<anonymous> (test/cast/cast_utils_unit.js:251:33 <- test/cast/cast_utils_unit.js:363:33)
        at <Jasmine>
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls overflow menu allows picture-in-picture only when the content has video FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls resolutions menu clears the buffer when changing resolutions FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) UI controls resolutions menu displays resolutions based on current stream FAILED
        Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()
Chrome 85.0.4183.83 (Fedora 0.0.0) CastReceiver sends update messages every stage of loading FAILED
        [object Object] thrown
Chrome 85.0.4183.83 (Fedora 0.0.0) ERROR
  An error was thrown in afterAll
  Failed: Unhandled rejection in Promise: shaka.util.Error {
    "severity": 2,
    "category": 4,
    "code": 4032,
    "data": [],
    "handled": false,
    "message": "Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()"
  }
  Error: Failed: Unhandled rejection in Promise: shaka.util.Error {
    "severity": 2,
    "category": 4,
    "code": 4032,
    "data": [],
    "handled": false,
    "message": "Shaka Error MANIFEST.CONTENT_UNSUPPORTED_BY_BROWSER ()"
  }
      at <Jasmine>
      at test/test/boot.js:56:5 <- test/test/boot.js:59:5
Chrome 85.0.4183.83 (Fedora 0.0.0): Executed 1744 of 1988 (5 FAILED) (skipped 30) ERROR (24.181 secs / 21.252 secs)
Firefox 80.0 (Fedora 0.0.0): Executed 1905 of 1988 (skipped 83) SUCCESS (6 mins 3.424 secs / 5 mins 58.52 secs)
Chrome 85.0.4183.83 (Fedora 0.0.0): Executed 1744 of 1988 (5 FAILED) (skipped 30) ERROR (24.181 secs / 21.252 secs)
[INFO] Run complete
[INFO] Result (exit code): 1

Those test failures are concerning because we don't see the same failures.  I would be interested to find out what is different about your environment so that we could improve our tests.  Can you please file a new issue about that so that we can discuss?
My only concern with your PR is that I would prefer to have a regression test or conformance rule (see build/conformance.textproto) to make sure we don't make this mistake again in the future.  Looking at the code, thought, I'm not sure if we can use a conformance rule for this, given that SrcEqualsPlayhead in the same file needs to use mediaElement_.currentTime, as there is nothing else in that context.  So I think a regression test would be best.

Those test failures are concerning because we don't see the same failures. I would be interested to find out what is different about your environment so that we could improve our tests. Can you please file a new issue about that so that we can discuss?

Yes, please see https://github.com/google/shaka-player/issues/2865

I will go ahead and submit this, then follow up with a regression test myself. I think it can be done fairly easily in the Playhead unit tests. Thanks!

Thank you very much, I was going to write you that I am afraid that I am not familiar with tests enough to be able to write it for this scenario so I am happy to hear that you'll do it and that it could be "fairly easily". :slightly_smiling_face:

The regression test was added in d20b9f0fda6c57e84db84b15d2a3bcdd78bcfd14 if you are interested to see how it works.  "Easy" is relative, I suppose.  It required us to emulate a very specific scenario and then suppress some other internal mechanisms to more easily measure the effect of the bug.


Change log level on the fly without restart

Overview
Currently the log level can be configured via environment variable NSM_LOG_LEVEL which means the process will use the configured value for its whole lifetime. It is fine in most of the cases, but when one would like to change the log level because closer monitoring is needed, it would be expected to be able to do the change without restarting the process that would be monitored. In the vast majority of the cases the restart makes impossible the immediate monitoring action, because after the configuration change we should initiate restart and then just wait for the issue to happen again.
For temporary setting it would be good if the log level could be changed via sending IPC signals to the process. For example, if we would like to monitor forwarder-vpp then we send SIGUSR1 to the process and it changes its log level to TRACE, when we captured the needed logs the original log level can be restored by sending signal SIGUSR2.

Hi,
We also ha a reference implementation, if this  signalling approach is okay with you guys
https://github.com/networkservicemesh/cmd-forwarder-vpp/pull/1171

Possible solution:

Add term "NSM config"
Move LOG_LEVEL configuring from ENV config to NSM config
[Optional] In future - move other ENVs

Algorithm for LOG_LEVEL:

Every cmd repo will contain structure, that we will read data from file
On every request there will be check if the file has been modified, in positive case - read new value and update LOG_LEVEL in structure
In scope k8s deployment - we will store data in configmap and mount it into each pod as file

cc @edwarnicke

Hi, did the prs for the cmds, here's a list:
https://github.com/networkservicemesh/cmd-cluster-info-k8s/pull/157
https://github.com/networkservicemesh/cmd-forwarder-ovs/pull/420
https://github.com/networkservicemesh/cmd-forwarder-sriov/pull/778
https://github.com/networkservicemesh/cmd-ipam-vl3/pull/221
https://github.com/networkservicemesh/cmd-lb-vl3-vpp/pull/107
https://github.com/networkservicemesh/cmd-map-ip-k8s/pull/246
https://github.com/networkservicemesh/cmd-nsc-init/pull/785
https://github.com/networkservicemesh/cmd-nsc/pull/655
https://github.com/networkservicemesh/cmd-nsc-simple-docker/pull/256
https://github.com/networkservicemesh/cmd-nsc-vpp/pull/723
https://github.com/networkservicemesh/cmd-nse-firewall-vpp/pull/541
https://github.com/networkservicemesh/cmd-nse-icmp-responder/pull/621
https://github.com/networkservicemesh/cmd-nse-icmp-responder-vpp/pull/705
https://github.com/networkservicemesh/cmd-nse-l7-proxy/pull/226
https://github.com/networkservicemesh/cmd-nse-remote-vlan/pull/249
https://github.com/networkservicemesh/cmd-nse-simple-vl3-docker/pull/271
https://github.com/networkservicemesh/cmd-nse-vfio/pull/538
https://github.com/networkservicemesh/cmd-nse-vl3-vpp/pull/368
https://github.com/networkservicemesh/cmd-nsmgr-proxy/pull/523
https://github.com/networkservicemesh/cmd-nsmgr/pull/710
https://github.com/networkservicemesh/cmd-registry-memory/pull/689
https://github.com/networkservicemesh/cmd-registry-proxy-dns/pull/670
I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?
https://github.com/networkservicemesh/cmd-exclude-prefixes-k8s/pull/319
https://github.com/networkservicemesh/cmd-nse-supplier-k8s/pull/355
https://github.com/networkservicemesh/cmd-registry-k8s/pull/481

Hello @arp-est ,
Many thanks for doing this.

I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?

Yeah, we have faced a problem in SDK K8s, and those components were not updated. Now that the problem is resolved, could you please rebase these PRs?

Hello @arp-est ,
Many thanks for doing this.

I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?

Yeah, we have faced a problem in SDK K8s, and those components were not updated. Now that the problem is resolved, could you please rebase these PRs?

Right away

Hi, I rebased it but in those prs its not the sdk-k8s, but the regular sdk, that is missing the SetupLevelChangeOnSignal function, I see only the version of sdk-k8s was updated in the go.mod file.

It seems like it's done.
Many thanks, @arp-est.
Created a ticket to add an integration test for this feature: https://github.com/networkservicemesh/deployments-k8s/issues/12522.
Closing this one.


Post job ad SMHI

Description

Mention the ESMValTool job ad at SMHI

Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review.

[ ] New functionality is working (e.g. links, images showing correctly, etc.)
[ ] Content is appropriate
[ ] Spelling and grammar are correct
[ ] All checks below this pull request were successful

Is there some way to see if the website renders correctly?


Transitive P2P: Tell solution build manager about the PackageRefs which are projects.

Solution build manager needs to know about the P2Ps which are transitive. The entire closure is probably not needed since the solution will build in dependency order.

Closing this since we have decided that all project references will transitive by default for .Net Core Projects


fix: Improve error message for SQLitePersister

I always forget about calling .initialize() on the persister to create tables and that's the error I get:
  File .../assistant.py", line 86, in build_assistant
    .build()
     ^^^^^^^
  File ".../burr/telemetry.py", line 276, in wrapped_fn
    return call_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../application.py", line 2333, in build
    self._load_from_persister()
  File ".../burr/core/application.py", line 2262, in _load_from_persister
    load_result = self.initializer.load(_partition_key, _app_id, _sequence_id)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../burr/core/persistence.py", line 242, in load
    cursor.execute(
sqlite3.OperationalError: no such table: burr_state

Notes:

it's a bit odd that the stack trace bounces between user code -> burr.telemetry -> user code -> burr.core.persistence
could catch the uninitialized persister at ApplicationBuilder().with_state_persistence(...) instead of ApplicationBuilder().build(). (seems like a unlikely pattern to pass the persister then initialize it later)
error message should say RuntimeError: Uninitialized persister. Make sure to call .initialize() before passing it to the ApplicationBuilder

@zilto can I take it up?

@arpitgupta-it Yes, you can work on it! Do you have any question? The goal is to catch more explicitly the missing table when calling self.initializer.load(...) and raise an appropriate error message

I just raised a PR. Please review if it looks good to approve.

@arpitgupta-it thanks I just added comments. We need to extend it to cover all persisters and maintain backwards compatibility with custom persisters people may have written.


Clicks not working on my Landing page in PhoneGap App

HI,
I am new to app build, I am modifying an app I bought it from a developer,
My app get stuck at landing page and clicks doesn't work, on speaking to the developer I was told it's limitation of phonegap desktop app on windows platform, as it won't support 3rd party plugin.
I tried by adding plugins externally using "Cordova plugin add" but no luck yet.
config.xml have with following plugins.
<gap:plugin name="org.apache.cordova.geolocation"  source="pgb" version="0.3.7" />
	<gap:plugin name="com.ionic.keyboard" source="pgb" version="1.0.3" />
	<gap:plugin name="org.apache.cordova.device" source="pgb" version="0.2.12" />
	<gap:plugin name="org.apache.cordova.device-orientation" source="pgb" version="0.3.9" />
	<gap:plugin name="org.apache.cordova.dialogs" source="pgb" version="0.2.10" />
	<gap:plugin name="org.apache.cordova.file-transfer" source="pgb" version="0.4.6" />
	<gap:plugin name="org.apache.cordova.inappbrowser" source="pgb" version="0.5.2" />
	<gap:plugin name="org.apache.cordova.network-information" source="pgb" version="0.2.12" />
	<gap:plugin name="org.apache.cordova.splashscreen" source="pgb" version="0.3.4" />

Hi @sidn041,
Since we now use npm scripts for the project, simply running PhoneGap/Cordova commands will not work properly. Please follow [these] (https://github.com/phonegap/phonegap-app-developer/issues/372#issuecomment-159705782) steps in order to add a 3rd party plugin to the app.

considering this answered.


Usage with Swift

Hi,
I suppose It's no t a bug, but I'm trying to uncompress a downloaded file in aplication's document folder but I don't find anyway of do it.
I've take the recipe in https://github.com/pixelglow/ZipZap/wiki/Recipes named "Unzip an existing zip file to the file system" and I've tried to translate to Swift 2.0 for Xcode 7 but I think i'm doing something wrong cause I don't know Objetive C. Please, can anyone help me?. I also think It would be a great Idea adding swift recipes to the wiki.
Thanks a lot to everybody.
David

Can you provide a little bit more about what is going wrong?


Unable to disable colored output

Issue Type

Bug report

Molecule and Ansible details
ansible 2.8.3
molecule, version 2.22

Molecule installation method (one of):

pip

Ansible installation method (one of):

source

Detail any linters or test runners used:

none

Desired Behavior
Colors should be stripped from output if PY_COLORS=0 is defined or if the output is redirected to a file.
$ PY_COLORS=0 molecule list > some_file
$ xxd some_file
00000000: 2d2d 3e20 5661 6c69 6461 7469 6e67 2073  --> Validating s

Actual Behaviour
Control characters defining colors are not stripped.
$ PY_COLORS=0 molecule list > some_file
$ xxd some_file
00000000: 2d2d 3e20 1b5b 3336 6d56 616c 6964 6174  --> .[36mValidat

If I rollback changes introduced in fecf50b7e4f2bc19d3981b645b3c9c267efc95a9, color stripping works as expected.

Have you tried using TERM=dumb?

Unfortunately, the result is the same:
$ TERM=dumb molecule list | xxd
00000000: 2d2d 3e20 1b5b 3336 6d56 616c 6964 6174  --> .[36mValidat

I am unable to reproduce it with code from master.

molecule 3.0.2
   ansible==2.9.6 python==3.8

This is still a present issue. PY_COLOR=0 appears to remove color from the help text but fails to remove control characters from general execution.

Help text

bash-5.0# PY_COLORS=1 molecule | xxd
00000000: 1b5b 306d 5573 6167 653a 206d 6f6c 6563  .[0mUsage: molec

bash-5.0# PY_COLORS=0 molecule | xxd
00000000: 5573 6167 653a 206d 6f6c 6563 756c 6520  Usage: molecule

molecule test

bash-5.0# PY_COLORS=0 molecule test | xxd
00000000: 2d2d 3e20 1b5b 3336 6d54 6573 7420 6d61  --> .[36mTest ma

logger.should_do_markup parses the PY_COLORS environment variable which is then passed to strip in colorama.init. However, click is passed a HelpColorGroup for all command groups which may be the source of the remaining control characters.


Cloud9 Deployment running into errors

Hi again, I'm deploying the Cloud9 installation flow and I'm getting the following error:
`Deploying CDK Application
Try to get prebuilt lambda
Try to get prebuilt lambda
[Error at /MLServerlessStack/MLServerlessStack-base] You are not authorized to perform this operation.
Found errors`
This looks like permission errors (I could be wrong), is there a way to debug this to find out exactly where the error is? Because it seems I have full permissions for all AWS services mentioned in this repo, so this is a bit surprising. Thanks again.

@InquestGeronimo It does look like permission errors.  Please ensure your identity or the IAM Role attached to the Cloud9 instance (if any) has the appropriate permissions to create the resources.  I would start with the resources being created with the code in /app/lib/base-resources.ts (VPC, Subnets, Security Groups, etc..)

thanks I'm double checking this, will keep you updated.

@medarg Hi Medarg, I passed the permission issues, thank you for pointing me in the right direction. I have your default setup deployment working however now I'm trying to deploy my own runtime (deepsparse) and as a result, i'm no longer using mxnet. However I'm getting errors in the stack when attempting to dpleoy almost as though it's sensing that it is not mxnet. What do you propose is the best method for using a custom layer in the realtime_inf directory? Is it as simple as substituting the folders in the zip file? or do I need to edit other things in this repo? My main concern right now is getting the zip folder in good order, which I believe is the problem area that is is currently giving me the errors.
Here is the error :
`Embedded stack arn:aws:cloudformation:us-east-1:831168642639:stack/MLServerlessStack-MLServerlessStacklambdasNestedStackMLServerlessStacklambdasNestedSta-1W0OM6VTB8ZTE/6f183bd0-a325-11ed-861d-0a6e48580663 was not successfully created: The following resource(s) f
ailed to create: [MLServerlessStackmxnetlayerF8898654].
❌  MLServerlessStack failed: Error: The stack named MLServerlessStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-east-1:831168642639:stack/MLServerlessStack-MLServerlessStacklambdasNestedStackMLServerlessStacklambdasNestedSta-1W0OM6VTB8ZTE/6f183bd0-a325-11ed-861d-0a6e48580663 was not successfully created: The following resource(s) failed to create: [MLServerlessStackmxnetlayerF8898654].
at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at deployStack2 (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24)
at /usr/local/lib/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/usr/local/lib/node_modules/p-queue/dist/index.js:163:29)`


Fluctuating values in twist publisher

Manually computing dt in twist publisher results in erratic fluctuations of both linear and angular velocities.
Patch replaces manually computed dt with the Unity's Time.deltaTime value which provides stable values.
Video Demonstration

Some other points:

An option to toggle between degrees/sec and radians/sec as the unit for angular velocity. AFAIK almost all packages in Ros adhere to radians and radians/sec.
An option to publish the transform's velocity in local frame.

Hi @abhayraw1 ,
please keep pull requests to different issues separated in the future. This simplifies the discussion, revision and merging.
Concerning your dt modification:

I agree that calculating dt based on real-time clock is not correct, as velocity is based on Unity's simulation time. This is the reason for the jitter that you have shown in the video. The jitter should decrease when the simulation runs more fluently, e.g. on a faster PC or when running the build.
We are executing UpdateMessage() every FixedUpdate(), so Time.deltaTimewhich is the time between two frames is not correct either. Please replace dt by Time.fixedDeltaTime .

Concerning the additional options:

Generally I am against options. They complicate things for users and developers. As we are producing a geometry.Twist message here, so it is clear that we must follow the specifications of that message type.
ROS uses SI units and geometry.Twist is in rad/s, not deg/s. so we must add this Mathf.Deg2Rad factor by default.
geometry.Twist is defined in local frame only (see e.g. here. So we should also skip the toggle button but produce twist message in the local frame only. This holds for position and rotation.
Instead of applying InverseTransformDirection(), please calculate the local velocities directly by

Vector3 linearVelocity = (PublishedTransform.localPosition - previousPosition) / dt;
Vector3 angularVelocity = (PublishedTransform.localRotation.eulerAngles - previousRotation.eulerAngles) / dt;

In this concern, please also correct initialization of previousPosition and previousRotation:
 previousPosition = PublishedTransform.localPosition;
 previousRotation = PublishedTransform.localRotation;

Best,
Martin

Referring to Unity's documentation for Time.fixedDeltaTime:

For reading the delta time it is recommended to use Time.deltaTime instead because it automatically returns the right delta time if you are inside a FixedUpdate function or Update function.

I think using Transform.localPosition won't work. As per the documentation:

Position of the transform relative to the parent transform. If the transform has no parent, it is the same as Transform.position.

Hi @abhayraw1 ,

Please replace Time.deltaTime by  Time.fixedDeltaTime. Also if in most cases Unity returns the correct value for Time.deltaTime, it is coding style in ROS# to use Time.fixedDeltaTime whenever we refer to the time between two physics steps.
Please replace PublishedTransform.position by PublishedTransform.localPosition and remove the InverseTransformDirection()transformation as descirbed in more detail above. Your current version:

Vector3 linearVelocity = PublishedTransform.InverseTransformDirection(
                (PublishedTransform.position - previousPosition) / Time.deltaTime

calculates the direction between two consecutive positions in world space and converts that into local space.
Twist messages encode a linear and angular velocity relative to some reference frame, i.e. in the local coordinate system:
Vector3 linearVelocity = (PublishedTransform.localPosition - previousLocalPosition) / Time.fixedDeltaTime

Please perform the same adjustments for angular velocity, which should also be returned in local space, not in world space.

We can accept the pull request when you realized these modifications.
Best,
Martin

The contents of this pull request including the comments above have been realized pull request #349.


support GCC 8.3 or document that 6 should be used

tried and bdb build fails. GCC 6 works
error output is huge, but this is repeated all over the place:
../dist/../dbinc/atomic.h:179:19: error: definition of ‘int __atomic_compare_exchange(db_atomic_t*, atomic_value_t, atomic_value_t)’ ambiguates built-in declaration ‘bool __atomic_compare_exchange(long unsigned int, volatile void*, void*, void*, int, int)’
static inline int __atomic_compare_exchange(
                  ^~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:2020: cxx_dbc.o] Error 1

Thanks @shyba .  The reported issue is specifically for building dependencies, via reproducible build.  A known work-around is to run the script as: CC=gcc-6 CXX=g++-6 ./reproducible_build.sh

Saving link: https://patchwork.ozlabs.org/patch/932375/

Yesterday I realized that the "depends" build had this fixed ages ago (in upstream bitcoin which we recently merged). @shyba tested it also and saw no problems.

in master now. (use build.sh or packaging/build_*)


transformCondition is not defined

Describe the bug
Trying to build an old React app in Solid, but trying to build anything that imports from @solidjs/router gives the error
<EMAIL_ADDRESS>transformCondition is not defined".
Even with a simple App component like this:
render( 	() => ( 		<Router>

 		</Router> 	), 	document.getElementById("root") );
I've used the old version, solid-app-router, at version 0.3.3, but upgrading to 0.5.0 gives me this error.
I don't understand what's gone wrong, since this error doesn't appear in forums or in the docs.
Your Example Website or App
https://github.com/Dynamic-Array/fathom-solid-analyser
Steps to Reproduce the Bug or Issue

Create a new Solid app
Install the latest @solidjs/router
Try & run a vite dev environment with any @solidjs/router components imported

Expected behavior
I should load the single-page component. There should be no navigation, and all URLs will load a simple div.
Screenshots or Videos
No response
Platform

OS: Windows
Browser: Chrome
Version: 0.5.0

Additional context
No response

I just did the following and it seemed to work okay....
npx degit solidjs/templates/ts-router my-app

Update package.json to current latest versions...
  "devDependencies": {
    "vite": "^3.1.8",
    "vite-plugin-solid": "^2.3.10",
    "vite-plugin-windicss": "^1.8.8"
  },
  "dependencies": {
    "@solidjs/router": "^0.5.0",
    "solid-js": "^1.6.0"
  }

then
pnpm i
pnpm run dev

Are you still seeing errors in your project?

I fixed this, this morning. Others have confirmed. Fresh install or blowing out lock files should get the latest for the babel plugin.


when I train RCAN,something wrong:RuntimeError: Expected 4-dimensional input for 4-dimensional weight 3 3 1, but got 3-dimensional input of size [1, 184, 270] instead

python main.py --template RCAN --save RCAN_BIX2_G10R20P48 --scale 2 --reset --save_results --patch_size 96
and then
Traceback (most recent call last):
File "main.py", line 33, in
main()
File "main.py", line 28, in main
t.test()
File "/home/zhj/EDSR-1.1.0/src/trainer.py", line 89, in test
sr = self.model(lr, idx_scale)
File "/home/zhj/anaconda3/envs/pytorch1.1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/zhj/EDSR-1.1.0/src/model/init.py", line 57, in forward
return forward_function(x)
File "/home/zhj/EDSR-1.1.0/src/model/init.py", line 135, in forward_chop
y = self.forward_chop(*p, shave=shave, min_size=min_size)
File "/home/zhj/EDSR-1.1.0/src/model/init.py", line 126, in forward_chop
y = P.data_parallel(self.model, *x, range(n_GPUs))
File "/home/zhj/anaconda3/envs/pytorch1.1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 204, in data_parallel
return module(*inputs[0], **module_kwargs[0])
File "/home/zhj/anaconda3/envs/pytorch1.1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/zhj/EDSR-1.1.0/src/model/rcan.py", line 107, in forward
x = self.sub_mean(x)
File "/home/zhj/anaconda3/envs/pytorch1.1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/zhj/anaconda3/envs/pytorch1.1/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 3 3 1, but got 3-dimensional input of size [1, 184, 270] instead

Same error on my machine

+1

how to solve it！！！help me

Same error on my machine

You could refer to #184 .

in my opinion，the error mentioned above is caused by the mismatch of the number of dimension.  Specifically，the required dimension is 4 whilst 3 was given，thus you should debug all the lines remain in the error message step by step in order to monitor the change of variables‘ dimension during the training process.

I was successful with the fix from #184 referenced above. In particular, model/__init__.py line 133 onwards became:
        else:
            for p in zip(*x_chops):
                p = [p_.unsqueeze(0) for p_ in p]
                y = self.forward_chop(*p, shave=shave, min_size=min_size)

I really don't have a good explanation but it seems to work.

I guess the gist of it is that x_chops contains a tensor for each input (args is List[Tensor(B x C x H x W)]) to forward_chop. That tensor is cut up into quarters and catted along the batch dimension. So now you have something along the lines of x_chops is List[Tensor(B*4 x C x H/4 x W/4)]. Then the "clever" line
for p in zip(*x_chops):

is equivalent to something like
for i in range(B*4):
    p = [x_ch[i, ...] for x_ch in x_chops]

which, as you can see when it's not so "clever", is going to drop the first dimension on each element in x_chops. Which is a problem because p is the recursive input to forward_chop. :(

I solved this problem by omitting "--chop" in option.(python==36, pytorch==1.1)


Add a script util to check which files are currently modified by open PullRequests

This is a proof of concept showing a GraphQL script that can check all the files touched by PR with a given label.
In the case of localization work, this kind of scripts can be useful to verify that a file is not currently being edited/reviewed.
I want to gather feedback about the scope that this kind of scripts should have. I think they can really help the localization team working more efficiently and track what work is currently being done.
What do you think?

It would be good to add a little documentation in the header to explain the scope of this script, and how to use it (especially the configuration of the GITHUB_TOKEN).

I would first like to gather feedback about scripts and what it takes to merge them. I would also like to have feedback about similar problems we could fix with GraphQL and alike scripts.

/lgtm

/lgtm

/lgtm

/approve

/assign @jimangel

@zacharysarah Could you merge that?

Always fun to encounter GraphQL queries in the wild.
/lgtm
/approve


Filter camera by USB port id

Start with the why:
For multiple cameras systems in production, it becomes quickly complex to manage a long list of camera serial numbers (current filter with i_mx_id). An alternative could be to filter by USB port id as it is done classically for cameras in ROS, see for instance usb_port_id in https://github.com/IntelRealSense/realsense-ros
Move to the what:
Add a parameter usb_port_id to depthai_ros_driver::Camera for the driver to open only the camera plugged to the corresponding port
Move to the how:
@mattmacleod

it would mean quite a few changes to depthai-core as well to expose USB bus information in the Device class

Thanks for that request, this shouldn't be a big issue to add it :+1:

Added in https://github.com/luxonis/depthai-ros/pull/229, available on v2.6.3


enableRowHeaderSelection hides filters

Setting:
enableRowHeaderSelection : true
with
enableFiltering: true
causes header height to become fixed and not display filters.
Using css to expand the header will display the filters:
    .ui-grid-header-cell {
        height: 57px;
        max-height: 57px;
    }

I'm assuming that this is not intentional.

I am using ui-grid - v3.0.0-RC.18-7774d30 and the behavior is seen in IE 11 and Google Chrome.

I'm not seeing this on http://ui-grid.info/docs/#/tutorial/401_AllFeatures, is there some other setting that you're using as well?  I know there is an outstanding request to make the enabling of filtering dynamic (#2067) - is it possible that you're initialising the grid without filtering then turning it on afterwards?
Otherwise, is it possible to create a plunker?

Yea, there was a little more to it than I thought. When creating the plunker I found that it exhibited the aforementioned behavior only when loading data and column definitions after the grid was initialized. Here is the plunker ( http://plnkr.co/edit/JJFFXV5FAOILik6b4oR5?p=preview ) , you have to press the load data button and you will see that the column filters are not displayed. Adding the above CSS will make the column filters visible.


Fix preserve color space for Color type

Summary
Fixes: #97.
By default, ColorBridge will convert Color into UIColor in non-macOS platform.
UIColor does not preserve the color space, but Color should store the color space.
Solution
Trying to preserve color space whenever Color.cgColor exists.
If SwiftUi.Color has cgColor property, Defaults will serialize and store its colorspace and components, then deserialize it with these properties.
If not, Defaults will use NSSecureCodingBridge to do the serialization and deserialization.
Thanks
Thanks for your code review 😄 !

After this PR is merged, do you plan any more changes, or should I do a release?

After this PR is merged, do you plan any more changes, or should I do a release?

I have no immediate plan for any more changes.
I think we should do a release after this PR gets merged.
Thanks!


Parsing "pl" breaks consecutive loading of same document

Load Document, load document again:
true     .
false.
?- es/errors/errors_loader.lgt').
exception: error(existence_error(procedure,es/errors/errors_loader),(/)/2)
false.
?-

Copy and paste issue? That query doesn't make sense.

It's loading from 'examples/errors'... I think the parsing of the "pl" is the problem.


SocketRef inside AckResponses

Is your feature request related to a problem? Please describe.
I recently wrote a broadcast implementation with acknowledgement from each client, and got frustrated to see that I had no way to know which response came from which socket. This is problematic because the frontend needs to know the client id from the response in order to work.
socket.on("broadcast", |socket: SocketRef, ack: AckSender, Bin(bin): Bin| async move {
    let client_id = socket.extensions.get::<ClientId>().unwrap().clone();

    let responses = socket.broadcast()
        .timeout(Duration::from_millis(4000))
        .bin(bin)
        .emit_with_ack::<Value>("sqw:data", ClientData { id: client_id }).unwrap()
        .filter(|ack| futures::future::ready(ack.is_ok()))
        .map(|ack| {
            let ack = ack.unwrap();
            return (ack.data, ack.binary);
        })
        .collect::<Vec<_>>().await

    // ???
});

Describe the solution you'd like
I have looked at the internal codebase and found out that the socket could be placed inside the Acknowledgement here. The problem with this solution is however that I had problems with getting a SocketRef inside here, as the function is written in Socket and has nothing to do with SocketRef. I would like the solution to look something like this:
Ok(AckResponseWithClient {
    client: /* self.clone() */, // not possible to get the SocketRef here. Or is it?
    data: serde_json::from_value(v.data)?,
    binary: v.binary,
})

where AckResponseWithClient is just a copy of the regular AckResponse, just with an extra field:
#[derive(Debug)]
pub struct AckResponseWithClient<T> {
    /// The subject of the acknowledgement
    pub client: SocketRef,
    /// The data returned by the client
    pub data: T,
    /// Optional binary payloads
    ///
    /// If there is no binary payload, the `Vec` will be empty
    pub binary: Vec<Vec<u8>>,
}

Hey, I will add the SocketRef inside the AckResponse in the PR #201.

There will be only the socket id provided along with the ack response and not the SocketRef. Because if we put the SocketRef inside the AckStream, if we disconnect it while waiting for the acknowledgement, a reference of the socket would still exist in the AckStream and therefore an AckError(SocketError::Closed) would not be returned as there is still an existing reference.
The best solution I found is therefore to only keep the socket id. You can get any socket in O(1) time with io.get_socket(Sid) though.

Good thinking!
@FabianHummel you might be interested in this discussion


Fix IllegalArgumentException when savedSiteAdded

Task/Issue URL: https://app.asana.com/0/488551667048375/1208580918293540/f
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

#5155  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @joshliebe and the rest of your teammates on  Graphite


[Chore] Consolidate assets folders including js, css, images

Current State of the Meshery.io site
These two folders and their content:

https://github.com/meshery/meshery.io/blob/master/js
https://github.com/meshery/meshery.io/blob/master/images

Desired State of the Meshery.io site
Can be combined under /assets.

Contributor Guides and Handbook
The meshery.io website uses Jekyll and GitHub Pages. Site content is found under the master branch.

📚 See the Contributing to Meshery.io Website section of the readme.md and other contributing instructions, too.
🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

Hey @leecalcote , Please Assign it to me.

Sounds good, @eagerashwani

Is this issue this open and doable?
Cause I can see wontfix issue label.

Help wanted.

I would like to work on this issue.

I can also work if @RahulGopathi is not working.

@leecalcote Can I work on this issue?

@BabyElias yes, that’s be great.

Should I create separate folders for both the js and images files under assets?

Done consolidating all the files and updating their locations throughout the repo

@leecalcote , merging on the main branch is causing conflicts for these files as people are still adding images to the images folder that has been removed by this PR.

@leecalcote Can you please review the PR whenever possible. Thanks in advance!


FIX: LiveDispatcher should work even if we do not see a start document

Description

Finally got around to playing with the LiveDispatcher and noticed this slightly annoying gotcha. If you subscribe a LiveDispatcher after the original start document, the secondary stream outputs an event document before ever outputting a start document. This is certainly confusing to subscribers and we also never perform any of the necessary start-up done in LiveDispatcher.start
This quick fix just checks right before we emit our first event that we emitted a start document, if not we force an self.start() with an empty dictionary.

Codecov Report

Merging #1022 into master will decrease coverage by 0.01%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master    #1022      +/-   ##
==========================================
- Coverage   90.47%   90.46%   -0.02%
==========================================
  Files          50       50
  Lines        8085     8087       +2
==========================================
+ Hits         7315     7316       +1
- Misses        770      771       +1

Impacted Files
Coverage Δ

bluesky/callbacks/stream.py
94.73% <50%> (-1.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c552859...2bc26c4. Read the comment docs.

I would prefer we drop documents on the floor until we see a start document and then forward the rest on (and the same with events / descriptors).

That approach means we can't subscribe any LiveDispatcher subclasses during any open run. It seems like an arbitrary restriction.

Correct, we don't support 'catch up' any where else (and went through this discussion with the 0mq proxy previously).
For callbacks that just need to see a start document, emitting an empty dict is fine, but for callbacks that make use of the information is the start document this approach will confusingly fail down stream.  When I think of the 'stream integrity' of a document stream the content of the documents is at least as important as the order of them!
You can go the other route and cache documents as they go by, but then you have to decide when to drop them (which can get messy) and exposes you to resource issues etc.   At that point we are re-implementing a message bus in python!

Being able to "catch up" is an obviously useful feature, and one that we should establish a recommended path for. I agree with @tacaswell  that providing this robustly requires more infrastructure than we can build here -- a real message bus. Sending an empty dict to downstream consumers breaks too many assumptions that they are otherwise allowed to make (e.g. all start documents have an accurate 'time' and a 'uid'). This change would force all callbacks to account for a new special case.

You can go the other route and cache documents as they go by, but then you have to decide when to drop them (which can get messy) and exposes you to resource issues etc. At that point we are re-implementing a message bus in python!

@ZLLentz and @teddyrendahl -- If you feel strongly that you need a way to subscribe mid-run, I think a simple caching implementation might be worth having, as long as it were presented with a huge asterisk flagging resource issues and recommending that users graduate to a proper message bus when they do anything at scale.

P.S. Speaking of "establishing a recommended path" for robust message buses integrated with bluesky, see https://github.com/NSLS-II/bluesky/pull/1023.

OK, yes I was misunderstanding. Thanks for clarifying.
I'm still concerned that this could create confusing bugs for callbacks. A callback might reasonably expect to see a 'plan_name' (an item guaranteed by the RunEngine to be present, but not required by the spec -- an intentional distinction). Recently, it might even be safe to assume 'hints' if the user is only using standard plans and hints-compliant plans. I would personally rather wait for the next full run that have to worry about special-casing a stub start document in my consumers.
Moreover, if we missed the 'start' document, aren't we likely to have missed one or more 'descriptor' documents as well? What happens then?
It's still possible we're talking past each other a little here. Should we set up a quick call to hash this out a bit more?

No, I think we are now hitting the same notes. I wasn't sure how concerned I should be about it so I am glad you guys are inquisitive! In my head there were two standard use cases:
Basic -> Created in global context, guaranteed to see all documents.
ld = LiveDispatcher(...)
ld.subscribe(cb)
RE.subscribe(ld)
RE(count(...))

Advanced -> Created in a plan where all the oddities are self-contained. This LiveDispatcher only lives within the context of this plan so you do not have to worry about shenanigans with arbitrary callbacks getting attached. This plan does not work as written unless ld.start({}) is called manually
def stub_move from_filtered_fit():
       ld = LiveDispatcher(...)  # Use LiveDispatcher to filter data
       lf = LiveFit(...)   # Fit of cleaned data
       ld.subscribe(lf)
       RE.subscribe(ld)
       for _ in range(...):
            yield from trigger_and_read(...)  # Collect data
       yield from set(motor, lf.model.intercept)   # Interactive decision based on fit

The once case I was really concerned about is lets say you setup one of these LiveDispatchers with a DataBroker, and also store the raw data with a separate DataBroker. In this case if you don't receive that start document you run the risk of storing a bunch of processed data with no way to link it back to raw data runs. This seems very bad. I think we need to think of a better way around this. I can also make the "Advanced" method just call LiveDispatcher.start() manually and circumvent this if I am know I am an expert and not breaking anything. I am going to close this PR for now and maybe we can think about a better implementation later. I do have two thoughts I want to pontificate about for a second:

When you guys have a chance I think it would be worth checking out the original inspiration for this PR https://github.com/pcdshub/nabs/blob/master/nabs/plan_stubs.py. The idea is basically a plan_stub that will take detectors and return the average of them. Super powerful for adaptive plans and only eight lines of real Python code!

@danielballan I'm a little concerned about having to have callbacks that depend on md dictionaries that are formatted with non EventModel keys. I would argue that any callback that depends on fields like "hints" with no contingency for if that is missing is probably in the wrong. I understand that these are included in most default plans but I hope we design at least the built-in bluesky callbacks to only use information guaranteed in the EventModel.

@danielballan I'm a little concerned about having to have callbacks that depend on md dictionaries that are formatted with non EventModel keys. I would argue that any callback that depends on fields like "hints" with no contingency for if that is missing is probably in the wrong. I understand that these are included in most default plans but I hope we design at least the built-in bluesky callbacks to only use information guaranteed in the EventModel.

Part of the idea is to start paving the cow-paths.  We are going to have to (soonish?) start having extensions to the EventModel to specify this sort of thing.  Likely based on what Nexus has already done.  We also need a way to allow variation between BL / domain / facility.


MBAR estimator automatically choosing the usable method

As is stated in #419, BFGS and adaptive each have cases that they cannot resolve, which could be resolved by the other estimator.
I'm thinking of adding a more robust solver in the downstream package alchemlyb.
The logic might be
try:
    MBAR('adaptive')
except:
    MBAR('BFGS')

I know that 4.0 will be out and things will be changed. So I wonder in 4.0, will there be a method that will fall back to BFGS when adaptive doesn't work or the user will have to deal with this by themselves?

@mrshirts : It would be extremely useful to fall back to the most robust (iterative) solver if the fancier solvers do not work, but sometimes a user will want to only use the specified solver. Perhaps an optional argument fallback=True by default, with the user able to specify fallback=False if they only want to use a particular solver?

It would be extremely useful to fall back to the most robust (iterative) solver if the fancier solvers do not work

It's more complicated than this , because in some of the cases, it fails to converge with the iterative solver, but minimizes fine with the other solver.  Improved robustness does need to be implemented though, and I'm trying to find time to get this finished before school starts in a month (along with GPU optimizations).

The suggestion is to fall back to a different solver by default if the
selected solver fails. We can try all the solvers, prioritizing the fastest
or requested solver. If none of the solvers succeeds, we can give up.

Attempting to solve with #446


Citation

Hello,
As a fan of promptfoo, I'd appreciate if you could add a CITATION.cff file to the repository. This would help users like me properly credit your work.For guidance, see: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-citation-files
Thank you for considering! 😁

Thanks for the suggestion @Sbkubuii! This just went into main. Please let us know if there's anything else we can do to accelerate your research!


Clear chat messages

Hi! is there a way to clear all messages in the chat history?

Got it working like this!
var history = document.getElementById('messages')
if (history !== null){
history.innerHTML = '';
}

import {dropMessages} from react-chat-widget

Thanks a lot!


a Feature like fence

你们可以加一个电脑桌面图标管理软件吗，就好像 Stardock公司 的 fence 那样。(if you can't see my word, use google translate)
如图：

Duplicate of https://github.com/microsoft/PowerToys/issues/1392

this fence is not that fence; they aren't the same;

ok


[Backport] [2.x] Upgrade forbiddenapis to 3.5.1 (#6840)

Backport https://github.com/opensearch-project/OpenSearch/pull/6840 to 2.x

Gradle Check (Jenkins) Run Completed with:
* **RESULT:** FAILURE x

* **URL:** https://build.ci.opensearch.org/job/gradle-check/13145/

* **CommitID:** [5622e22](https://github.com/opensearch-project/OpenSearch/commit/5622e2243d72668f7a09a1a61da4e6a2bb311e0e)
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?

Should be fixed by https://github.com/opensearch-project/OpenSearch/pull/6918

Codecov Report

Merging #6898 (5622e22) into 2.x (ac79a63) will decrease coverage by 0.03%.
The diff coverage is 31.93%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##                2.x    #6898      +/-   ##
============================================
- Coverage     70.41%   70.38%   -0.03%
+ Complexity    59480    59476       -4
============================================
  Files          4810     4811       +1
  Lines        285453   285551      +98
  Branches      41503    41511       +8
============================================
- Hits         200993   200985       -8
- Misses        67704    67789      +85
- Partials      16756    16777      +21

Impacted Files
Coverage Δ

...ark/store/remote/filecache/FileCacheBenchmark.java
0.00% <0.00%> (ø)

...ensearch/gradle/precommit/ThirdPartyAuditTask.java
0.00% <0.00%> (ø)

...pensearch/ingest/useragent/UserAgentProcessor.java
63.41% <ø> (ø)

.../index/reindex/AbstractBaseReindexRestHandler.java
6.66% <ø> (+0.69%)
:arrow_up:

...on/admin/indices/forcemerge/ForceMergeRequest.java
48.71% <0.00%> (-4.06%)
:arrow_down:

...ensearch/index/mapper/ParametrizedFieldMapper.java
98.59% <ø> (ø)

.../java/org/opensearch/index/mapper/TypeParsers.java
81.39% <ø> (ø)

...main/java/org/opensearch/rest/BaseRestHandler.java
69.33% <0.00%> (-7.14%)
:arrow_down:

...rest/action/admin/indices/RestOpenIndexAction.java
34.78% <0.00%> (-22.37%)
:arrow_down:

...va/org/opensearch/index/mapper/MapperTestCase.java
95.33% <ø> (ø)

... and 11 more

... and 463 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Migration distances not in KM with a lat/long dataset

When a dataset is in lat/long, the migration distances calculated in app 4 (i.e., the CalcSeqDistances.R function) are not calculated in KM, they are in decimal degrees. Need to add a line in CalcSeqDistances.R function to note if the sequences are in lat/long, and if so, transform into UTM before calculating the distances.

Hey @jmerkle1 - I am looking into this now. When sequences are written as RDS files, they're always converted to 4326 firstbefore being saved. I think we could solve this by converting to the UTM before writing files.
I am not sure why I chose to write to 4326 before saving. Do you see any downstream issues with the RDS sequences if they're in UTM instead of decimal degrees?
https://github.com/jmerkle1/Migration-Mapper/blob/dc32cf3618c079057d067783cf9c8bbf3592dcb5/app3_sequenceWriting/scripts/app3_calcSequences.R#L68

It probably makes sense to convert the RDS sequences to UTM. The sequence RDS files are used to calculate distances but are also fed into the cluster when calculating UDs. I just want to make sure it won't cause issues when we're calculating UDs

Sequences are now calculated in UTM. You can pull down a new copy and see. I can now see reasonable values in the ...migration_distance_info.csv files.


refactor: extract operatorship management into sharable interface

AXE-6841

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.00%. Comparing base (df2d7d8) to head (0ada12a).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
+ Coverage   88.71%   89.00%   +0.29%
==========================================
  Files          44       45       +1
  Lines        2880     2956      +76
==========================================
+ Hits         2555     2631      +76
  Misses        325      325

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add attribution link to pin

How would you add the link from which a pin originated as seen here? https://www.pinterest.com/pin/617063586420559182/

You could try checking the link or original_link properties on the Pin?


Images are not being uploaded directly on GCS

There is an issue when I upload the category image from Admin and images are not getting uploaded on GCS. Instead, it stores on file system.

It's not direct sync you have manually sync to cloud.

Is manual sync is possible through cmdline ?

yes we use command line sync also
php bin/magento gcs:storage:sync


cannot retrieve the id of the products in the cart

when i try to get the id of a product in the cart.. its not possible. the ids of products aren't store in the cart. only names, quantity, price and total amount is stored

my bad.


Provision fails when Dynamic ThingsBoard usage

When setting  Dynamic ThingsBoard usage:
#define THINGSBOARD_ENABLE_DYNAMIC 1
provision fails with the following message:
[TB] Unable to de-serialize received json data with error (DeserializationError::NoMemory)
Tested with<EMAIL_ADDRESS>and<EMAIL_ADDRESS>thingsboard version 12.2
Thank you in advance, have a nice day.

Can you add logs to this section of the code https://github.com/thingsboard/thingsboard-client-sdk/blob/master/src/ThingsBoard.h#L1979.
Simply add
Serial.println(reinterpret_cast<char*>(payload));
Serial.printf("Allocated memory: %d should be enought to hold payload of size: %d", dataStructureMemoryUsage, length);

Here is raw data:
Sending provisioning request
{"credentialsValue":"jKexqfuXSlqYs7ow7L8z","credentialsType":"ACCESS_TOKEN","status":"SUCCESS"}zi8t3pnqxdlipqz"}t��q�E
(ou��,���
x�9dD��eSЍ6w� ��S�'��
��i����,H�E9�a�       ����j|ol$���@(v
��w�L��T��Qщ}nwS/ Rn�
Allocated memory: 48 should be enought to hold payload of size: 95[TB] Unable to de-serialize received json data with error (DeserializationError::NoMemory)

Weird 48 should be exactly enough to hold the received payload in the allocated JsonDocument. I even verified it with the Arduino JSON Assistant.
For now can you simply change this line https://github.com/thingsboard/thingsboard-client-sdk/blob/master/src/ThingsBoard.h#L1971. To be the received length arugment instead and retry it again.

Hello,
used received length argument with no fix of the issue.
I have also tried to add a "0" at the end the of received payload, with no efects:
memset(payload + length, 0, 1); // Set bytes from length to end of buffer to 0
Also I have plenty of free heap, I have an ESP32 with the provision example running only.
For the moment I am not using THINGSBOARD_ENABLE_DYNAMIC and I am increasing Default_Fields_Amt since I am sending telemetries all together in a JSON.
Let me know if I can help me something.
Thank you for your work, have a nice day.

If it is possible can you add this log call as well after the JsonDocument has been initatied call Serial.printf("Allocated capacity: %d", jsonBuffer.capacity());


Extend list of known error numbers

@yoavbls I updated the list to a massive 320 fixed errors. 💪

Thank you, Benny! Great work on typescript.tv!
I will release a version with the new errors soon


Revise basic_reporting in mediumroast_js

Introduction
There are several improvements that need to be built into reporting so that it can accommodate the new data from interactions and eventually company similarity reporting.
Bug fixes

[x] Fix when any company link attribute is set to 'Unknown'. Today it won't be eliminated from the list of links it should be not present to cause confusion.
[x] Remove the maps warning as GitHub claims this is now fixed.

Changes

[x] Modify the maps warning as GitHub says this is now fixed pointing to the message from GitHub about it being fixed.
[x] Look at changing how the SIC is displayed by making it a badge and moving it to the top and removing the section on links.
[x] Move the Weblinks section higher up in the report.

Improvements

[x] Reimplement interactions reporting. Key items are collapsible, tags, etc.

Completed


Package request: edgedriver

Project description
Microsoft Edge WebDriver, similar to chromedriver or geckodriver.
Metadata

homepage URL: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
source URL: N/A
license: proprietary
platforms: windows, linux, darwin

Duplicate of #192050


k-Means is very slow, scales badly with k

This variant of k-means (which is overly complicated) scales very badly.
Some observations:

at the beginning of each iteration, the mean is recomputed (in if (!fixed_centers)). Afterwards, it is incrementally updated, but that data is never used. All use of mus in the second loop can be deleted.
in each iteration, each vector will be retrieved three times, and freed three times, where one should be enough ( get_feature_vector).

But that still does not explain my runtime measurements (using 3.2.0 - I only now realized that the version preinstalled with Linux distributions is much older - albeit there does not appear to be any relevant difference):
k=10 5.91 seconds, k=100 137.009 seconds, k=1000 2409.55 seconds.
Where usually, the performance would increase by ~7.5 when increasing k by 10 (because of fewer iterations I guess), it increases by ~20 in Shogun with k, making it the slowest k-means I benchmarked except for flexclust (pure R), sklearn 0.14 (fixed since, 100x faster now), and scipy.cluster.vq. In these, the interpreter is probably to blame. For the record, the best K-means-variant I benchmarked on this data is 0.80, 3.76, 7.35; the best Lloyd implementation is 2.15, 42.16, 202.19... so Shogun is 12x slower than necessary, and better algorithms are over 300x faster.
Above observation (duplicate computation of centers) would explain a factor of 2 or 3 maybe. That would be enough to make k=10 competitive, but there is a superlinear cost with increasing k that I cannot discover in Shogun source code.
Note that #1263 suggests to drop legacy code like this completely.

Update: I found another issue with this k-means. It is also initialized very badly. Every point is randomly assigned to a cluster, which will cause cluster centers to be very close to each other. The more common random initialization is to simply choose k random instances, and use these as centers.
The k-means++ initialization does not appear to be available using the command line interface.

Ouch. It also chooses n random instances during each iteration, instead of using each one time? That only means it randomly ignores about 1/3rd of instances in each iteration:
const int32_t Pat=CMath::random(0, XSize-1);

@kno10 yeah indeed when i started some benchmarking of shogun, i've started with k-means and currently i do agree that's it's in a rather horrible state. :(

For k-means, I've made a pull request #2988 with some of these changes, but I have not tried to compile it, nor tested or benchmarked it. But this may save you some work.
I don't know if anybody in Shogun has much interest in k-means, so #1263 may have a point.
Hierarchical clustering also benchmarked rather slow. It runs out of memory at 50000 and took 32 seconds for 20000 instances. The better implementations all do 20000 in 3 seconds and scale to 500000 instances without problems.

Hi @kno10, thanks a lot for your analysis and the comments.
Regarding the first observation in your first comment, why do you say that mus is never used in the second loop (the for loop that starts after the block if (!fixed_centers)  { ... } ends)?

mus is only modified, but never read. All distances are computed to the copy in rhs_mus. The algorithm terminates when the centers did not change anymore (then mus has not changed either in the second loop). At the beginning of the next iteration, mus is set to zero again in line 53.
Thus, the only case when these updated values are used is if max_iter is hit. Then the updated values from that last iteration are returned - but that is not any better than returning the previous centers to which the points were last assigned (neither is a consistent solution, because it has not converged).

I think (perhaps I am getting it wrong) that mus is both read and written. Note that in lines 110 and 124 the operator used is -=. It is not probably not the most readable way of doing this :-)
Does that make sense to you?

Its modifed, but not used afterwards. There are no "side effects" beyond updating mus, are there?

It does, because fixed_centers can be true. However, I do agree with you that all the work done to update mus is a waste in case fixed_centers is false -- which may be the most common use case.
I think we should break up this method into two methods, and use either of them depending on the value of fixed_centers.
It is great that you pointed this out! Thanks a lot.

fixed_centers makes no sense, because if the centers are fixed, iterating is nonsense - it's not k-means anymore. fixed_centers should be handled by the NN classifier instead, every point is assigned the nearest center as label.

@kno10, can you share the code you used for the benchmark results you mentioned above? A gist or similar would work fine.

Thanks a lot @kno10 for investigating this. I totally am for dropping such disasters, as they are doing more bad than good.
In fact, I am thinking about a GSoC project that is just about going through our "simple" ML algorithms, benchmarking them a bit, identifying problems (as you did here), and then either dropping or re-writing.

@kno10 I just had a look. If you are interested, you could attempt a re-write that solves the above problems. This might be a nice entrance task for the mentioned GSoC project as well

Sorry, my focus is on ELKI. I used Shogun solely for comparison.

No worries. Thanks!
I used your report in a GSoC project proposal here:
https://github.com/shogun-toolbox/shogun/wiki/GSoC_2016_project_fundamental_usual_suspects
Hopefully we resolve a few of those during the summer!

hi guys, ran some timers on handy clustering datasets, results (seconds) :
The shogun(new) is after updating some of the changes discussed above

remove random sampling of instances
remove unnecessary updating of centers (which would mean we have to handle fixed centers separately )

instances = ~5000

k
10
100
1000

scikit
0.09
0.58
3.565

mlpack
0.08
0.493
3.00

shogun
0.18
1.61
8.19

shogun (new)
0.09
0.99
4.96

instances = ~20,000

k
10
100

scikit
2.65
32.53

mlpack
1.14
127.68

shogun
3.07
428.86

shogun (new)
1.66
276.89

What do you guys think? I had already attempted a broader restructuring of KMeans class in #2648 #2815 but we were stuck because apparently there were not enough unit tests.

Good to see you back here @Saurabh7
Thanks a lot for this example. These results are quite extreme. Can you post the scripts that generated them somewhere? (Your own public repo for example). I would like to have a look at them.
Do the methods produce the same results?
We now have to find out what makes the other implementations so fast (and copy the design). Then we can attempt to fix things.
The Shogun (new) version that is based on the discussions above seems already much faster, but still way beyond the others -- so maybe rather than re-structuring we can have a look at the other libs and attempt a re-implementation from scratch....after all KMeans is a dead simple algorithm.
One reason for speed differences might be multicore and vectorised linear algebra, which are relatively straight forward to put in our too.
When re-implementing/re-structuring, we should make use of linalg. @lambday can probably help here as well as he ran benchmarks before.

Thanks @karlnapf , I have posted the scripts at https://github.com/Saurabh7/shogun-benchmarks/tree/master/kmeans . You can reproduce quickly with the python scripts i guess.
I haven't looked at the quality of the results yet.
As you said lot of things could be changed: using new SGVector, SGMatrix operators , linalg operations, etc. after which we can compare the results again.
One problem I find with changing the code by re-implementing/structuring  is that  parts of the current implementation (some of which are oddly different from std implementations in other libraries) are not documented/tested. So maybe we would have to address that first too.

Thanks! Some comments:
Sanity check: you build the shogun library with compiler optimizations enabled right? That is DBUILD_TYPE=Release ? and the script as well?
I agree with all the other points. Using std anywhere is not really necessary.
I agree with the documenting/testing. However, if the current code design is slow and needs to be replaced anyways, there is not really a need for this. Maybe as a first step, you could have a look into the sklearn implementation to check what they are doing from a high level perspective (same data-structures and operations?). This should give an answer to the question why it is so fast compared to other libs.

@karlnapf Yes I did build with DBUILD_TYPE=Release.
Updating with a few more changes:

Random initialization : Now chooses k random instances as cluster centers rather than randomly assigning clusters. Then assign nearest points to cluster centers.

Used linalg to scale the mus matrix. (Used a hack here as scaling each row of matrix by different scalar is not possible)

instances = ~20,000

k
10
100

scikit
2.65
32.53

shogun
3.07
428.86

shogun_1
2.02
276.89

shogun_2
5.00
49.51

The increase for k=10 in the latest shogun_2 must be because of the new constant cost of computing distances during initialization.
Now some comparisons with scikit I came across:

Computing distances:
For euclidean distances their version is much faster:
dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y))  . If only one argument changes (as in case of kmeans) they use precomputed values of dot(x, x) using x_squared_norms.

But in shogun we use simple loops for (int32_t i=0; i<alen; i++)          result+=CMath::sq(avec[i] - bvec[i]);
To mimic the performance of computing distances in kmeans with varying k i wrote  some quick scripts https://github.com/Saurabh7/shogun-benchmarks/tree/master/distance, results(seconds) :
instances = ~ 20,000

k
10
100
1000

scikit
1.94
20.49
226.03

shogun
8.35
77.47
860

Convergence:
They use a concept of inertia which is basically within-cluster sum of squares (WCSS) . When the change in it is less that some tolerance tol convergence is declared.
In shogun change in each and every point's cluster is checked till there is no change.
Maybe this could lead to unnecessary iterations even when the WCSS is not changing a lot.

Very good investigation @Saurabh7
I think we should focus on making our distance computation faster then since that seems to be a major game changer. AFAK our Gaussian kernel pre-computes the squared norms of the diagonal of the kernel matrix. Maybe we can change our CDistance class to do the same here.
But let's go very systematically:

Use a precomputed distance and benchmark against a precomputed distance in sklearn. This gives us an estimate on how much the distance computation makes a difference. There is the CCustomDistance in Shogun, dont know how to do this for sklearn, but you could use a hack. Based on the outcome of this, we can identify more potential bottlenecks.

use precomputed x_squared norms in Shogun's Euclidean distance. The CGaussianKernel has inspiration for this.

use linalg dot products to compute the distance rather than the loop over elements

Sometimes computed on multiple data. Look out for loops over distance calls. This should be done via a single call that uses dot products between matrices and then using linalg. We might need to add a method distance_multiple(SGVector indices_a, SGVector indicesb) to the Distance class for this.

Finally, we could parallelise on multi-cores. But let's keep that out for now

The other points, can you comment on each?

scaling every row/column in a matrix by a number should be added to linalg. @lambday can you have a look at this?
random init as you described sounds good. How does scikit do it?
the ```shogun_2```` looks much better. But I did not get this "constant cost" reason.
Convergence. I agree, the WCSS is maybe a better idea. But let us only do this after the distance thing is solved.

Precomputing distances does not work, because k-means does not use pairwise distances.
Precomputing squared norms does not help that much for dense data - and k-means is designed for dense data. Using dot products is great if you can exploit sparsity only; for dense data the memory bandwidth is usually the limit and it does not pay off to save the extra sub instructions.
Stopping early (with a WCSS threshold, e.g. tol) should not be the default. It causes non-intuitive results where a point is not assigned to the closest center, or the cluster center is not the mean of the assigned points (if both were the case, it would be converged). It can be good to approximate k-means if you need maximum speed, and for probabilistic approaches such as Mini-Batch-Kmeans it is the best that you can do. But I would default to running until convergence (and stop when no point is reassigned anymore), and keep this optional. For fairness, set tol=0 in sklearn and maxiter to a sufficiently large value so sklearn also produces a converged result.

Thanks for your comments @kno10

You are right. But precomputing the distance allows to benchmark the algorithm itself, without the time spent computing distances. Therefore, it is instructive to hunt for bottlenecks. Especially if we cannot gain a lot via point 2.) we have to find another handle on speed.

True. Memory managment is key. @Saurabh7 maybe you can profile the memory usage to see what is going on, I saw a few "copy" commands on matrices when I had a look earlier. However, precomputing the squared norms helped in the Gaussian kernel case. Doesn't hurt for sure. And we have to start somewhere.

Agreed. @Saurabh7 maybe this can be made an option later (and turned on by default for the minibatch version)

Setting the sklearn tolerance is a good idea to compare

Wrt. 1: What do you want to precompute? k-means does not use pairwise distances. And the means move around. So how would you precompute anything useful?

 Too many things going on at once. Precomputing is not useful at all. Thanks for the hint!
@Saurabh7 how are things going?

@karlnapf Regarding your previous points

yes scaling every row by a different scalar is not possible in linalg
Random init , yes scikit uses the same concept.
In the new initialization we have to compute distances to centers which was not present in previous version, that is what i meant by a new constant cost. I will have a look at this deeper.
Adding tol for convergence : yes we could make it optional or default to zero.

Regarding squared norms: This could again be a case by case basis, depending on the data as discussed: might help or might not help much, could use up more resources, etc. This could be made optional too. I will try to find a way to compare this.
Next I am trying to use the precomputed squared norms in CEuclideanDistance as you suggested also making changes with new SGVector methods and linalg methods. getting some mixed results.
Have some academic commitments so going to be slow for a bit, will update with details soon ^^

Great, let us know when you had a chance to continue

@karlnapf  some udpates:

Used SGVector and linalg::dot to compute distances between vectors which led to decreased performance for some reason. Had a chat with @lambday in irc regarding linalg stuff. Anyway, to make this conclusive we decided this might need benchmarking itself with something like vectors starting with 2 dimensions going to say 100k dims logarithmically

used  CMath::dot , passing vectors as pointers (shuld be possible with linalg::dot soon with a new wrapper for pointers) along with precomputed x_squared_norms. This gave slightly better performance in some cases:

k
10
100

scik
3.1
32.5

shog
3.6
39.6

As for high level implementation only differences left as I see are the way they handle convergence and some minor things like how to handle empty clusters, etc.

Thanks. Interesting that we are still slower! Shogun being slower should not be the case if the Implementation is similar
The linalg dot is interesting -- it should also not be the case ...
For convergence, did you set the sklearn criterion to mimic Shoguns one as we discussed above?
Also, could you put your codes into a notebook and post the link here, so that we can see what is going on in a single file? Like #2991
Finally @kno10 is really right. The implementation is overly complicated. At some point, we should make it simpler to read

Yes I did set the tolerance to zero.
The current implementation is definitely too complicated and is almost 10x slower without this changes. The unnecessary things can be easily removed though.
not all implementation details are same, they handle some things with cython too, but we can definitely improve with time. I can keep trying in parallel.
Posting the ipy notebook soon.

@karlnapf alright i think I figured most of the differences out

I was actually wrongly calculating the lhs_squared_norms inside the iterations , fixed that moved it outside. Also removed some complicated linalg:: stuf i was using on matrices, using just linalg::dot on vectors now and it looks good

pre-calculating rhs_squared_norms now too. This is what I have to do inside iterations after rhs (center matrix) has been updated for the iteration.

also tried something like SGVector<float64_t> CEuclideanDistance::multiple_distance(int32_t idx_a)  and computing using matrices, as discussed in your one point, this worked good for bigger k but for smaller k it was not worth. So did not use this.

Results for both libraries after this changes are always alarmingly close now for different k ;)
Using all this I think I understand why the performance diference was always so big for this data.
Since randomly initialized and results vary while being close, average over 10 runs:
instances = ~20,000

k
10
100
1000

scikit
3.47
27.45
50.7

shogun
2.79
27.01
54.9

Nice!
Ok, so it seems sklearn does somehow better for large k, where for small k the Python overhead makes it slower. This must be some systematic thing as well. However, this is great work and we should get it merged in the next step. Can you send a PR?
The other thing you might want to check is using openmp to parallelise the distance computation among multiple cores. BUT I would do that after your current changes are merged.

This is now solved


ci: make tests more reliable and debuggable

Issue #, if available: The most common case for macOS tests failing seems to be vm init or vm start failing (or, partially failing), see: https://github.com/runfinch/finch/actions/runs/7493200344/job/20398390952, https://github.com/runfinch/finch/actions/runs/7493150724/job/20398225275, https://github.com/runfinch/finch/actions/runs/7493910137/job/20400678411
Description of changes:

Adds a delay between vm state changes. I'm not sure if this will fix the issue, but its worth a try

Testing done:

[x] I've reviewed the guidance in CONTRIBUTING.md

License Acceptance
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Weirdly, a lot of failure seem to be because this init command exits with a non-0 status. But in the code, it says it should ignore the exit status. Still investigating... Maybe you have an idea, @mharwani

Seems like all of the tests that use Virtualization.framework take minutes to init new VMs due to the conversion from qcow2 to raw disks. Investigating why it takes so long and possible ways to speed it up.

@weikequ can you take another look at this?

Oh it also looks like we need to update our required tests to account for the additional matrix options.


How to Get Progress Outside of Audio Component

Hey, this isn't really an issue but I didn't know where to ask this...
How can I get the current audio progress outside of the currently playing audio component?

You could do that with a template variable or with the ViewChild API:
<audio-track #audio *ngFor="let track of myTracks"  [track]="track" (onFinish)="onTrackFinished($event)">
...
</audio-track>
<div>{{ audio.progress }}</div>

import { AudioTrackComponent } from 'ionic-audio';

@Component({
  templateUrl: 'build/pages/page1/page1.html'
})
export class Page1 {
  @ViewChild(AudioTrackComponent) audioTrack: AudioTrackComponent;

  ionViewDidLoad() {
     let progress: number = audioTrack.progress;
  }
}

Let me know if it works.  The AudioTrackComponent  class exposes all audio properties as public.

#192

I am getting viewChild is undefined error...

sorry got it fixed... i did not import viewchild from the angular core... after import it works fine.


Add BITS token for Juno

Updated. Thanks for the feedback @JeremyParish69

Updated. Thanks for the feedback @JeremyParish69

Thanks. I still see the image now has transparency, but it still has that random marking in the top right...

Updated. Thanks for the feedback @JeremyParish69

Thanks. I still see the image now has transparency, but it still has that random marking in the top right...

Sorry about that. Should be all good now :)


Fix decode error for bllip parser

Closes #1920
This was fairly trivial to fix, but this seems like it was broken since py3 was rolled out. Do we know why more people didn't complain about this? Is it possible that this parser isn't being used at the moment?
I also added a very simple unit test, to capture simple breakages in the BllipParser for next time

It looks like the bllip-parser has been broken on Windows for quite some time now https://github.com/BLLIP/bllip-parser/issues/48
I'm not sure what to do here. We could just not have these new tests run on CI, or perhaps we should only install the pip package on non-Windows environments?

okay looking for a bit of advice here!
nltk's BllipParser has likely been broken for years. as a result, we can reasonably assume it is not being used
The parser also isn't unit tested. adding tests to the CI build is tricky, since the bllipparser pip package itself has been broken on windows since 2016. i see a few paths forward, and am unsure which to take:

fix the BllipParser, don't add unit tests to the CI
fix the BllipParser, add unit tests to the CI, but have our CI build only download the bllipparser pip package on mac and linux

this will add some complexity to the CI build for a modest gain in test coverage

deprecate / eventually delete the BllipParser since we don't think it's being used

This is a bit of a tricky one. I'm not familiar with BllipParser, which means I don't know whether it has functionality that isn't already possible through some other parser. That would affect whether we should care to fix the BllipParser.
However, if the fix is simple, then I don't see why we shouldn't at least push the fix.
Regarding CI, we already have some third party tools which are downloaded for Mac and Linux only, but adding a package which must be installed via pip to that might be a hastle. I already believe the CI is a bit vague and complex, and I don't really want to add to that even more.
Perhaps all of this means that I'm leaning towards option 1.
An interesting mix of everything would be to add the tests as quick unit tests to method/class bodies of nltk/parse/bllip.py itself. These doctests are not ran automatically with the CI, but interested users can pytest --doctest-modules nltk/parse/bllip.py. That way, your work of coming up with the tests isn't wasted either. (With # doctest: +NORMALIZE_WHITESPACE to get the tests passing etc.)
That's my view, others might disagree.

@dmcc do you have any advice for us please? Is there any point in us maintaining NLTK's BLLIP parser?

@dannysepler
#2903 has fixed a broken test case which snuck into this PR. I've merged develop into this PR to resolve this broken test.
That said, this PR still has broken tests from the PR itself.

Thanks @dmcc

sorry for the delay @tomaarsen, and thank you for the guidance!
thank you as well to @dmcc for confirming its usefulness!

Thanks Danny! Well done.


Add MMF OM5 as a cable choice

As ...
Nelly - Network Engineer
I want ...
I'd like to have Multimode Fiber (OM5) appear in the cable type drop-down
So that ...
We can accurately model our networks
I know this is done when...
I can set the cable type to Multimode Fiber (OM5)
Optional - Feature groups this request pertains to.

[ ] Automation
[ ] Circuits
[X] DCIM
[ ] IPAM
[ ] Misc (including Data Sources)
[ ] Organization
[ ] Plugins (and other Extensibility)
[ ] Security (Secrets, etc)
[ ] Image Management
[ ] UI/UX
[ ] Documentation
[ ] Other (not directly a platform feature)

Database Changes
None
External Dependencies
None

Added patch against ltm-1.6; PR for develop available shortly.

Patches added; I saw there's also a 'next' branch...for tracking 2.1? Is there anything I need to do to ensure these changes (and others, such as the ones for #4873) are rolled into 2.1?

Thanks for the PRs! next is indeed tracking 2.1 - we'll make sure all of the content in develop gets merged with that branch before we ship 2.1.0. :-)

Saw this got assigned to me; do you need any action from me on this?

Nope, sorry for the noise - just since you've been kind enough to open the PRs to implement this, wanted to make it clear that no one else needs to do so. Thanks again!


React Native Prelease Builds and CI

React Native often has breaking changes.  It'd be cool to stay on top of this by trying out their preleases (or even nightly builds).
We have the ability now to say ignite new OmgYisss --react-native-version 0.42.0-rc.3 --max.
It'd be great to hook this + our integration tests up to a robot to run these for us.
@GantMan has more info on Travis if we go with that.

And, in my humble opinion, the React Native core team could keep an eye on our CI to see what they're breaking. ¯_(ツ)_/¯

Mr Shruggy!  Your arm!!?
React Native is amazingly tested. The breaks are more by design. They're doing the right thing.

Not intending to implement this; closing.