How to show a message when customer orders more than current inventory quantity?

Hi, first of all, very, very thanks for your excellent plugin.
Here's the question: I'm trying to display a message in order to prevent users from ordering more than current inventory quantity. But always failed (tried to output it as the max property of an input element, or just a data-* attribute of a span containing item.quantity).
I think the problem is that I can't figure out how to output variant.inventory_quantity by using rivet.js...please help me!!
Btw, can I comparing these two values in rv-show, to let rivet.js decide whether or not to display the message?

Part of the problem here is that the object that Shopify returns when you request the Cart via AJAX does not include all the properties for a variant - annoyingly, inventory_quantity is one of those missing properties.
@gavinballard This is a problem that we had to solve on our current project as the client really wanted to provide useful feedback to the user as they add items to the cart up to the current inventory limit, but it involved making Shopify.getProduct requests for each item in the cart every time it changes - due to other complexities we solved this outside of CartJS, but would be interested to know your thoughts on building something into CartJS that would retrieve extra Product/Variant properties whenever the cart changed...

Thanks @tsutshiau!
As @michaelrshannon said, it's a bit tricky to render inventory_quantity from Rivets as it's not something returned by the Ajax API. Two things that could help:

Use the rivetsModels option (see here) when initialising Cart.js on your product page to pass a full JSON object containing the product data (which should include inventory quantities), like this:

<script type="text/javascript">
  CartJS.init({{ cart | json }}, {
    rivetsModels: {
      "product": {{ product | json }}
    }
  });
</script>

The next major update to Cart.js is going to contain much, much better paradigms for returning information and error messages to customers, so doing things like displaying out of stock error messages should be greatly simplified.

@gavinballard Also looking for a solution to this, is this something the library has solved in the meantime or do I need to pursue something similar to what @michaelrshannon describes?

@lmartins No change in the released version of the library at the moment I'm afraid. Inventory quantities are officially no longer returned in the Ajax API so for this to work we need a pattern to provide levels via Liquid.

Because I'm using the data attribute method of adding items to the cart I've found it difficult to add callbacks to the addItem function so I came up with my own work around for an error when adding an item to the cart:
I've added the following code to the request in CartJS.Queue
statusCode: {
  422: function(err) {
    if (CartJS.errorHandling.cartError) {
      CartJS.errorHandling.cartError(JSON.parse(err.responseText));
    }
  }
}

I've also added a place holder for CartJS.errorHandling in my own fork of cartjs which looks like this
CartJS.errorHandling = {
  cartError: function(error) { console.log("Error adding item to the cart: ", error.description) }
}

But of course this is modifiable in the actual Shopify theme files like product.liquid or theme.liquid after CartJS has be initialised.

Thanks for sharing @allanarmstrong! We definitely need a better error-handling story.

Thank you so much for sharing @allanarmstrong, this was indeed helpful.

Are there any plans to implement improved error handling when using data attributes or has this project gone stale?

@kyl-newton I would love to implement this (along with a raft of other improvements), but I have pretty much zero time to work on it at the moment. Happily accepting contributions from others.

Just for reference, I prefer the approach detailed in this gist rather than changing core - I use this if it is helpful to anyone:
$(document).ajaxComplete(function(event, jqxhr, settings) {
  if (settings.url == '/cart.js') { // CART GET COMPLETE

    console.log('GET /cart.js COMPLETE');

  } else if (settings.url.indexOf('/cart/') !== -1) { // CART POST COMPLETE

    if (settings.url == '/cart/add.js') { console.log('POST cart/add.js complete'); }
    if (settings.url == '/cart/update.js') { console.log('POST cart/update.js complete'); }
    if (settings.url == '/cart/change.js') { console.log('POST cart/change.js complete'); }
    if (settings.url == '/cart/clear.js') { console.log('POST cart/clear.js complete'); }

  } else { // NOT CART AJAX

  }
});

Would be great if in due course, Cart.js accommodates success/error status in callbacks. If I have a moment, I will surely PR this!


[Desktop/Web] Support native triple click text selections

Looking at https://github.com/flutter/flutter/pull/119046 it looks like triple clicking and some other keyboard text selections are now supported by flutter directly. Can we leverage the built in text selection in flutter to do selection in quill? This should replace the timer based stuff in the example project.

I agree

https://github.com/singerdmx/flutter-quill/issues/1131#issue-1625852384
did you solve this problem?

No

I guess this alteady was added and it is supported by the editor currently. I will close by the reason mentione before. @Blquinn, if the feature is not supported, please reopen

There was an issue 2 months ago referencing this, so we probably want to keep it for now. We also would want to have some labels for the priority of the issue.
Critical, Major, and Minor.


Link to Source-File does not work...

The Link to the sources does not work (http://www.beanshell.org/bsh-2.0b4-src.zip\)
Could you please correct this or give me a link where I could download the file?
thx in advance
nomike aka Michael Postmann
Reported by: nomike
Original Ticket: "beanshell/bugs/316":https://sourceforge.net/p/beanshell/bugs/316

This has been fixed now. Thanks!


Rename webdoc-template package

Did some clean-up today on the webdoc-template:

Rename package from pixi-webdoc-template to @pixi/webdoc-template
Updates highlight.js dependency security issue

https://pixijs.download/dev-rename-webdoc-template/docs/index.html

Codecov Report

Merging #7172 (458f88d) into dev (c7acb04) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               dev     #7172   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           17        17
  Lines          699       699
=========================================
  Hits           699       699

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7acb04...458f88d. Read the comment docs.

Codecov Report

Merging #7172 (458f88d) into dev (c7acb04) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               dev     #7172   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           17        17
  Lines          699       699
=========================================
  Hits           699       699

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7acb04...458f88d. Read the comment docs.


Relay support

Introduces relay support for AWS SQS.
You can run plumber in relay mode in foreground (via plumber relay aws-sqs ..) or via Docker.
Updated some of the flags to make the syntax a bit easier to use (and have less to type).
README updated; minor bugfixes.

Thank you to @uzarubin and @blinktag for helping me debug some weird git issues.

+1


remove old polyfills that should not be needed for IE9+ - #3038

for https://github.com/ractivejs/ractive/issues/3038

Looking at the kangax ES5 table and checking (caniuse)[http://caniuse.com), I'd say it's safe to remove all of these. After that, there're only two polyfills left out of the main build - performance.now and requestAnimationFrame, which aren't particularly big, so it'd probably make sense to just include them in the main build as well?

Yup. If we're only supporting IE11, pretty much all ES5 and web polyfills can be removed.

According to the kangax chart, we would still support IE9+. We might lose some very old versions of safari or the android browser, but it looks like even those are partially supported enough to get by.

What features won't work for "some very old versions of safari or the android browser?" I see in kangax that a few JS engines fail on "Enumerable properties can be shadowed by non-enumerables" (which I think should be worded in reverse), and "Array.prototype.sort: compareFn must be function or undefined," both of which do seem like very edge case problems.

I think "android browser" means pre-"Chrome WebView" Androids, versions below 4.4. Apple keeps iOS fairly updated so this shouldn't be an issue, unless you still use an iPhone 5 (which I think Apple ditched recently because they're not 64-bit) or older (which don't get iOS updates anymore).

@evs-chris, in this comment you mentioned the "polyfills [will be] in the default build", and in a comment in this PR, you said, "it'd probably make sense to just include them in the main build as well." What did you mean by "default build" versus "main build"? Also, are you thinking of putting the few remaining polyfills within built ractive.js, or would the build still have a separate polyfills.js file?

default build, main build, and ractive.js are all the same thing in my ramblings.
It didn't really make sense to have a polyfill for two tiny functions, so I pulled them into ractive.js. I didn't remove the polyfill file, since that might cause errors for people that have a merged CDN link. It now just prints a message to the console that it is no longer needed, and it will be completely removed for 1.0.


fix: performance issue caused by excessive use of clearTimeout/Interval

When a large number of tooltips are mounted, there is an excessive amount of unnecessary clearTimeout() and clearInterval() calls being made, which cause very slow re-renders. In our case, this PR alone shaved off much the re-render time of one particularly expensive re-render (850ms to 472ms).

Looks good, thanks!


publish<EMAIL_ADDRESS>

Requested by: @romtsn
Merge target: (default)
Quick links:

View changes
View check runs

Assign the accepted label to this issue to approve the release.
Leave the #retract comment under this issue to retract the release (original issuer only).
Targets

[ ] maven
[ ] github
[ ] registry

Targets marked with a checkbox have already been executed.  Administrators can manually tick a checkbox to force craft to skip it.

@kahest could you retrigger please? Feels like nexus is super slow, so we might need to reconsider the timeout limits we have

I guess we have to hold off until Monday or something, they still have incidents https://status.maven.org/


demag_gui does not handle multiple locations

@moonshoes87 @Caoimhinmg
Using the Python 3 stand-alone of pmag-gui and these files:
SheepCreekMagICFiles.zip
when I open the demag gui, only the sites in the first location "Western Locality" is displayed. If I change the display level to location the only option is "Western Locality" and the two others in the locations.txt file do not display. Maybe the Nick S-H group has only worked on files with one location, so this problem has not come up before. We definitely need a way to switch between viewing different locations and it would also be nice if when one is at the site display level that the location level is displayed somewhere.

@njarboe It should be able to handle multiple locations I used to test regularly with multi-location projects just in case. I'll give it a look and see if I can figure out what's up.

Hey @njarboe, sorry about the like 2 month delay on a real response to this. So I just looked in depth on this and the problem seems to be mostly with the data. The measurements.txt has the no dec or inc data for any of the specimens in any of the locations besides Western Locality so Demag GUI doesn't allow the user to try to display the measurement data as it doesn't exist. In addition the some of the fields are improperly filled in measurements.txt with what appears to be sample and specimen name data in dir_csd and dir_dec, though you could probably fix this in a spreadsheet program. Though the GUI seems to work normally on other data with more than one location, so I'm going to close this issue.

@Caoimhinmg, Thanks for taking a look at this. The file was quite malformed. I'll take a closer look at that in the future before reporting problems with pmag_gui. Hope you are enjoying Rice so far and the flooding is not too crazy where you are.


Create a merge task

It would be helpful to have a merge task.  At my company, we have many branches that need merged from the parent branch daily.  We achieve this with two calls to tf.exe (one to merge, one to checkin).  You already have a checkin task, just need a merge task.

Could you elaborate on how that task would deal with merge conflicts?
Would it only be able to merge between existing branch relations?
Does the build workspace already contain both source and target branches?
would it work on a gated/shelveset build?

I suspect that you'd be able to use 'tf merge' from the commandline before
calling my tf checkin task.
Merge doesn't have to interact with the repository I suspect,  so the
commandline should work just fine.
On Jul 26, 2016 14:41, "mslagle"<EMAIL_ADDRESS>wrote:

It would be helpful to have a merge task. At my company, we have many
branches that need merged from the parent branch daily. We achieve this
with two calls to tf.exe (one to merge, one to checkin). You already have a
checkin task, just need a merge task.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jessehouwing/vsts-tfvc-tasks/issues/34, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AD-uS0_uxFxKUDwiA2PJ2AK3I2L3QTnoks5qZlTugaJpZM4JVdXL
.

The tasks should on fail on merge conflicts that cannot be automatically resolved.  Maybe in later revisions we can add options for default actions (take source, take destination).
Yes - Otherwise a baseless merge would merge all items.  However, I think maybe just using empty textboxes for input would be better.  Let people decide what branches/folders to use.
No, just the target branches.  You don't need the source branch mapped to merge from it.
I honestly have no idea how that would interact.  (Lets not think about that).

I am currently using command line, so I don't think anything complex is needed.

Given that I need to provide support for the tasks I write. Or at least try
to do as best I can, we do need to think what happens in gated and
shelveset builds. Or at least provide a default option to skip the task in
those cases.
I don't have time until October for this, looking at my current priorities.
If you want to provide a pull request, we'll be able to at least test some
scenarios.
The update gated checkin is probably a good base task to start from, it has
all the conditional logic already built in.

Any updates on the merge task? It would be nice to have it.


(PUP-9008) Use initctl version --quiet to detect upstart daemon

Previously, we'd confine the upstart provider to platforms where the
/var/run/upstart-socket-bridge.pid file exists because that would mean
that the upstart daemon was running. This was not quite correct. The
upstart daemon could still be running even if that file did not exist
(e.g. such as on a RHEL 6 machine, for example).
We can use initctl version --quiet instead to check if the upstart
daemon's running. If that command fails, then the upstart daemon isn't
running so we cannot use the upstart provider. If it succeeds, then the
upstart daemon is running so we can use the upstart provider. We confine
our provider to platforms that result in a successful execution of this
command.

I've tested this on an Ubuntu 16.04 installation w/ upstart, and w/o upstart. See below:
Ubuntu 16.04 w/ upstart
root@j2ku2awdo1la09n:~# initctl version
init (upstart 1.13.2)
root@j2ku2awdo1la09n:~# /opt/puppetlabs/bin/puppet resource service hwclock ensure=running provider=upstart
Notice: /Service[hwclock]/ensure: ensure changed 'stopped' to 'running'
service { 'hwclock':
  ensure => 'stopped',
}

Ubuntu 16.04 w/ upstart package, but w/o upstart daemon running:
root@b5eptnkb6sbqywk:~# initctl version
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
root@b5eptnkb6sbqywk:~# /opt/puppetlabs/bin/puppet resource service hwclock ensure=running provider=upstart
Error: /Service[hwclock]: Provider upstart is not functional on this host
Error: Could not run: Provider upstart is not functional on this host

RHEL 6 machine:
[root@yj63a8kwqeevhcr ~]# initctl version
init (upstart 0.6.5)
[root@yj63a8kwqeevhcr ~]# /opt/puppetlabs/bin/puppet resource service start-ttys ensure=running provider=upstart
Error: Could not start Service[start-ttys]: Execution of '/sbin/start start-ttys' returned 1: start: Job failed to start
Error: /Service[start-ttys]/ensure: change from 'stopped' to 'running' failed: Could not start Service[start-ttys]: Execution of '/sbin/start start-ttys' returned 1: start: Job failed to start
service { 'start-ttys':
  ensure => 'stopped',
}

(the RHEL 6 failure shows that Puppet's correctly using upstart as the provider which is the point of this PR).

Would be nice to see the tests updated for this, if possible. I'm not sure of the current state of the upstart provider tests

@branan Sounds good. I added some unit tests. Acceptance tests seem a bit overkill for this, but I can add those too if you like.

CLA signed by all contributors.

I'm totally happy with unit tests. Given that upstart is on its way out, I don't see much value in investing in any sort of integration/system-level testing


Other than MySQL is not an error

PHP Version
any
Shopware Version
any
Actual behaviour
Performance recommendations render other than a MySQL DB as an error.
It is clearly not such and other, like MariaDB, have been recommended not long ago.
Expected behaviour
Render other than MySQL as a fyi in performance recommendations.
Steps to Reproduce?
Open Tools / System Status on any instance with other than MySQL DB.

I changed it to a warning.

If MariaDB is used in place of MySQL I think it also should just return a warning, as MariaDB is also officially supported.
diff --git a/src/Components/Health/Checker/PerformanceChecker/MySQL8Checker.php b/src/Components/Health/Checker/PerformanceChecker/MySQL8Checker.php
index 02b8fad..3dfef20 100644
--- a/src/Components/Health/Checker/PerformanceChecker/MySQL8Checker.php
+++ b/src/Components/Health/Checker/PerformanceChecker/MySQL8Checker.php
@@ -24,7 +24,7 @@ class MySQL8Checker implements PerformanceCheckerInterface, CheckerInterface

         if (isset($extractedVersion['mariadb'])) {
             $collection->add(
-                SettingsResult::error('mysql8', 'MySQL', $version, 'MySQL 8.0', 'https://developer.shopware.com/docs/guides/hosting/performance/performance-tweaks#mysql-instead-of-mariadb')
+                SettingsResult::warning('mysql8', 'MySQL', $version, 'MySQL 8.0', 'https://developer.shopware.com/docs/guides/hosting/performance/performance-tweaks#mysql-instead-of-mariadb')
             );

             return;


Update naming in module to adjust with latest

Description
LogDNA -> Log Analysis
Sysdig -> IBM Cloud Monitoring

By submitting this issue, you agree to follow our Code of Conduct

Do we want to rename variable names (breaking change) or just documentation?

@vburckhardt I am assuming that we want to change the variable names and proceeding ahead as per it.

https://github.com/terraform-ibm-modules/terraform-ibm-observability-instances/pull/221

#221

Closed this PR as this issue will require a re-write of the entire code after the observability module is breakdown into individual components is merged.

A lot of the variables (input and outputs) for submodules were renamed as part of the refactor. Hopefully that makes updates easier.

PR Merged


More/better customisation of the "active/highlighted/current" line in synced lyrics

Requested here: https://www.reddit.com/r/foobar2000/comments/mm70rb/i_made_an_opensource_alternative_to/h58wwjy/
In particular allowing you to make the current line have a larger font size

Note to my future self: I tried this because it seemed really simple, and it actually would be if not for the fact that both position and colour of the highlighted line gets interpolated smoothly when it goes in and out of being the active line. While we can scroll smoothly and interpolate colours smoothly, it appears that this doesn't work nearly as well for font sizes.
The attached video has an example of the stuttery resizing behaviour I'm getting with what I tried (it actually looked like this, its not stuttery because of low fps on the video):
https://user-images.githubusercontent.com/4639662/153944964-986756c7-b717-4f2b-94f4-82246c8b1381.mp4
On top of this, it looks really bad how the text gets re-flowed as it grows/shrinks. Maybe I'll come back to this later when I'm looking to spend more time on it.

Animating text size is complicated and it's not at all clear how it is intended to look. Closing.


glitch

Type: Bug
for some reason my vs code software is glitching
VS Code version: Code 1.76.2 (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:55:54.936Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: Yes

System Info

Item
Value

CPUs
AMD Ryzen 7 4800H with Radeon Graphics          (16 x 2895)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
undefined

Memory (System)
7.42GB (0.85GB free)

Process Argv
--crash-reporter-id 12d8960e-bc11-4c9c-9062-7e4ee91731a5

Screen Reader
no

VM
0%

Extensions (22)

Extension
Author (truncated)
Version

python-image-preview
076
0.1.2

vscode-django
bat
1.10.0

pytorch-aws-scratch-helper
chi
0.0.11

code-runner
for
0.12.0

chatgpt-vscode
gen
0.0.4

chatgpt-vscode-plugin
Jay
1.1.10

data-workspace-vscode
ms-
0.4.0

mssql
ms-
1.18.0

sql-bindings-vscode
ms-
0.3.0

sql-database-projects-vscode
ms-
0.21.0

isort
ms-
2022.8.0

python
ms-
2023.4.1

vscode-pylance
ms-
2023.3.40

jupyter
ms-
2023.2.1200692131

jupyter-keymap
ms-
1.1.0

jupyter-renderers
ms-
1.0.15

vscode-jupyter-cell-tags
ms-
0.1.8

vscode-jupyter-slideshow
ms-
0.1.5

LiveServer
rit
5.7.9

php-debug
rob
1.14.0

pytorch-snippets
SBS
1.0.2

php-debug
xde
1.32.1

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vscaat:30438848
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
nodejswelcome1:30587005
282f8724:30602487
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
a9j8j154:30646983
6233i204:30672705
vsctsb:30677850
vscodeenable:30660116
pythonb192cf:30669361
azdwalk:30687957
pythonms35:30701012

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.
Happy Coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


arc patch failed for referenced diffs

investigate why https://reviews.llvm.org/D70656 fails

running arc patch D70656on the command line on master branch:
INFO  Base commit is not in local repository; trying to fetch.
Created and checked out branch arcpatch-D70656.
Created and checked out branch arcpatch-D65433.
Checking patch clang-tools-extra/clangd/unittests/TweakTests.cpp...
Checking patch clang-tools-extra/clangd/unittests/TweakTesting.h...
Checking patch clang-tools-extra/clangd/unittests/TweakTesting.cpp...
Checking patch clang-tools-extra/clangd/refactor/tweaks/DefineInline.cpp...
Checking patch clang-tools-extra/clangd/refactor/tweaks/CMakeLists.txt...
Applied patch clang-tools-extra/clangd/unittests/TweakTests.cpp cleanly.
Applied patch clang-tools-extra/clangd/unittests/TweakTesting.h cleanly.
Applied patch clang-tools-extra/clangd/unittests/TweakTesting.cpp cleanly.
Applied patch clang-tools-extra/clangd/refactor/tweaks/DefineInline.cpp cleanly.
Applied patch clang-tools-extra/clangd/refactor/tweaks/CMakeLists.txt cleanly.

 Cherry Pick Failed!
 Exception
Command failed with error #1!
COMMAND
git cherry-pick 'arcpatch-D65433'

STDOUT
Auto-merging clang-tools-extra/clangd/unittests/TweakTests.cpp
CONFLICT (content): Merge conflict in clang-tools-extra/clangd/unittests/TweakTests.cpp
Auto-merging clang-tools-extra/clangd/unittests/TweakTesting.h
CONFLICT (content): Merge conflict in clang-tools-extra/clangd/unittests/TweakTesting.h
Auto-merging clang-tools-extra/clangd/unittests/TweakTesting.cpp
CONFLICT (content): Merge conflict in clang-tools-extra/clangd/unittests/TweakTesting.cpp
CONFLICT (add/add): Merge conflict in clang-tools-extra/clangd/refactor/tweaks/DefineInline.cpp
Auto-merging clang-tools-extra/clangd/refactor/tweaks/DefineInline.cpp

STDERR
error: could not apply f0188049e4e... [clangd] DefineInline action availability checks
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

(Run with `--trace` for a full exception trace.)

After some discussion:

This is a common problem to be expected when using Phabricator without a "staging area".
Right now, we do not want to change people's workflows, so we'll not set up a staging area.

I created a ticket to consider/discuss this: #66


General Issues & Solutions

I thought I might write down some issues I had setting up echonetlite2mqtt and the solutions I inevitably came too.
Hopefully this saves people the hours spent debugging.
My setup is a WTY2001, with 2 WTY24173W.
These use リンクプラス (Link Plus) however the WTY2001 does support EchoNetLite out of the box.
My first mistake was plugging it into a separate VLAN that was on a different subnet, I thought this would not be an issue.
However from my understanding "ECHONET_TARGET_NETWORK" env variable is used to match your interface IP, then your interface IP is the IP used to Initialize the EchoNetCommunicator. If your subnet does not match, it looks like this never gets initialized properly?
My second mistake was not setting docker to network host. This can cause issues with the "ECHONET_ALT_MULTI_NIC_MODE" due to Multicast.
However I was not using that env variable, instead, using a virtual interface of course my interface subnet is going to be different to the "ECHONET_TARGET_NETWORK". Causing the same issue as above.
Hopefully this helps someone having the same issues I was,
Glad to say now everything is working and using switches through Home Assistant has never felt so good.

@AnthonyOGorman
Thank you for your report.
As you report, ECHONET_TARGET_NETWORK is used to pick one network interface when a host has multiple network interfaces.
echonetlite2mqtt cannot find the device even if you specify a different subnet for ECHONET_TARGET_NETWORK.
For the same reason, docker environments cannot find devices without the --net=host option.
I believe ECHONET Lite only works within the same subnet.
I will add it to Readme.md later.

@AnthonyOGorman
But I don't know about the "ECHONET_ALT_MULTI_NIC_MODE" problem.

My second mistake was not setting docker to network host. This can cause issues with the "ECHONET_ALT_MULTI_NIC_MODE" due to Multicast.

What happened at this time?
If this option causes the application to crash, I have to fix it.
The following command had no problems other than not finding the device, but what about your environment?
docker run -d -e ECHONET_ALT_MULTI_NIC_MODE=1 banban525/echonetlite2mqtt

Just to confirm as there’s discussion in another thread, are you able to control your Link Plus  lights via echonetlite2mqtt with the WTY2001?
I have the aiseg2, which is a similar device from Panasonic with echonetlite support, but it only allows me to control other companies’ devices via echonetlite. It does nothing to let me control my Link Plus lights via echonetlite.

Did not realize this thread has comments, sorry for the late reply!

@AnthonyOGorman
But I don't know about the "ECHONET_ALT_MULTI_NIC_MODE" problem.

My second mistake was not setting docker to network host. This can cause issues with the "ECHONET_ALT_MULTI_NIC_MODE" due to Multicast.

What happened at this time? If this option causes the application to crash, I have to fix it.
The following command had no problems other than not finding the device, but what about your environment?
docker run -d -e ECHONET_ALT_MULTI_NIC_MODE=1 banban525/echonetlite2mqtt

No crashes or issues here, just trying to use "ECHONET_ALT_MULTI_NIC_MODE" without the --net=host docker option. Which was my mistake.

@AnthonyOGorman Just to confirm as there’s discussion in another thread, are you able to control your Link Plus lights via echonetlite2mqtt with the WTY2001?
I have the aiseg2, which is a similar device from Panasonic with echonetlite support, but it only allows me to control other companies’ devices via echonetlite. It does nothing to let me control my Link Plus lights via echonetlite.

Yes everything is working and controllable using the WTY2001, I was about to buy the AISEG2 but did not need the screen so ill consider myself lucky for now. Ill drop by the thread and see if I have any solutions.

Thank you for your reply! Wow your link plus lights are discoverable over echonetlite using the WTY2001? I am incredibly jealous. Do you mind posting a (redacted is ok) screenshot of what echonetlite2mqtt’s web ui shows for a link plus light?

Thank you for your reply! Wow your link plus lights are discoverable over echonetlite using the WTY2001? I am incredibly jealous. Do you mind posting a (redacted is ok) screenshot of what echonetlite2mqtt’s web ui shows for a link plus light?

Here are some screenshots:

What do you mean before by "only allows me to control other companies’ devices".
The spec sheet did mention AISEG2 being supported for Link Plus devices.

With Aiseg2, I’m unable to discover any of my lights over echonetlite, but I can connect other companies’ echonetlite products to the aiseg2 and control them via the aiseg2 web ui, which isn’t really ideal.

I believe Link Plus. WTY2421, WTY22173, and similar.

I believe Link Plus. WTY2421, WTY22173, and similar.

And you paired the devices to the AISEG2, as the devices don't communicate with the AISEG2 over echonetlite, they communicate via 923mhz.

No crashes or issues here, just trying to use "ECHONET_ALT_MULTI_NIC_MODE" without the --net=host docker option. Which was my mistake.

Thank you for the reply.
I was relieved that echonetlite2mqtt seemed to have no problems.
And, thank you for reporting your light's behavior.
I believe this report will be useful to other users as well.

And you paired the devices to the AISEG2, as the devices don't communicate with the AISEG2 over echonetlite, they communicate via 923mhz.
See page 2 of manual.

Correct, the lights are paired with the Aiseg2. I presume your lights are paired with the WTY2001 over 923mhz, and the WTY2001 acts as a bridge for echonetlite, is that not correct?

Correct, the lights are paired with the Aiseg2. I presume your lights are paired with the WTY2001 over 923mhz, and the WTY2001 acts as a bridge for echonetlite, is that not correct?

Yes, Had a look though AISEG2's documentation to see if you have to enable Echonetlite, could not find anything.
It could be that echonetlite2mqtt is detecting all other echonetlite devices that are on the network but cant detect the AISEG2.
Is there a web interface for the AISEG2?

 Aiseg2 does have a web ui, though as best as I can tell there’s no means of enabling lights <-> echonetlite bridging.
In any case I pulled the trigger on a WTY2001 and I’ll try it out once it arrives! Saves me from a larger project.

Just one more check, which docker image tag are you using?

I don’t recall which tag, but fwiw even rolling it manually with the various echonetlite libraries out there doesn’t turn up any devices.

Give this image a shot if you have time banban525/echonetlite2mqtt:master

@aftertommy @AnthonyOGorman
I will explain about tag:master.
tag:master is a release candidate for the next v2.3.0.
The major differences from v2.2.0 are as follows.

Updated the library for EchonetLite, so finding the device is a little faster.
Added mode for multiple IP environments.

But, Both are unlikely to solve the aftertomy's problem.
I think there is a problem with the WTY2001 settings or network environment.
Unfortunately, I don't have リンクプラス(Link Plus), so I can't experiment with it.

Sorry for hijacking this thread. Confirming that the WTY2001 is indeed broadcasting the lights over echonetlite as @AnthonyOGorman described.
Thanks everyone.

I would love to do some debugging with it if I ever find AISEG2 on sale or second hand. But glad to hear its all working with the WTY2001!

I would also like to hijack this thread for a short question. Is it possible to do a soft on with LEDs?
Basically that the light goes not on 0 seconds from 0-100 brightness but over for example 2 seconds from 1-100.
I assume it may not be possible with normal LEDs, if that's the case, does it work in combination with smart bulbs?

@flutschepfeil
Is the feature you want a light bulb that gradually brightens?
ECHONETLite2MQTT does not have such functionality.
However, it seems that this function can be achieved by sending a brightness change command many times to a smart bulb that can change brightness.
But it's not easy.

@banban525 @aftertommy
Correct.
But I have to admit that this question is maybe more Panasonic Advance Plus related than echonetlite2mqtt.
If we take the Philips Hue with home assistant as example, the turn on gradually during maybe a 1 sec period instead of instantly.
Since the Panasonic Advance Plus switch turn on the power to the bulb, I do not know if it can automatically start at 100% or if you can somehow manage a soft transition to 100%.
banban525, I assume you mean that first the switch gets switched on (hoping that the smartbulb wont activate instantly) and then a command is issued from Home Assistant, that the bulb is turned on gradually.
This may indeed work, but here I would like to ask if someone tried it and confirm if it is possible.
I assume, normal switches can not dim the bulbs, meaning it is not possible via the switch itself.


[Bug]: Error with OpenAI server: API request failed with status code 400

Your current environment
The output of `python collect_env.py`

🐛 Describe the bug
I encountered an error with the OpenAI server:
[openai_api_compatible] Error: API request failed with status code 400: {"object":"error","message":"{'type': 'string_type', 'loc': ('body', 'messages', 2, 'function_call'), 'msg': 'Input should be a valid string', 'input': {'name': 'function call', 'arguments': '{}'}, 'url': '[https://errors.pydantic.dev/2.6/v/string_type'}]","type":"BadRequestError","param":null,"code":400}

After thorough investigation of the code, I suspect the issue may lie in the formatting of the 'message'.

Should be resolved by https://github.com/vllm-project/vllm/pull/3467


Can we implement e-mail support

We have not seen any method for sending automated e-mails (for registration, password resets, ...). It would introduce the need for a SMTP/IMAP service I think...

consider creating a tag "extra", for features that are not part of the assignment, but could be good/nice-to-have/etc...

We can implement this, but will require a SMTP server. Marking this as extra.

Done


Roll engine f4951df193a7..471a2c89a689 (11 commits)

https://github.com/flutter/engine/compare/f4951df193a7...471a2c89a689
git log f4951df193a7966f9ed4da43d555eee0913d84d1..471a2c89a689ff1d5c70a5deb4565f646380bce6 --no-merges --oneline
471a2c89a Send scroll events from the macOS shell (flutter/engine#8056)
2fe9c9b5f Roll src/third_party/skia 72542816cadb..801a9c16d81e (46 commits) (flutter/engine#8060)
3335764ae Skip skp files in license check (flutter/engine#8050)
7f16789b2 Remove redundant thread checker in FML. (flutter/engine#8053)
840c5233a Correct URL for Cirrus CI build status badge (flutter/engine#8054)
57c120a29 remove extra source files (flutter/engine#8052)
4773375c6 Used named conditionals for platform specific dependencies and suppress Android and Windows hooks on Mac. (flutter/engine#8051)
70a18b515 Add clang static analysis support to gn wrapper (flutter/engine#8047)
b30f9897b Improve elevation bounds for physical shape layers (flutter/engine#8044)
e37bd27b4 Fix weak pointer use violations in shell and platform view. (flutter/engine#8046)
dd80fc9ff Add engine support for scrollwheel events (flutter/engine#7494)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (mklim@google.com), and stop
the roller if necessary.

Auto-roller completed checks. About to merge.


Looking for new maintainers

:warning: As many have noticed, this project could do with some work. I'm looking for people to take over the project. If you're interested please leave a comment below. Once we have one or two people, I'll make a Github organisation and move this project there.
Required tasks

Resolve outstanding issues
CoffeeScript is becoming obsolete - we need to change the source to ES6 via Babel

Future tasks

Implement more styles

Style changer in the docs

Possibly remove jQuery dependency?

See http://substack.net/weaning_yourself_off_jquery
Will need a small animation library? CSS3 animations?
Maybe this will need two builds notify.js (modern) and notify.legacy.js (which includes jqlite)

Make me a maintainer. I'll take it over.

Just nuke the coffee script and work with the outputted source. CoffeeScript sucks, and it's not necessary. All of notifyjs-combined is 600 lines. There is no need for this project to have a build system or use coffeescript.

Hey @EvanCarroll Yep agreed - it should be pure JS. A few notes:

I'm using gh-pages instead of master to host the docs/
I'm happy for it to just use ES5. Without a build system, how would we achieve https://github.com/jpillora/notifyjs/tree/gh-pages/dist ?

@EvanCarroll could you please fork and do the switch to JS? I'll review then give you commit access

@jpillora  https://github.com/jpillora/notifyjs/pull/45

Let's put this in the README.md and start to create real docs for the project and close this. It's not an issue that will go away and every project wants productive people.

Agreed


Adding TAC option

closes https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/7843
Customizing video filters is now available to set in the TAC

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@CarolynRowe PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Add Urban Dictionary plugin

I can work on this one, I've done a Twilio SMS script for Hubot before so this should be somewhat similar.
@CapacitorSet - Do you just envision that this plugin takes a word or phrase from the chat input (or given to the bot) and uses the Urban Dictionary API to return both a definition of the word/phrase as well as a link to view the definition on UD?

Yes, pretty much. Something like the following:

User: /ud

Bot: telegram: A once-popular system of telecommunications, in which the sender would contact the telegram service and speak their message over the phone.  The person taking the message would then send it, via a teletype machine, to a telegram office near the receiver's address.  The message would then be hand-delivered to the addressee.  From 1851 until it discontinued the service in 2006, Western Union was the best-known telegram service in the world.
Hey, let's send Mom a telegram for her birthday! She just loves receiving that little yellow envelope.

Since you're at it, could you try writing your plugin based on the documentation in the wiki? You would help spot any missing or incorrect information.

Will do, I believe you meant the usage to be:

User: /ud telegram

And then the bot respond with what you queried

Oh welp, indeed.

One of the things I found upon initial setup that may be an issue for new users:
Running npm run firstrun when it's nearly finished will ask you for an array of user ID's to treat as global admins. I did not try to skip this, though I assume you can. It seems like a user has two options during this step:
Skip it by pressing ENTER and optionally adding them later in config.json under "globalAdmins"
Find their Telegram user_id by searching Telegram for what they named their bot, send it a test message, and then visit (or make a GET to) https://api.telegram.org/botYOUR-API-KEY/getUpdates which will return some json. The key result will be an array of objects, each containing a message key that is an object, this object has a key called from which has a property called id this is your user ID. In my case that was: result[0].message.from.id if you're looking at it from a return versus in the browser.

Hmm, indeed, that's not super friendly. You're welcome to make a PR - though, remember that in order to count towards the Hacktoberfest they must be opened after October 1.
I opened a new issue about that, and wrote down a couple of notes on how I'd fix that. If you wish to "claim" it (perhaps in occasion of the Hacktoberfest) just let us know in the linked issue.

I'll focus on this issue for now. When I go to send the PR if the other isn't claimed i'll take a stab at fixing the documentation, I think the setup script is working well.

How's this looking?

Looks just right! However, why the long space between the beginning of the line and the example usage?

Not sure, I'll try tweaking it later on. Not sure if it's my "formatting" in the template string or something else.

Fixed. I replaced my returns with two new line characters.
https://i.imgur.com/mWTlzRs.png

Sounds good.

Fixed in #139.


Support both Namespace and dict for hyperparameter saving/loading.

🚀 Feature
Let the user pass a dict to LightningModule so that after model saving it can be restored using load_from_checkpoint or load_from_metrics.
Motivation
Currently, there is nothing that prevents the user from passing in hyperparameters to LightningModule via dictionary (or even somthing else). However, the model loading/saving assumes it is always a argparse.Namespace. This could potentially be an issue when load_from_checkpoint restores the module with a Namespace passed in instead of dict.
Pitch
The model saving currently converts Namespace to dict and the model loading converts it back to Namespace.
Pitch 1: Also save the type inside the checkpoint, e.g.,
checkpoint['hparams'] = dict(hparams)
checkpoint['hparams_type'] = type(hparams)

and when restoring we instantiate with the appropriate type.
Pitch 2: Dump the whole hparams object (Namespace, dict, ...) into checkpoint without converting it to dict first and let pickle take care of the rest. Most flexible option but could give problems when loading from checkpoint.
Alternatives
Somehow restrict the user to only use argparse.
Additional context
The idea was suggested by @williamFalcon in PR #919.

support both argparse and dict...

yes, that's the idea. changed the title to make it more clear.

partially related to #525
Also, there was an attempt to pickle the hparams directly but it never got merged: #615
@neggert do you have any thoughts about this? would pickling the hparams object directly work?

implemented in #1029


SVM: add a sample stand-alone application based on SVM

Problem
SVM API enables development of advanced applications based on Solana execution engine. Writing such applications from scratch is a non-trivial task currently requiring knowledge of Solana run-time internals.  A real-life SVM based example application is helpful for developers to jump-start working with SVM.  An example presented in this PR is a client-server pair of applications that demonstrate using SVM to develop a Json-RPC server that responds to transaction simulation requests from a client program.
Summary of Changes
Add an extended example to the SVM crate

json-rpc-server is a server application that implements several Solana Json RPC commands, in particular simulateTransaction command to run a transaction in minimal Solana run-time environment required to use SVM.
json-rpc-client is a sample client application that submits simulateTransaction requests to json-rpc server.
json-rpc-program is source code of an on-chain program executed in the context of the simultateTransaction command submitted by the client program.

@joncinque, @pgarg66 This is still a very early draft, as I'm working on cleaning up the code and integrating this example in CI.

@dmakarov It's looking good.
I haven't reviewed the RPC server code. Just wondering if it's least amount of code for the tx simulation. Maybe it is.
At some point, we should consider refactoring the server code a little bit more, to isolate SVM related work. This will help the audience/users of the example to see what they need to do to implement something similar on their own. This can be done in a future PR though.
Leaving it open for now, so others can review it as well.

Initially, I placed the code under svm/examples/ subdir, so that cargo workspace could be used to specify dependencies, but because of circular dependencies solana-core <--> solana-svm, and solana-rpc <--> solana-svm, reported by order-crates-for-publishing.py, I had to move everything under a separate subdir.

Check out #2474 - I may have gotten it to work for me. If it can be done for this example, too, it would be great to have all of these SVM examples under svm/examples/, and also use workspace dependencies and configurations.

I still get Error: Circular dependency: solana-rpc <--> solana-svm from order-crates-for-publishing.py script.

I removed some of the redundant code. Also moved SVM related code to a separate module svm_bridge.

@dmakarov the refactoring is looking great! Let me know when this is ready for another look.

I'm going to merge this now with the intention to continue improving this in follow-up PRs, which seems to be more manageable.


Add unit tests for get_topic.go

The file is in here get_topic.go. Have a look at create_topic_test.go as an example

@matheusmosca I would like to work on this issue if it is still not assigned to anyone else. Please assign it to me.

@matheusmosca I would like to work on this issue if it is still not assigned to anyone else. Please assign it to me.

Sure, I'll assign it to you @amoghrajesh.

@matheusmosca the usage of the application is not clear yet. We are able to build it but can not figure out how the usage goes.


Use Sentinel-1 GRD image products as input

Consider whether Sentinel-1 GRD products can be directly used for this target analysis.
If they can, the advantage is that users do not need to first use some other processing software (Gamma, SNAP, etc) to generate input images for CoRAL.

Since CoRAL relies on co-registered image stack as input, additional functionality would need to be coded (or borrowed from other packages) as follows:

a cross-correlation algorithm to align GRD images
an algorithm that converts lat/lon CR positions in to radar range and azimuth coordinates


refactor: add debug logging

:tada: This PR is included in version 1.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Overwrites existing .bash_profile :O

Tried running the git-free install. I was surprised to learn that my .bash_profile was overwritten.
Any way I can revert or find my original .bash_profile?
This is a huge concern for me...

Any way I can revert or find my original .bash_profile?

Nope, sorry. What did you think “installing” the dotfiles would do? Should I clarify this in the readme?

@mathiasbynens I would suggest instead of overwriting the existing .bash_profile you can create with different name such as .dotfile_profile and then you can import this at the end of file in existing .bash_profile.
I lost all my important settings. :(

There is a warning when you run bootstrap.sh: https://github.com/mathiasbynens/dotfiles/blob/e7ef66da4f6a077261286df3729d823a1d39e04b/bootstrap.sh#L16

@mathiasbynens pst! https://github.com/nvartolomei/dotfiles/blob/master/script/bootstrap do_backup

Seriously???
I've just lost my ./bash_profile!!!

This may overwrite existing files in your home directory.

Well... it may doesn't quite cut it doesn't quite say which files! Something like this probably make you thing twice:

This will overwrite your .bash_profile file.

And why do you override anyhow? When you install nvm it just adds to that very file - doesn't override it.
I'd seriously consider not doing what you are doing at the moment. It's pretty insane.

@Izhaki
This is Mathias’s dotfiles. Not dotfiles enhancer for everyone.
Also there is huge warning at the top of readme:

Warning: If you want to give these dotfiles a try, you should first fork this repository, review the code, and remove things you don’t want or need. Don’t blindly use my settings unless you know what that entails. Use at your own risk!

Why should it append .bash_profile? This is a backup of all settings for one particular person. This means that person have ability to quickly restore all settings in one command. Appending .bash_profile only make sense for some general public tools that need it to work.
This is not "awesome macOS booster", this is "Mathias’s private settings"! It shouldn't be used straight from his repository without any alternations!

@ealeksandrov
I completely understand what you are saying.
But there is some disregard to how people actually behave:

You search how to add your git branch to the prompt.
You find a link to this repo on SO.
It looks awesome, under active development, and has gazillion of starts (making it trustworthy).
You decide to try it out.

When was the last time you, or anyone you know, have read T&C?
One pivotal concept in HCI is that users are either in:

task completion mode
exploration mode

Whilst in the former, people take any shortcut possible in order to complete the task.
So accidents like this are destined to repeat.
I don't think the solution is important here (@nvartolomei one would save me). It is the denial of the problem that is the real issue.

I don't think that Mathias should do something just because his project is so popular and many people don't read readme (that's not T&C, that's top of Readme).
This is not an issue for source value of project, so it's ultimately for author to decide how to handle it. It's just a dump of shared code, it's not a product with support.
Even License states it clearly - NO Liability or Warranty.

As stated before, even if you choose to ignore the README, you are presented with a warning when running bootstrap.sh: https://github.com/mathiasbynens/dotfiles/blob/e7ef66da4f6a077261286df3729d823a1d39e04b/bootstrap.sh#L16
I’m locking this issue as I believe everything has already been said.


Change success message on permission change

Fixes #1880  .
Changes proposed in this PR:

Change success flash notice message when permissions are updated.

@xsrust I suggest we leave this one for @briri to review, since he's done work recently on generalising the flash notices and might want to have a say in this change.

It looks like the perms.first is a nil for some reason and being passed into that success_message function. If you pass in a Perm it should produce the correct error message.
I think it would be better to send in Perm.new than hard code a message here so that we can keep those messages consistent in the future should we need to modify them.

@briri agreed, that's probably a better solution. Have updated now.


calibration UI

Adds a generic system for:

Starting a calibration procedure on a hardware
rendering calibration procedures actions that constitute a procedure
dispatching the actions that make up the procedure.

Next steps.

backend needs to maintain procedure progress state, which actions have been taken so far. This will allow the ui to reflect procedure progress.
add ui for the transformers, this will tell user when procedure was last run.
documentation, the Calibration Procedure framework on the backend should be described
Calibration Procedure for optical density hardware.

Demo

Config used in development:
{
  "name": "Evolver",
  "experiment": "unspecified",
  "vials": [
    0,
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10,
    11,
    12,
    13,
    14,
    15
  ],
  "hardware": {
    "temp": {
      "classinfo": "evolver.hardware.standard.temperature.Temperature",
      "config": {
        "name": "Temperature",
        "calibrator": {
          "classinfo": "evolver.calibration.standard.calibrators.temperature.TemperatureCalibrator",
          "config": {
            "name": "TemperatureCalibrator",
            "dir": "calibration_files",
            "created": "2024-11-20T21:06:13.563442",
            "expire": "P168D",
            "input_transformer": null,
            "output_transformer": {
              "0": {
                "name": "LinearTransformer",
                "dir": "calibration_files",
                "created": "2024-11-22T14:10:28.034639",
                "expire": "P168D"
              },
              "5": {
                "name": "LinearTransformer",
                "dir": "calibration_files",
                "created": "2024-11-22T14:10:28.034639",
                "expire": "P168D"
              },
              "10": {
                "name": "LinearTransformer",
                "dir": "calibration_files",
                "created": "2024-11-22T14:10:28.034639",
                "expire": "P168D"
              },
              "15": {
                "name": "LinearTransformer",
                "dir": "calibration_files",
                "created": "2024-11-22T14:10:28.034639",
                "expire": "P168D"
              }
            },
            "calibration_file": null,
            "vials": [
              0,
              5,
              10,
              15
            ]
          }
        },
        "vials": [
          0,
          5,
          10,
          15
        ],
        "addr": "temp",
        "slots": 16,
        "serial_conn": null
      }
    }
  },
  "controllers": [],
  "serial": {
    "classinfo": "evolver.serial.EvolverSerialUART",
    "config": {
      "name": "EvolverSerialUART",
      "port": "/dev/ttyAMA0",
      "baudrate": 9600,
      "timeout": 1
    }
  },
  "history": {
    "classinfo": "evolver.history.standard.HistoryServer",
    "config": {
      "name": "HistoryServer",
      "experiment": null,
      "partition_seconds": 3600,
      "buffer_partitions": 3,
      "default_window": 3600,
      "default_n_max": 5000
    }
  },
  "enable_control": true,
  "interval": 20,
  "raise_loop_exceptions": false,
  "abort_on_control_errors": false,
  "abort_on_commit_errors": false,
  "skip_control_on_read_failure": true,
  "log_level": 21
}

A few code comments, and a few I'm dropping here. Some may not be relevant to this PR (I think some are perhaps backend related or co-dependent), but dropping here in case:
* I didn't notice a place to name the procedure, eventually users should be able to select a named calibration. If this is part of config, it would still make sense I think to be able to name it here.

* A number of the steps on the temp procedure (on test evolver) look like they not only don't take input, but also shouldn't even need confirmation. For example, while filling the vials should require clicking "done", the "hardware will read ..." and "calculate the fit" don't really need anything from the user (and they wouldn't be sure what to do). Is this mostly a backend thing?

With the CalibrationProcedure being on the Calibrator class, itself on Hardware there's a 1:1 mapping of Hardware to associated / attached CalibrationProcedure.
So yes, the user can select a "named calibration" but that means setting a named "CalibrationProcedure" on the "Hardware" in config, where the name is whatever the CalibrationProcedure class is named, in this PR, see the description that includes a Config, there we can see : "classinfo": "evolver.calibration.standard.calibrators.temperature.TemperatureCalibrator",
i.e. the Config is the tool for doing this, and associated UI/UX for managing / editing the config.
Regarding the actions that don't require input. Yeah I agree, these can be rolled into extant actions. There is a piece of work here that i'd like to do, that boils down to  modelling these side effects (reading values from hardware, saving state) within the "pure" model of the actions. I will do this as a next step also.


The httpPort property from server settings contains the host too.

Type | Version
Bug|4.25

Description
When the port number is 0 and it gets assigned by the OS, when accessing the httpPort property it contains the host too.
for example <IP_ADDRESS>:61809
I would expect the httpPort to contain only the the port number.
thanks a lot for the great work you doing.

fixed

Hi @Romakita ,
This is still happening on latest version of master.
I think it is because of this:  https://github.com/Romakita/ts-express-decorators/blob/b7145079ef127550add1f6a358ab75f3b9f62d33/src/common/config/services/ServerSettingsService.ts#L471-L473
The httpPort is set to be "address:port" so when I call this.settings.httpPort in $onReady it returns host+port, not only port.
Is this a bug? or it works as intended?
Thank you,
Dumitru

Sorry bad response on this issue. The httpPort can contain a host + port when the configuration is setup on ServerSettings decorator. By convention, I always store the host + port.
You can use this method: ServerSettingsService.getHttpPort(): {address, port}
See you,
Romain

I see, that is ok, I was just wandering if it'a bug or not, but if you store them together by default that is alright.


Server still doesn’t stop on database errors.

Of course it won’t if you haven’t merged the fix yet …


O(n^2) complexity in kde-frameworks/extra-cmake-modules/setup-hook.sh

Describe the bug
The file pkgs/development/libraries/kde-frameworks/extra-cmake-modules/setup-hook.sh has this function:
ecmUnseenHostPath() {
    for pkg in "${ecmHostPathSeen[@]}"
    do
        if [ "${pkg:?}" == "$1" ]
        then
            return 1
        fi
    done

    ecmHostPathSeen+=("$1")
    return 0
}

Since this is called for every dependency, we get quadratic complexity. This may cause a non-trivial slowdown - building digikam I noticed it takes a few seconds just to get to unpackPhase, but there might be other causes. It also causes huge amounts of debug output when you add a set -x to preHook (about 600 MB for digikam).
We should use an associative array for this like stdenv does (see 8d76effc178747f0c8f456fe619c1b014736a6af).
Notify maintainers
@ttuegel

We also need to do this in the qtbase setup hook for paths seen by qmake.

We also need to do this in the qtbase setup hook for paths seen by qmake.

Not stale.

Fixed in #154136.


JSON functionality

In order to be able to work with JSON in EdgeDB, we need the following minimal set of functionality implemented:

[ ] casting to json of any types (much like str); this includes object types and views.
[ ] casting from json of any scalars types
[ ] element access using [...] of json
[ ] json_array_unpack(json) -> SET OF json function that converts a json array into a set of json elements. It could return a <json>{} if the argument is not a json array.
[ ] json_array_elements(json) -> SET OF json function that converts a json array into an array of json elements. This is so that we don't have to lose the ordered structure of an array when converting.
[ ] json_object_unpack(json) -> SET OF tuple<json, json> function that converts a json object into a set of json key/value tuples. It could return a <tuple<json, json>>{} if the argument is not a json object.
[ ] json_typeof(json) -> str function that returns the name of the type for a json element.

Using the special json_ functions would allow to handle heterogeneous json data in EdgeQL, by first unpacking it and filtering based on type, etc.
Casting a JSON null into any scalar type should yield an empty set of that scalar type.
db> SELECT <int><json>'null' ?= <int>{};
[[True]]

A JSON null is a valid element of a set of json type. It is not the same as an empty set of json.
db> SELECT <json>'null' ?= <json>{};
[[False]]

Here's a different way to get an empty set of json:
db> SELECT (SELECT j := <json>'1' FILTER j = <json>'2') ?= <json>{};
[[True]]

Casting a json array into an EdgeDB array should be the same as creating an array literal from individual elements of the json array.
db> SELECT <array<int>><json>'[1,2,3]' = array_agg(<int>json_array_unnest(<json>'[1, 2, 3]'));
[[True]]

In particular this means that any null elements in the json array will make the result of a valid cast an empty set {} of arrays.
db> SELECT <array<int>><json>'[1,2,3, null]';
[[]]

We should probably change json_object_unpack signature to: json_object_unpack(json) -> SET OF tuple<str, json>  @elprans your thoughts?

re "casting to json of any types (much like str); this includes object types and views." -- is this done?


Add support for Alpine 3.12

This issue is created to track progress for the work to enable Alpine 3.12 support for .NET Core 3.1 and .NET 5.
Product and Test updates

[ ] [All teams] Do a compatibility check for the product with the new OS
[ ] [CoreCLR/CoreFX] Runtime / RID Support update https://github.com/dotnet/runtime/pull/37031
[ ] [Infra] Ensure that the correct Docker images used by teams are built in the dotnet-buildtools-prereqs (if applicable)
[ ] [Infra] Replace docker tags used in builds across repos and branches
[ ] [CLI team] Update CI legs to latest updated Dockerfile
[ ] [All teams] Update test infrastructure to support the new version
[ ] [Acquisition] Add support within nightly repos
[ ] [Acquisition] Deploy to master (GA) during next patch release cycle

Release team:

[ ]  Notify .NET Core teams regarding new OS release
[ ] Add feed for new OS release to packages.microsoft.com (if applicable)
[ ]  Update release manifest files
[ ] Create new Docker files to test the new OS
[ ] Update the supported OS content on GitHub and marketing sites.

Alpine Linux 3.12 is out!
https://twitter.com/alpinelinux/status/1266471727678337024

Please add an item to update docs

Update: We are on track to support Alpine 3.12 in our upcoming June releases, including publishing new container images.

Alpine 3.12 container images are available and the Distro is now supported with the latest Core release


end value has mixed support, consider using flex-end instead

To avoid postcss warnings, end and start can be replaced with flex-end and flex-start.
Line to be updated: https://github.com/microsoft/vscode/blob/68ca9a7f4c6ec11004294bd236244139473b0745/src/vs/editor/contrib/hover/browser/hover.css#L46

Has been changed since then


Cloud Firestore OpenSSL issue

Describe the bug
My app runs fine without cloud_firestore.
When adding it results in the following trace:
/Users/rafaellucena/Documents/personal_projects/firebase_sample/ios/Pods/gRPC-Core/src/core/lib/security/security_connector/security_connector.h:33:
               /Users/rafaellucena/Documents/personal_projects/firebase_sample/ios/Pods/gRPC-Core/src/core/tsi/ssl_transport_security.h:28:10: fatal
error:
               'openssl/x509.h' file not found
               #include <openssl/x509.h>
                        ^~~~~~~~~~~~~~~~
               1 error generated.```

Flutter Doctor:
[✓] Flutter (Channel master, 1.20.0-3.0.pre.112, on Mac OS X 10.15.5 19F101, locale en-US)
    • Flutter version 1.20.0-3.0.pre.112 at /Users/rafaellucena/flutter
    • Framework revision d30c987ceb (2 hours ago), 2020-06-30 16:50:18 -0700
    • Engine revision 2dc202d823
    • Dart version 2.9.0 (build 2.9.0-19.0.dev 22da8934ac)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/rafaellucena/Library/Android/sdk
    • Platform android-29, build-tools 29.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.5)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.5, Build version 11E608c
    • CocoaPods version 1.8.4

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 47.0.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.46.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.12.1

[✓] Connected device (2 available)
    • sdk gphone x86 (mobile) • emulator-5554                        • android-x86 • Android 10 (API 29) (emulator)
    • iPhone 11 (mobile)      • 4BBFF3D7-C8DE-4DFB-8DC0-7099E8F07A36 • ios         • com.apple.CoreSimulator.SimRuntime.iOS-13-5 (simulator)

• No issues found!

Any ideas on what might be happening?

Hi @arctouch-rafaellucena
Go to ios folder of your project, delete pod files and then
execute  following from root of your iOS folder
pod init
pod install
then `flutter un --verbose
if the problem persist after this,  can you please provide your flutter run --verbose and a complete reproducible minimal code sample
Thank you

Actually, this was a Cocoapods version problem. I was using 1.8.4. Once I updated to 1.9.3 it works beautifully. Thanks for your response anyway @TahaTesser ☺.
This issue can be closed.

Hi @rafellk
given your last message I feel safe to close this issue, if you disagree please write in the comments and I will reopen it.
Thank you


Create release.yml

After this is merged, you add a tag v0.0.1 to this commit, and push. Github action will auto release the JSON.

Hi @linonetwo
If I create a tag locally and push it, the workflow fails

@linonetwo I figured it out ;) you can add the plugin to CPL now! Thank you very much!


[RFC] : Boosting OpenSearch Vector Engine Performance using GPUs

Introduction
As the popularity and adoption of generative AI continues to grow, the demand for high-performance vector search databases has skyrocketed. Customers are now dealing with vector datasets that have scaled from hundreds of millions to tens of billions of vectors, and they require these vector databases to handle continuous updates and queries at massive scale. Building vector indices at such massive scales takes significant time and  it is imperative that we significantly improve the performance and scalability of the OpenSearch Vector Engine to handle such workloads.
This document outlines a strategic plan to leverage GPU acceleration to dramatically boost the indexing performance of the OpenSearch vector engine. By harnessing the massive parallelism of GPUs, we aim to achieve faster index build times compared to CPU-based solutions, all while maintaining a compelling price-performance ratio. This will enable OpenSearch to meet the growing demands of our customers.
The key elements of this RFC include:

Offloading vector index building to a GPU-powered remote fleet that can leverage the latest advancements in GPU-accelerated vector indexing algorithms
Enabling seamless integration of GPU-built indices with the CPU-based OpenSearch vector engine for search and query serving

By implementing this GPU-powered approach, we will significantly enhance the indexing capabilities of the OpenSearch vector engine and better position it to meet the evolving needs of our customers in the generative AI era.
Opportunity
Index Build Acceleration
In 2024, the Vector Engine team made significant investments to improve the performance of the OpenSearch vector engine. This includes adding AVX512 SIMD support, implementing segment replication, transitioning to the more efficient KNNVectorsFormat, and employing iterative graph builds during merges to reduce memory footprint, greedy graph builds. These optimizations have greatly improved the overall performance and stability of the vector engine.
However, we believe we have now reached the limits of what can be achieved with CPU-based hardware alone. The next logical step is to leverage GPUs and other purpose-built hardware to further accelerate vector indexing. Our experiments with Nvidia's GPU-accelerated vector indexing algorithm, CAGRA, have shown very promising results - achieving 10-30x faster index build times compared to CPU-based solutions, all while maintaining a compelling price-performance ratio. A snapshot of the results are added in Appendix C.
The bar graph illustrates the substantial percentage improvements in index build time achieved by leveraging GPU-based infrastructure compared to CPU-based solutions for different datasets.

These performance gains are critical as our customers continue to scale their vector workloads into the billions of vectors. Faster indexing will allow them to more efficiently build and manage their vector indices.
Feasibility Study
To ensure full compatibility of GPU-created indices with CPU machines, multiple experiments were conducted. One crucial aspect that was validated is the ability to search a Faiss Index built on a GPU machine(with CAGRA algorithm) using a CPU machine. The performance benchmarks, as mentioned in the previous section, were set up in a manner where the CAGRA index was built on a GPU with Faiss, converted to a CPU-compatible index using Faiss, and then searched using Faiss. This proves that we can build the Faiss Index on a GPU machine and search it on a CPU machine. The CAGRA graph, before serialization, will be converted to an HNSW-based graph (IndexHNSWCagra), which can be searched on a CPU. This conversion is supported internally by Faiss and is maintained and contributed by the Nvidia team. Benchmarking code and code to convert a Cagra index to HNSW index is present in repo named VectorSearchForge
Proposed Architecture
Below is the high level simplified flow for improved vector index build. The overall flow goes like this:

Vectors are ingested in OpenSearch Vector Engine via standard bulk apis and segments will start to get created and stored on the disk.
During the flush/merges if Vector Engine Recognize that rather than building the index on current data node, handing it over the index build service is more beneficial, vector engine will stream/uploads the vectors data to an object store/intermediate store.
Once the data is uploaded Vector Engine will call the createIndex api of IndexBuildService(new component).
IndexBuildService based on the details of createIndexRequest will choose the right worker/instance for building the index and will trigger the index build and return a token back to Vector Engine. (More details related to this step is provided in later sections)
Once the index is created it will be uploaded to remote store (Refer below sections on how we can reduce the size of the index) by IndexBuildService.
Vector Engine will be notified back with details on where the index is stored and few other details.(Refer the below sections about more details on this interaction)
Vector Engine will then download the index from the remote store and store it with other segment files in Lucene Directory.

Note: This is an oversimplified architecture and provides a 10K ft. view of the overall system.
Components Definition

Vector Index Build Component: This is new component that is being proposed as part of this design. It  will be responsible for building a vector index for a segment and then notifying Vector Engine that index is ready to download. This index build component can have both GPU and CPU machines or any customer hardware to build the vector index.
Opensearch Vector Engine(k-NN plugin): This nothing but k-NN plugin which is responsible for providing vector related capabilities in Opensearch.
Object Store/Intermediate Vector Store: This an intermediate storage component which will temporarily store the vectors and index for different components. The store will ensure that it has right deletion polices to remove these vectors. This is not the same store where we store the segments for Remote store feature in Opensearch. This will be a separate store.

Pros:

Decoupled Architecture: With the decoupled architecture the index build component can evolve separately and new algorithms/machines can be added without impacting the overall system. There will also be no coupling between OS version and Vector Index Build Component version.
Better GPU node usages: The GPU nodes will only be building the vector index and will not be indexing any text data.
Better Error Handling: Since we are uploading the data to an intermediate store even if a GPU node crashes in between we don’t need to again transfer the data from data nodes to GPU nodes.
Multi-Tenancy: Since the fleet is outside of the cluster, multiple clusters can use the same Vector Index build service.

Cons:

May be Higher Upload Times: The overall benefits might get reduced(actual number still needs to be found) as vectors first needs to be uploaded to object store. But this still debatable as with object stores we can parallelize the uploads. Refer FAQ section for details on the upload times

Alternatives Considered
Alternative1: Hosting IndexBuild Service on GPU based Data Node Instances in OpenSearch Cluster
The idea here is rather than having a separate service/component outside of OpenSearch cluster, have a GPU based node in the cluster and can be used to build the index. On exploring further on this idea we can have GPU based instances in 2 forms:

GPU based data node doing only Indexing: With IndexReaders and Writers separation feature/project in OpenSearch, we can very well have create GPU machines as Indexing nodes.

Pros:

No intermediate Store: One of the biggest advantage of such architecture we don’t need to transfer vectors back and forth between OpenSearch cluster and Index Build Service.

Cons:

GPU machine indexing non vector data: Since this node will also be indexing the non vector fields(in OpenSearch we can have a vector field and a text field in same index), GPU machines doing indexing will not be economical. (we know similar price GPU machine has both low memory and vCPUs as compared to CPU machines with same price. Refer Appendix D.
Price Performance: Since OpenSearch will be running on machine, which will take CPU resources and also RAM to run the OpenSearch cluster, which will further make this approach not economical.

A separate node role which builds the k-NN index: In this approach we will still have the GPU nodes running OpenSearch process on it, but the node will not have any shards/index writer capability in it. The node will have a separate role lets say vector-index-builder, where data nodes will transfer their vectors(similar to original recommended solution). The node will need to build and run a very simple  OpenSearch process

Pros:

Since the GPU node is also part of the cluster no separate security controls/encryption is needed to be built.

Cons:

Low Price Performance: Since the node is in the cluster, and once ingestion is completed the node is no longer needed. But it will still remain in the cluster and cannot be shared across the different clusters. We saw similar thing with ML node, and at end we switched to using remote models to do the inference as that is more scaleable
Maintenance Overhead: This will also require building a trimmed version of OpenSearch which might not be useful, since for every feature we need to see if we want to add it in the trimmed version or not.

Alternative2: Using Node to Node communication to transfer the vectors from data node to IndexBuildService
The idea here is rather than using an intermediate storage in between we directly transfer the vectors from data nodes to GPU machine to index build service where the actual index build will happen. This will completely remove building the support for different remote stores.
Pros:

As mentioned above since there is no remote store users can save cost of the remote stores and from code standpoint it becomes simple to manage as Vector Engine doesn’t need to support different remote stores.

Cons

Data loss: If a GPU machine fails, all data must be resent from the data node, wasting time and resources. Since the data is getting transferred to GPU machines if that machine is no longer active the whole data needs to be transferred again from data node to GPU machine.
Tight Coupling:  Vector engine would become overloaded and too tightly coupled by handling all the decision-making for graph building. Decisions like which node to use for graph building, which tenant needs to be prioritized

Vector Index Build Component
Below is a sneak peek into the Vector Index build node and it components. This node/nodes will be responsible for building the Vector Index. The Vector Engine will calling/interacting with these nodes to build the index. Opensearch Project will be providing a docker image containing different components that can be  deployed into on-prem/cloud to create/run/manage the Vector Index Build Service. Below is a very high level diagram and flow for how create Index api flow will happen.

Components Definition
Below is high level responsibilities and purpose of the components added in the above diagram. This is still a high level roles and responsibilities. During the detailed low level design we will further explore and cement it.

API Handlers: A thin layer REST/gRPC to handle the incoming requests. There will be no auth/authz happening here. Users who are hosting this service can add a side car container to handle the auth/authz.
Index Builder: This component will be responsible for triggering the creation of the index. The component will first load the vectors in memory and then pass that to Faiss for Index build. Once the index is build and converted to CPU based index it will then trigger the upload.
Vector Index Data Accessor: The accessor component will be a thin abstraction between the Builder and the remote client. The component will handle trigger the downloads(if required in parallel) using the remote store client.
Remote Store Client: A wrapper over different remote store clients that can be used to download the vectors data.

Pros:

Reproducible Setup: The main benefit of having a image is, the setup becomes super easy and can be deployed on different machines in the cloud/on-prem.
Efficient Resource Management: With containerized setups a user can run multiple containers on a machine with more resources which can help provide isolation if needed between different index build processes.
Ease of Development and upgrades: Since the whole env is containerized, the development and upgrades becomes easy as there are already multiple tools like Kubernetes etc that can easily do the upgrades.
Side Car Architecture: With containerized environment, user can plugin in any side car container for security, metrics of their choice without making any changes in the main container.

Cons:

The learning curve for accessing the GPUs via container could be a problem. We might have to use GPU vendor(in this case Nvidia) specific images as base images. But with documentation and proper developer guide this can easily be overcome.

Alternative Considered
The below alternatives are mainly from distribution of the above software standpoint. The discussion on the internal components will happen as part of a separate RFC.
Alternative 1: Provide the whole software as an zip/tar rather than a container
The idea here is rather than containerizing the service, we can actually give direct .zip/tars containing the software just like Opensearch tar.gz.
Pros:

There will be no learning curve for containers setup with GPUs.

Cons:

All the pros mentioned in the recommended approach will be a con for this approach.

Next steps
Overall the high level design is divided among various RFCs/issues. The division will look like this:

Remote Vector Index Build Feature in OpenSearch Vector Engine: This will be a generic capability in OpenSearch Vector engine which will allow Vector Engine to take the advantage of a remote fleet to build the vector index. The capability and interfaces will be used as building blocks for integrating GPUs based Vector Index builds. REF: https://github.com/opensearch-project/k-NN/issues/2294
Remote Store and Vector Index Build Component Integration Interfaces: This RFC will add the details on how Vector Engine will be integrating with Remote store and also with Vector Index Build Component. The RFC will talk about the Remote Index Build Client as a first class citizen to integrate with vector index build component.
Vector Index Build Component: In this RFC we talked on very high level about the create index api for vector index build service. But this is not the only API that could be exposed. In the upcoming RFC more details about other APIs will be added, hosting framework for the APIs, multiple internal components that are just added in above diagram but haven’t been explained in much details.

Security
Since there are multiple moving pieces in the design, details on the security aspect/certification will be covered with each component level design/RFC.
FAQs
Why we are focusing on improving the indexing time and not necessarily the search time?
The acceleration provided by GPUs specially comes when you have high number of vectors either for indexing or search. Since for indexing always we buffer the vectors in segments and then do a graph creation indexing can take huge advantage of GPUs.
But if we look at the Search use-case we just get 1 query vector at a time. Plus there is no capability in Opensearch currently where we can batch/buffer the vectors for doing the query at segment/shard level. So at-least initially we want to introduce GPU acceleration with indexing. We also believe that when the use case of batch vector search query will come in that case we can start leveraging the GPUs for search.
Why we choose GPUs rather than a purpose built vector search hardware?
One of the biggest reason around this was Nvidia’s hardware is popular and already available in most of the clouds. Another thing is the k-NN algorithm of Nvidia is Opensource and compatible with CPUs and Faiss.
How a GPU based Faiss Index(built using CAGRA) can be searched on CPU machine?
The CAGRA graph cannot be searched on a CPU machine, this is where when the index will be built on the GPU machine, before doing the serialization, the IndexBuild Service convert the Cagra Index to a IndexHNSWCagra, which can be searched on CPU.
Does CAGRA supports fp16, Byte quantization along with Binary Index for disk based vector search?
We had a discussion with Nvidia team and got the confirmation that Cagra support fp16 and byte quantization. The Binary Index is also present with Cagra. But of Tuesday, November 5 the support is only present in the raft library but not exposed via Faiss. Nvidia team will be working on enabling the support in Faiss for these quantizations.
How much time will be taken to upload the Vectors to remote store?
As per writing of this RFC, actual upload times have not been measured, but if we look at the raw numbers in terms of build time improvement (refer in Appendix C) for 30GB raw vector (openai 1536D) vector, we see a difference of close to 90mins. So even if we spend like 5-10mins in uploads and downloads(~250 Megabit/sec) we would still be saving huge amount of time. Also, when index is built on the GPU, only the HNSW structure will be uploaded to S3, this will further reduce the total upload time of the index(ref this code on how we will do this).
Appendix
Appendix A: Reference Documents

Overall Initiative to improve the index build time: https://github.com/opensearch-project/k-NN/issues/1599
Serialize Faiss Index build with HSNW structure only: https://github.com/facebookresearch/faiss/pull/3487
Carga Algorithm: https://developer.nvidia.com/blog/accelerating-vector-search-fine-tuning-gpu-index-algorithms/#cagra
Library Vending out the Cagra algorithm via Faiss: https://github.com/rapidsai/cuvs
Benchmarking code: https://github.com/navneet1v/VectorSearchForge
Meta issue: https://github.com/opensearch-project/k-NN/issues/2295

Appendix B
Current Architecture
The OpenSearch k-NN plugin supports three different types of engines to perform Approximate Nearest Neighbor (ANN) search: Lucene(Java Implementation), Faiss(C++ implementation) and Nmslib(C++ implementation). These engines serve as an abstraction layer over the downstream libraries used to implement the nearest neighbor search functionality.
Specifically, the plugin supports the following algorithms within each engine:

Lucene Engine: HNSW algorithm
Nmslib Engine (Native): HNSW algorithm
Faiss Engine (Native): HNSW and IVF algorithms

At a high level, OpenSearch index data is stored in shards, which are essentially Lucene indices. Each shard is further divided into immutable segments, created during the indexing process.
For indices with k-NN fields, the k-NN plugin leverages the same underlying Lucene architecture. During segment creation, in addition to the standard Lucene data structures (e.g., FST, BKDs, DocValues), the plugin also generates the necessary vector-specific data structures for each vector field. These vector-related files are then written to disk as part of the segment files and managed by Lucene.
When performing ANN search, the plugin loads the relevant vector data structures from disk into memory (but not the JVM heap) and uses the respective library (Lucene, Faiss, or Nmslib) to execute the search operation.

Appendix C
Comparison between g5.2xlarge and r6g.2xlarge
By maintaining similar recall and search time below table provides the index build time improvement numbers

Dataset Name
Dimension
Number of Vectors
CPU Build Time (sec)
GPU Build Time (sec)
%age improvement s with GPU in Build Time (CPU/GPU)

sift-128
128
1M
490.42
16.61
30

ms-marco-384
384
1M
1098.01
35.68
31

cohere-768-ip
768
1M
3251.76
67.01
49

gist
960
1M
3146.3
68.64
46

open-ai
1536
5M
5647.2
181.17
32

bigann-10M
128
10M
9489.49
226.81
42

Legends:
CPU Build Time: Total time in sec required to build + serialize the graph on CPU
GPU Build Time: Total time in sec required to build the graph on GPU + converting to CPU compatible graph + serialization of graph
Observations

For similar recalls and almost same overall search latency are able to get 30x-50x faster build times with GPUs, which are just 3x extra price. Ref price here: Appendix D
The conversion time from GPU based index to CPU based index is similar to what we observe with pure CPU based indices.
Increasing the compression rate reduces the build time but the return diminishes after we cross 16x compression rate. This is observed across all the different datasets.

Appendix D
Prices for GPU and CPU machines.
EC2 machine prices

Instance Size
GPU
GPU Memory (GiB)
vCPUs
Memory (GiB)
Price

g5.2xlarge
1
24
8
32
1.212

r6g.2xlarge
0
0
8
64
0.4032

r6g.4xlarge
0
0
16
128
0.8064

r6g.8xlarge
0
0
32
256
1.6128

@navneet1v Thanks for proposal - its very exciting!
Question 1: A very important aspect of a separate index build service will be defining compatibility/contracts between the index build service and opensearch. Is there a plan for this? Will the index build service follow a release cycle of a plugin? That being said, why not come up with a very specialized node type for index build as opposed to a separate architecture? The node does not even have to be a part of the cluster - it could operate independently - something like cross cluster search.
Question 2: In future, I could see GPUs/specialized hardware being used in very high throughput search use cases. For instances, maybe we cache frequently visited vectors in HNSW graph on GPU cache or create some kind of high cardinality partition (like IVF) and search all cluster centroids in parallel on GPU. I assume we could still do this regardless of decisions we make around remote vector index service correct?
Question 3: In regards to

Why we choose GPUs rather than a purpose built vector search hardware?

I understand that Nvidia Cagra is a good first place to start, but in general, whatever we build should be generic enough so that a different org/company can plug in their own hardware and build a similar component. That being said, will the index build service be generic? Should we come up with a list of tenets to follow for remote index build service?

Question 1: A very important aspect of a separate index build service will be defining compatibility/contracts between the index build service and opensearch. Is there a plan for this?

Yes there will contracts around this. As we start to explore the details of these components in details the contract will start to formulate. Like what APIs to be exposed by an index build service, in what format the vectors will be written to object store etc.

Will the index build service follow a release cycle of a plugin?

Till this point, I don't see a strong reason to follow the release cycle. I am thinking similar to clients of Opensearch for the release of the image which will be exposed to users.

That being said, why not come up with a very specialized node type for index build as opposed to a separate architecture? The node does not even have to be a part of the cluster - it could operate independently - something like cross cluster search.

Not sure what you mean by come up with a separate node. Since we are exposing a docker image which can be deployed and maintained by distribution owners themselves, it is very well independent. It is no upto the providers how they want to host this image and use it.

Question 2: In future, I could see GPUs/specialized hardware being used in very high throughput search use cases. For instances, maybe we cache frequently visited vectors in HNSW graph on GPU cache or create some kind of high cardinality partition (like IVF) and search all cluster centroids in parallel on GPU. I assume we could still do this regardless of decisions we make around remote vector index service correct?

Yes that is correct. I see these GPU based index build nodes/images being used as stateless workers for now. They gets spun up, do their work and then peacefully goes back to the pool. Now if Opensearch providers can make them stateful by building a control plane aspects around this. But this design doesn't touch base on those things since they are specific to the usecase which a Opensearch provider want to do.

Question 3: In regards to

Why we choose GPUs rather than a purpose built vector search hardware?

I understand that Nvidia Cagra is a good first place to start, but in general, whatever we build should be generic enough so that a different org/company can plug in their own hardware and build a similar component. That being said, will the index build service be generic? Should we come up with a list of tenets to follow for remote index build service?

See the parts of k-NN plugin side will support any remote endpoint that can do vector index build and implements the APIs use by k-NN plugin like createIndex, getIndexBuildJobDetails etc(pending RFC creation). The details of these APIs will be shared in upcoming RFCs. So in that sense if tomorrow any specialized hardware comes, all one needs to do is flip the workers to use that hardware. I hope this clarifies. Ref: https://github.com/opensearch-project/k-NN/issues/2294 for plugin design


Enhancements to Localization, Server Configuration, and Code Clarity

This pull request introduces several enhancements across the project. It adds multi-language support for dynamic message display based on system language settings, improves server host configuration flexibility, and refactors code in several modules to enhance readability and maintainability. These changes aim to improve user experience, adaptability across different locales, and ensure cleaner code management.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[explore] Fix and test slice id  logging  issue

This PR fixes and tests the slice_id logging issue (https://github.com/apache/incubator-superset/issues/3311)
@mistercrunch @john-bodley
I was able to successfully test the /explore case but I had issues with the ajax call (or /explore_json) case. The error was CSRF token is missing. I might be missing something either in the login or in the process of making the request for JSON. The login call triggers the error. This is the same way other functions that needed to login did so.
If you want to dig in, the stack trace I get is :
Traceback (most recent call last):
  File "/Users/timi_fasubaa/Desktop/incubator-superset/tests/core_tests.py", line 739, in test_slice_id_is_always_logged_correctly_on_ajax_request
    self.login(username="admin")
  File "/Users/timi_fasubaa/Desktop/incubator-superset/tests/base_tests.py", line 131, in login
    self.assertIn('Welcome', resp)
AssertionError: u'Welcome' not found in u'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>400 Bad Request</title>\n
Bad Request
\n
The CSRF token is missing.
\n'
    """Fail immediately, with the given message."""
>>  raise self.failureException('u\'Welcome\' not found in u\'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\\n<title>400 Bad Request</title>\\n
Bad Request
\\n
The CSRF token is missing.
\\n\'')

see how the other tests include a self.login call.

Also looking at Travis build logs it looks like json.loads doesn't like None

I took inspiration from test_user_profile and test_fetch_datasource_metadata and what I have as far as I can see is consistent with what they do.
Strangely, when I run
nosetests tests.core_tests:CoreTests.test_user_profile
nosetests tests.core_tests:CoreTests.test_fetch_datasource_metadata

They both  error out with the same CSRF issue. Assuming both tests worked before, the fault may lie somewhere in the underlying authentication logic on my computer and not the tests. Can you please try on yours and see if the issue is still there?

Why nosetests over ./run_specific_test.sh? It sets env vars you need.

@mistercrunch  I don't know why nosetests came to mind when I was on the terminal. I tried out ./run_specific_test.sh and it worked fine.
@xrmx  I also addressed the possible issue with the case where neither is filled in. I don't know if making slice_id 0 after the exception is thrown is the best solution. Anything that doesn't log the right slice_id will fail the tests I wrote.
On line 73 in core_tests.py, I called explunge_all() to store things out of a session. I don't understand why things were out of the session in the first place (all other similar calls to get_resp() never had to call that function but after I did, it worked. Am I doing something wrong there?

Coverage decreased (-0.09%) to 69.297% when pulling 2028c787040bbc1fa61c0b3ead4c647fda9548e6 on timifasubaa:fix__slice_id__logging__issue into 6fc837db51cf4d9bafff19862da7b045e57388e8 on apache:master.

Personally I would have kept what you had but wrapped in a int(...) call per https://github.com/apache/incubator-superset/issues/3311#issuecomment-322939564.

@john-bodley I was in agreement until @xrmx 's comment above. The reason I changed it is that in the event there is not form_data and json.loads takes in None, it will throw a TypeError, which won't be caught by the existing catch block. Would you prefer I changed it to also catch ValueError in addition to Exception?

Also rebasing may help with coveralls checks

Coverage decreased (-0.07%) to 69.319% when pulling 8f5d9c872223fae3f27c4e43fd81c335f85aa74f on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

Coverage decreased (-0.07%) to 69.319% when pulling 8f5d9c872223fae3f27c4e43fd81c335f85aa74f on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

LGTM

@xrmx  You're right. I reviewed and updated it. What do you think of the current state?

Coverage decreased (-0.07%) to 69.319% when pulling 9d254a42b1130d4c2f33bccd7a893b03d894e456 on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

Coverage decreased (-0.08%) to 69.314% when pulling 9d254a42b1130d4c2f33bccd7a893b03d894e456 on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

@timifasubaa i think that you should fix issues pointed by the reviewer instead

Coverage decreased (-0.2%) to 69.238% when pulling 84d49bdcca923678aacb49c2b1f52684fb426ba1 on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

@xrmx how about now?

Coverage decreased (-0.08%) to 69.314% when pulling b4aa907ffae0df249024447233896ffb527324ad on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

Coverage decreased (-0.08%) to 69.314% when pulling c6036ac1429a656a4bf3d0edcbb2884df2a383b2 on timifasubaa:fix__slice_id__logging__issue into 1fda6f0745280a84f6b2ba293a755953a85aff03 on apache:master.

LGTM, merging