Error in logs: Strict Notice: Declaration of Cloudinary_Cloudinary_Model_Cache::save() should be compatible with Mage_Core_Model_Abstract::save() in public_html/app/code/community/Cloudinary/Cloudinary/Model/Cache.php on line 0

Hi,
as per subject: lots of occurrences of error:
Strict Notice: Declaration of Cloudinary_Cloudinary_Model_Cache::save() should be compatible with Mage_Core_Model_Abstract::save() in public_html/app/code/community/Cloudinary/Cloudinary/Model/Cache.php on line 0
Magento ver. <IP_ADDRESS> with all patches up-to-date to latest.

Hi @ProxiBlue
Could you provide additional details regarding how are you receiving the errors?
Is it during installation, or while trying to deliver images? Something else?
Also, could you please upgrade Magento to <IP_ADDRESS> at least and see if you are still getting these errors?
Let us know?

Hi,
Error is produced during active usage of site, and images are loaded by client access, so i woudl say, during delivery of images. The errors appear in system.log (magento log file)
Glancing at the error, i would say that it involves saving information into the cloudinary magento cache.
Your code:
Cloudinary_Cloudinary_Model_Cache::function save($key, $value)
core:
```Mage_Core_Model_Abstract::function save()````
Unfortunately upgrading a magento site is not a frivolous task, and requires some work, extensive testing, which my client is not currently ready for this procedure.
Anyhow, looking at the latest code (from magento <IP_ADDRESS>) via open-mage mirror, I can for sure say the issue would remain...
https://github.com/OpenMage/magento-mirror/blob/magento-1.9/app/code/core/Mage/Core/Model/Abstract.php#L302

Hi @ProxiBlue
Thanks for letting us know.
We have an open request with our dev team to look into this.
As soon as there will be any updates, we will be happy to let you know.

Thanks.
Would be great to get this resolved as soon as possible, as an urgency. Logs are being saturated and causing a large volume of logging, which essentially slows the site down, having to take the time to log out so much.

I have up on their module a long time ago.
The fact it completely breaks admin due to media folder symlinks...


CS triage effort: EF docs repos: ROPC

dotnet_EntityFramework_Docs

[ ]  Connection Strings
[ ] Connection strings and models
[ ] Scaffolding (Reverse Engineering)
[ ] Configuration File Settings
[ ] EF Core tools reference (Package Manager Console) - EF Core
[ ] EF Core tools reference (.NET CLI) - EF Core
[ ] Multi-tenancy - EF Core
[ ] Azure Cosmos DB Provider - EF Core
[ ] What's New in EF Core 5.0
[ ] Entity Framework 6 SQL Server provider based on Microsoft.Data.SqlClient
[ ] Visual Studio Releases - EF6
[ ] Applying Migrations - EF Core
[ ] Port from EF6 to EF Core - Detailed Cases
[ ] Using migrate.exe - EF6
[ ] Database Providers - EF Core
[ ] Porting from EF6 to EF Core - Porting a Code-Based Model - EF
[ ] Tools & Extensions - EF Core
[ ] DbContext Lifetime, Configuration, and Initialization - EF Core
[ ] Enum Support - EF Designer - EF6
[ ] Code First Conventions - EF6
[ ] Interceptors - EF Core
[ ] Database First - EF6
[ ] What's New in EF Core 6.0
[ ] Code First to an Existing Database - EF6
[ ] Advanced Performance Topics
[ ] Entity Framework Designer Keyboard Shortcuts - EF6
[ ] Breaking changes in EF Core 7.0 (EF7) - EF Core
[ ] Databinding with WinForms - EF6
[ ] Databinding with WPF - EF6
[ ] The Entity Framework 6 provider model - EF6
[ ] Overview of Entity Framework Core - EF Core
[ ] Upgrading from previous versions to EF Core 2 - EF Core
[ ] Get Started with Windows Forms - EF Core
[ ] Dependency resolution - EF6
[ ] Porting from EF6 to EF Core - Behavior changes
[ ] Porting from EF6 to EF Core - Porting an EDMX-Based Model - EF

CS effort for repos:
dotnet_EntityFramework_Docs
~dotnet_EntityFramework_ApiDocs~ replaced by #33669
For details see:
SFI info in Teams
ROPC:
35  Files in EntityFramework.Docs
0 in  EntityFrameworkApiDocs
See CS Dashboard per email thread.

Associated WorkItem - 295456

Assigning all three of us until we have it sorted out for who does what.

@wadepickett  how did you get  ?  I'm getting many dozens of ROPC hits.
dotnet_EntityFramework_ApiDocs r

This list completes the triage, mitigation effort tracked in #33699


Move consulcatalog provider to only use health apis

What does this PR do?
This PR drops the usage of the Catalog API within the Consul Catalog provider and switches to only using the Health API(s).  This is nicely compatible with the Connect endpoints as they live under the same API and output exactly the same types.
Motivation
The Health API with the cache query parameter set to true uses the streaming health check system.  This means that if streaming is enabled in a Consul cluster, Traefik is now effectively 'free' to run, going so far as an operator can set the refresh interval down to 1 second and likely not notice anything in their Consul cluster.
Even if streaming is disabled, this still cuts the request rate in half, meaning ~1/2 bandwidth usage on top of that which is helpful in really large clusters.
More

[x] Checked tests

@ldez @kevinpollet is the semophore failure a thing I could've caused or a flakey test?  it's hard to determine from the output (for me).

@kevinpollet for some internal research I compared the following structs:
The catalog API uses https://pkg.go.dev/github.com/hashicorp/consul/api#CatalogService this type.
For comparison both the health service and connect apis use: https://pkg.go.dev/github.com/hashicorp/consul/api#ServiceEntry
ServiceEntry has https://pkg.go.dev/github.com/hashicorp/consul/api#Node and https://pkg.go.dev/github.com/hashicorp/consul/api#AgentService types within it.
When looking at the CatalogService type, The "Service(...)" fields are all contained within AgentService, and the non-prefixed fields are identical to the Node fields.

Hello @chuckyz,
As your pull request is made from an organization repository, it is not possible for us to rebase it, and our automation will not work properly.
Can you re-open this pull request from your personal fork?

@chuckyz we opened a new pull request (#9140) and added you as a co-author, thanks again for your contribution.
Superseded by #9140


[D3.1] Conduct research on potential partners, focusing on all current Soroban projects.

[x] Conduct research on all current Soroban projects, including SCF submissions and those found in the Discord channel.
[x] Identify projects that may require interaction with an AMM.
[x] Imagine and generate proposals on how Soroswap AMM could be integrated with these projects.
[x] Write a blog post about the research findings and engage in discussions on Discord.

Check dev.to blog:  https://dev.to/esteblock/soroswap-exploring-potential-partnerships-across-the-soroban-and-stellar-ecosystems-53k


When building plugin scores, we should ignore repos that are private

sets GIT_TERMINAL_PROMPT=0 when we build plugin scores

@KrauseFx it just came to me in a vision while I was sleeping last night.

So visionary


rlp: slice-like decoder

Make a trait decoder that keeps the current slicing. This standardizes all the current slice-like decoder we have.

Modify the Deserialize trait to take a lifetime, and then create another trait DeserializeOwned using the for<'a> syntax. This is similar to the serde transition from 0.9 -> 1.0.


Replace maas systemd service with kubernetes job

also added configurable maas admin credentials

This change is

Rebased per changes in https://github.com/att-comdev/openstack-helm/pull/150

@larryrensing This was approved but for some reason never got merged.  Can you resolve the conflicts so we can merge it in?

done @alanmeadows @v1k0d3n

LGTM. @larryrensing thanks for your patience!


Improve inspection and generation of node keys

Closes https://github.com/paritytech/substrate/issues/11518

This PR wants to improve the user experience of subkey by expanding a bit the possibilities offered by Substrate's InspectNodeKeyCmd and GenerateNodeKeyCmd.

Node Key Generation.
The command has been extended to allow the user to:

write the output in binary format (added --bin flag). Default is hex string as before.

Node Key Inspection
The command has been extended to allow the user to:

read the input in binary format (added --bin flag). Default is hex string as before.
Now is also easier to parse the node key file under the network folder (when auto-generated at node startup, this is stored in raw binary format).
the file input is now optional (as it is for key generation), if the file is not passed the key is read from stdin.
some tolerance for whitespaces when input is not binary (trim front/back)

The new interface allows to more easily to use the subkey CLI together with other commands via Unix pipes.
Examples (not doing anything special... just to show the new possibilities):
#  write as raw binary to stdout  |  convert to hex string   |  read as hex string from stdin
$ subkey generate-node-key --bin | xxd -p -c 32 | subkey inspect-node-key

#  write as hex string to stdout  |  convert to binary   |  read as raw binary from stdin
$ subkey generate-node-key | xxd -p -r | subkey inspect-node-key --bin

# write as raw binary to file
$ subkey generate-node-key --file secret.out --bin

# read as raw binary from stdin
$ cat secret.out | subkey inspect-node-key --bin

# trim whitespaces from hex strings
$ echo "     07b03b7a2f4bc8e1885b2ae2a51b1fcb96178048ed0ada021f2905c75c5099d4    " | subkey inspect-node-key

Additionally the --network command has been ~removed from the inspect-node-key option~ deprecated since:

it was not used at all
it was not used because actually the network type doesn't influence the node-key (i.e. a peer id of libp2p)

@kpp thanks for your review. Your points should have been addressed

Wonderful! Can't really officially review the code since Rust is not my forte :)


Nested collapse

Describe the bug
Nesting Collapse tags leads to redundant panels in the outer Collapse
To Reproduce

See this snippet in the playground, which nests a Collapse inside another.
See that the Panel with the value Subpanel is shown in the outer Collapse as well.

Expected behavior
Only the nested subpanel should be there.

My bad, this will probably be fixed by https://github.com/heartexlabs/label-studio/pull/3754


feat(sourcecode): Serialize URI string

Context
Currently, only the component name gets serialized into the URI string, the remaining parameters ar serialized in the parameters dictionary as follows:
  - from:
      uri: timer
      parameters:
        period: "1000"
        timerName: template

While this works for the Camel CLI, CamelK doesn't support this schema at the moment, causing the deployment to fail.
This commit writes the path parameters into the URI string, in order to support CamelK deployments, f.i.:
  - from:
      uri: timer:template
      parameters:
        period: "1000"

fixes: https://github.com/KaotoIO/kaoto-next/issues/884

Awesome, thanks for catching it, I pushed a fix for a related issue

The component can be modified:

in source code, notice the resta instead of rest:
- route:
    id: route-4144
    from:
      id: from-1532
      uri: timer
      parameters:
        period: "1000"
        timerName: template
      steps:
        - log:
            id: log-3429
            message: template message
        - to:
            id: to-2923
            uri: resta
            parameters:
              uriTemplate: atemplate

surely related to previous one as it seems to be one-off shift but slightly different case:

reported issues fixed

not spotted new ones

seems all cases covered by automates test cases

Thanks for the @apupier, but now I think I spotted something else that I think it would be better to fix.
As an example, the rest component has the following syntax: rest:method:path:uriTemplate, and while method and path are required parameters, urlTemplate it is not.
At this moment, when serializing the URI, if the urlTemplate is not provided, we end up with the following URI:
uri: "rest:get:myPath:"

I think it's not ideal, as it would be preferable to ignore the last : when there's an optional parameter.
This will require a bit more love since we would need to check the upcoming parameters as well, considering that for multiple optional parameters, we could provide just the last one, in which case, we would need to serialize the entire URI up to the last parameter :scream_cat:

Af this update, the optional parameters are only written if the next value is either required or provided.


Gracefully handle network deserialization failure

What current issue(s) does this address, or what feature is it adding?
While syncing over the last few days i noticed a couple of malformed network packets resulted in exceptions when trying to deserialize them.
How did you solve this problem?
This pr checks the deserialization results before proceeding and returns otherwise.
How did you make sure your solution works?
no test, just common sense 🤔
Are there any special changes in the code that we should be aware of?
no
Please check the following, if applicable:

[ ] Did you add any tests?
[x] Did you run make lint?
[x] Did you run make test?
[x] Are you making a PR to a feature branch or development rather than master?
[x] Did you add an entry to CHANGELOG.rst? (if not, please do)

Coverage decreased (-0.05%) to 80.185% when pulling a4b98dc8ce7d30cad728a261da6ae313a6f7a03e on ixje:cleanup_network into 5cec42e3a965f522a2ff3ec559f8a936e23e671a on CityOfZion:development.


Fixes 2474: Recover from tasks panics

Summary
Previously if a task paniced for some reason, the worker routine would get 'stuck' and refuse to process anything else.  Now the task is marked as failed and the worker can continue to pick up another task.
Testing steps
modify the pkg/taskss/introspection.go  to panic sometimes or all the time:
	if rand.Int()%2 == 0 {
		panic(fmt.Errorf("You are unlucky!!"))
	  With this PR, they should always recover and mark the task as failed.   You can check the tasks table' status column to confirm they are marked as failed.
Checklist

[x] Tested with snapshotting feature disabled and pulp server URL not configured if appropriate

https://issues.redhat.com/browse/HMS-2474

/retest

/retest

merging as this does not require qe and the failing backend test is unrelated


add nginx to docker compose

@nscuro @mehab Is it good to include an example bundle?

Yeah we should have an example bundle.


Allow optimizer to be pluggable

Currently, it's only possible to override the optimizer by implementing (or patching) the built-in Asciidoctor::PDF::Optimizer class (see README). Instead, it should be possible to pass a custom class or object to the API using the :pdf_optimizer option.
The interface for the Optimizer is:
class Optimizer
  def initialize quality, compliance_level; end
  def optimize_file path; end
end

An alternative would be to allow the class to self register, similar to how the syntax highlighter adapters or converter work in core.

I'm going to create a separate issue for allowing new optimizers to be registered by name, then activated using the pdf-optimizer attribute. This issue will be repurposed to add the :pdf_optimizer option to the API.

The :pdf_optimizer option expects either a class or an object that responds to the new method.


Option 1: Cleaner support for JupyterHub + Binder

JupyterHub sets a JUPYTERHUB_SERVICE_PREFIX environment variable with
the base URL of the currently running user server. This takes into account
various factors, such as the base URL of the JupyterHub itself, named
servers (if the user has multiple servers running), etc. Binder also
sets the same environment variable, since it uses JupyterHub behind the
scenes.
jupyter-server-proxy proxies arbitrary HTTP requests sent to
$JUPYTERHUB_SERVICE_PREFIX/proxy/ to localhost:
on the server.
This transformer assumes you are running on a JupyterHub (or Binder),
and constructs the appropriate URL for vega to reach the altair server.
You can optionally pass in urlpath to override the default.
This change is backwards compatible, since passing urlpath will
still work. The default is just a lot more robust.

I've tested this on classic notebook + lab on a traditional JupyterHub, and classic notebook + lab on binder (https://mybinder.org/v2/gh/altair-viz/altair_data_server/better-hub-support).
Did I mention that I'm a huge fan of altair? Not sure if I have done that before, but I AM A HUGE FAN OF ALTAIR :)

We can also just roll this into data_server itself, since we can detect what to do based on presence of the environment variable. If you're running on JupyterHub / Binder, data_server won't actually work ever. If we merge data_server_proxied into data_server, then the same code can run on local notebooks and remote systems.

I made https://github.com/altair-viz/altair_data_server/pull/47 instead of this, and I much prefer that :)

Closing in favor of https://github.com/altair-viz/altair_data_server/pull/47


Remove redundant include

Fixes #9068

Note: for 1.1.1 as well. Please check that it's applicable.

Merged to master and 1.1.1.
1.1.1 built okay here.  The full CI will tell for sure.


Update documentation for VSAN policy

This is a follow up from https://github.com/vmware/docker-volume-vsphere/pull/561
The VSAN policy strings and how they are used and demo needs to be updated.

https://github.com/vmware/docker-volume-vsphere/blob/master/CONTRIBUTING.md
has the steps to preview the changes via mkdocs serve

Depends on #576. Will update the policy info when #576 is in

The final version of #576 actually includes examples and a link to official doc for VSAN policy strings, so closing the isue.


Add avatars and background images to organizations

Adds avatars and background image Medium associations to Organizations.
Also adds a can_filter_by :organization_id to the ProjectSerializer, for which there is already an index.
Also adds some missing model specs to Organizations (translatable, activatable) and reorganizes them a bit.
Review checklist

[ ] First, the most important one: is this PR small enough that you can actually review it? Feel free to just reject a branch if the changes are hard to review due to the length of the diff.
[ ] If there are any migrations, will they the previous version of the app work correctly after they've been run (e.g. the don't remove columns still known about by ActiveRecord).
[ ] If anything changed with regards to the public API, are those changes also documented in the apiary.apib file?
[ ] Are all the changes covered by tests? Think about any possible edge cases that might be left untested.

the hound has defeated me, i don't even know what it wants anymore.
i am vanquished.


Dependency upgrades

Upgrades dependencies to their latest versions

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


The H5 Writer callback doesn't take into account offset induced by limit_bounds

In ahcore, we use limit_bounds on slides which do not have any rois specified for it. This is especially helpful when we have whole slide images (WSI) which have a lot of redundant (unwanted background) tiles.
If we use limit_bounds, an offset is required to properly demarcate the region where we can find the tissue specimen on the WSI.
However, the H5FileImageWriter doesn't account for this currently and it breaks while writing h5 files.

See https://github.com/NKI-AI/dlup/pulls/210 for a streamlined way to address this


Cranelift: add option to use new single-pass register allocator.

In bytecodealliance/regalloc2#181, @d-sonuga added a fast single-pass algorithm option to regalloc2, in addition to its existing backtracking allocator. This produces code much more quickly, at the expense of code quality. Sometimes this tradeoff is desirable (e.g. when performing a debug build in a fast-iteration development situation, or in an initial JIT tier).
This PR adds a Cranelift option to select the RA2 algorithm, plumbs it through to a Wasmtime option, and adds the option to Wasmtime fuzzing as well.
An initial compile-time measurement in Wasmtime: spidermonkey.wasm builds in 1.383s with backtracking (existing algorithm), and 1.065s with single-pass. The resulting binary runs a simple Fibonacci benchmark in 2.060s with backtracking vs. 3.455s with single-pass.
Hence, the single-pass algorithm yields a 23% compile-time reduction, at the cost of a 67% runtime increase.
Fixes #9596.

For now at least, I think I'd prefer to keep it an opt-in default -- let's let it bake in wasmtime's continuous fuzzing for a little longer. We can always switch the default later.
@alexcrichton updated to add cargo-vet, could you rubber-stamp the new commit? Also fixed silly issues in fuzz build (which I never test beforehand because Ocaml; I should fix my setup!).


Add manifest for wskdeploy itself

Since wskdeploy can be deployed, we should provide a minimal manifest:
package:
  name: wskdeploy
  version: 0.4.0
  description: Utility for deploying and managing Apache OpenWhisk packages and projects
  license: Apache-2.0
  repositories:
    - url: http://github.com/openwhisk/openwhisk-wskdeploy
  actions:
    exec:
      location: actions/exec/exec.zip
      runtime: blackbox

Notice the exec location pointing to a generated zip file, which needs to be checked in so that it can be included by other OpenWhisk projects.
An alternative solution is to have script section with commands to generate the zip file.
Ideas?

Yes we should do our best, so users don't have to checkin a zip into git source control.
So adding a pre-deploy, and post-deploy  where user can point to a script in the repo would be good, or an inline script also works. in this case it can be something like pre-deploy: "zip -r exec.zip actions/exec/*

Hum, thinkin for this case, it the wskdeploy go code, should take a directory as location, and zip and deploy for the user
location: actions/exec/
runtime: blackbox
zip: true

This ways to address, but the key thing to keep in mind, always assume that user doesn't have build artifacts checked into git source control like compiled code, zip, or node_modules folder

Totally agree regarding the generated artifacts.
The problem with pre-deploy and post-deploy is the added complexity. Deployments can now fail because of missing commands (like go), unless we require the use of docker files. It also means docker has to run inside docker (for remote deployment), which seems to be possible but I'm not 100% sure.
The other possibility is to rely on github release to provide pre-compiled actions. This is good for dependencies and remote deployment, but not so good for local deployment.
So maybe we need both solutions. pre-deploy and post-deploy for local deployment and github release for the other cases.
How does that sound?

yeah it can be a mix, maybe @rabbah and @starpit have some thoughts around it, but for now I would say let wskdeploy zip the folder start simple, the scripting it's something user needs to handle like you said like github releases or other means.

zip action folder is already supported now.

@krook  Daniel might want to view this thread...
As David mentioned...@csantanapr @lionelvillard
See https://github.com/openwhisk/openwhisk-wskdeploy/issues/136
"Support action location to be a directory (ZIP support)" (Daisy)
Of course, we could make this explicit rather than implicit via the manifest, but do not see a downside to make a safe assumption that a directory (path) as "location" implies "all actions under".
If we need additional function to formalize an exclude mechanism, perhaps we should discuss under a specific issue with that topic?

@lionelvillard why can't we build from the ZIP (build) support that David and Vincent worked on which allows any user to build (via gradle) their own ZIP and use that as the location?

The zip support is one piece of the fully automated solution.
gradle (or other build system) is another piece. However:

wskdeploy does not support invoking external scripts (see @csantanapr pre-deploy and post-deploy proposal)
it's hard to run gradle inside a whisk action (which is what I'm trying to do).


test-consensus: off-by-one in the HFC A.hs ledger

Small diff.
FWIW, my draft PR for the more general test with varying epoch sizes, slot lengths, tx slot (and hence A era size), etc and epoch size and slot length changing at the fork, fails without this change. Even without that, I think we can demonstrate that @>=@ is wrong.
For example, countSlots 6 5 is 1. But if we're asking "how many slots (which are 1-to-1 with blocks in this test) after the block forged in slot 5 will there be before the onset of slot 6", then the number we want to is actually 0. Note that we should count neither slot 5 (that's the block of interest) nor slot 6 (that's the first slot of the next epoch).
master currently includes the transition tx in slot 2, and the A-B fork happens at the end of the first epoch. So the A era is slots 0-5, and B starts with slot 6. If we instead include the tx in slot 4, then the transition still happens after the first fork. That is wrong, since k = 2 and the tx's block is only buried under 1 block as of the onset of slot 6. After this PR, adding the tx in slot 4 correctly delays the A-B fork to after the second epoch.
My PR #2066 is hopefully imminent; as I said above, it exercises/needs this.

I've re-requested review, for two reason:

I changed the computation of the scheduled slot of the transition in the A ledger. It now supports arbitrary epoch sizes (>0). Please see the asTypeOf there and related comment.

I expanded the TestSetup type and its Arbitrary instance to generate many more test setup parameters. In particular, it randomizes when the transition transaction is included. The test therefore computes how many slots it needs to run for in order to see some B blocks. It fails if it doesn't see as many A and B blocks as it expects.

P.S. - I think we can remove pgaCustomLabelling. Do you agree? Please see diff.

(Also rebased this on latest master)

bors merge


More readable gnuplot rendering and measurement accuracy improvements

Hello,
as part of another project consisting in measuring various atomic ops latency (https://github.com/wtarreau/atomic-tests), I wrote an ugly script that measures the C2C latency by iterating over the tool for each pair of cores. It's pretty slow because it restarts the full test for each combination, and I never devoted time to refine it. A coworker just pointed me to your project which is clearly focused on producting this matrix, and as such is much faster.
However I was missing a few things that I added:

more distinguishable colors in gnuplot, allowing to spot smaller differences (thanks BTW for showing how to use the "image" mode, I didn't know about it and was producing HTML instead)
values are printed by default (as they're the important part), and now they're readable on all colors
support interleaving threads with their cores, as nowadays most CPUs are enumerated with thread0 first on all cores, then thread1 on all cores. This gets lines and columns sorted by locality in a much more visible way.
added the ability to place a machine name on the gnuplot title (it started to be confusing when multiple gnuplots were open on one screen)
added an option to pre-heat the CPU core after pinning it to leave the time to CPU freq to react and reach full frequency and avoid wrong measurements when switching cores
also add a measurement of round-trips using compare-and-swap that more closely matches what locks use and the possible fairness issues that can happen there.

I'm attaching below examples of outputs on a Ryzen7-2700X and a Xeon-W2145 of similar generation, both 8-cores, which clearly shows the differences in L3 behavior (AMD prefers it fast but short distance while intel prefers a unified but slower one).

A more modern AMD EPYC 74F3 (24-cores, 4 GHz) clearly shows the 8 distinct chiplets between which the latency is horrible, and inside which it's pretty good. It also reveals that they're connected as a torus with diagonal links, as non-adjacent and non-diagonal transfers are the worst ones. Note that the output gnuplot script contains commented out directives to ease showing large outputs by shrinking font size.

I hope you'll be interested, as these are minor changes, which would permit me to switch to using your tool by default.

This looks awesome! How did you come up with the palette (I need to make
sure it's a free one, looks like the matplotlib ones are CC0)?
I'm not sure about the warmup being necessary, the idea is to take the
minimum measurement across several samples (nsamples), so that should
already cover the warmup. Actually warmup could potentially make freq
slower, because turbo frequency gets downclocked due to thermal
constraints. I expect these latencies to be bounded more by the core
interconnects than the core frequencies.
I'm not sure this CAS setup is representative if you run this on ARM or
RISC-V. A typical lock would use compare_exchange_weak since the outer lock
loop will handle retries anyway and would use acquire semantics for the
success case and relaxed semantics for the failure case. It's a good idea
to add this benchmark.
I really should add support for using rdtsc to get a cleaner measurement.
Thanks for this!
On Mon, Dec 26, 2022 at 10:03 AM Willy Tarreau @.***>
wrote:

Hello,
as part of another project consisting in measuring various atomic ops
latency (https://github.com/wtarreau/atomic-tests), I wrote an ugly
script that measures the C2C latency by iterating over the tool for each
pair of cores. It's pretty slow because it restarts the full test for each
combination, and I never devoted time to refine it. A coworker just pointed
me to your project which is clearly focused on producting this matrix, and
as such is much faster.
However I was missing a few things that I added:

more distinguishable colors in gnuplot, allowing to spot smaller
differences (thanks BTW for showing how to use the "image" mode, I didn't
know about it and was producing HTML instead)
values are printed by default (as they're the important part), and
now they're readable on all colors
support interleaving threads with their cores, as nowadays most CPUs
are enumerated with thread0 first on all cores, then thread1 on all cores.
This gets lines and columns sorted by locality in a much more visible way.
added the ability to place a machine name on the gnuplot title (it
started to be confusing when multiple gnuplots were open on one screen)
added an option to pre-heat the CPU core after pinning it to leave
the time to CPU freq to react and reach full frequency and avoid wrong
measurements when switching cores
also add a measurement of round-trips using compare-and-swap that
more closely matches what locks use and the possible fairness issues that
can happen there.

I'm attaching below examples of outputs on a Ryzen7-2700X and a Xeon-W2145
of similar generation, both 8-cores, which clearly shows the differences in
L3 behavior (AMD prefers it fast but short distance while intel prefers a
unified but slower one).
[image: xeon-w2145]
https://user-images.githubusercontent.com/8141789/209565122-2f5c3542-f7e9-44e6-9e15-c686d6d63429.png
[image: ryzen-2700x]
https://user-images.githubusercontent.com/8141789/209565126-40543f7b-8f25-4cc8-bd55-6b84565865a6.png
A more modern AMD EPYC 74F3 (24-cores, 4 GHz) clearly shows the 8 distinct
chiplets between which the latency is horrible, and inside which it's
pretty good. It also reveals that they're connected as a torus with
diagonal links, as non-adjacent and non-diagonal transfers are the worst
ones. Note that the output gnuplot script contains commented out directives
to ease showing large outputs by shrinking font size.
[image: epyc-74f3]
https://user-images.githubusercontent.com/8141789/209565336-b2a410f0-6f77-4bd7-9471-fdbc5c8ba27c.png
I hope you'll be interested, as these are minor changes, which would
permit me to switch to using your tool by default.
You can view, comment on, or merge this pull request online at:
https://github.com/rigtorp/c2clat/pull/5
Commit Summary

e41c3d8
https://github.com/rigtorp/c2clat/pull/5/commits/e41c3d885a09dcf852aef8d493f8804ed3cbbd45
Add "notitle" to the gnuplot "plot" command
fcb16f2
https://github.com/rigtorp/c2clat/pull/5/commits/fcb16f239fdfff707e51ccf4fcea24a6322acadd
Enable labels by default in gnuplot output
8e66e5e
https://github.com/rigtorp/c2clat/pull/5/commits/8e66e5e04e59b6ebf4f18c20d118e8582cc4935d
Provide a more colorful palette in gnuplot heatmap output
e3d0f6d
https://github.com/rigtorp/c2clat/pull/5/commits/e3d0f6de393d4933b5f023dac9c0a7fedf1c134f
Support interleaving hardware threads and CPU cores
e1403b0
https://github.com/rigtorp/c2clat/pull/5/commits/e1403b08a9651fe521585ba7a37ab5e71859730e
Add indications in the gnuplot output to reduce font size
56f3aeb
https://github.com/rigtorp/c2clat/pull/5/commits/56f3aebdc07b36f02afd9a3f35de0c746c3a8d35
Support passing an optional name with "-n" for the gnuplot title
65b6c28
https://github.com/rigtorp/c2clat/pull/5/commits/65b6c28efbd21b6b6d88111e382625bde23d6d73
Also implement write-exclusive access measurements using CAS
abb6993
https://github.com/rigtorp/c2clat/pull/5/commits/abb6993da10764d06da1de3c6660e7aae4d00f0d
Add a preheating option to allow the CPU to switch frequency

File Changes
(1 file https://github.com/rigtorp/c2clat/pull/5/files)

M c2clat.cpp
https://github.com/rigtorp/c2clat/pull/5/files#diff-832639752a1f3445af04dfab2c522339704af0dd4a9c78954be81acf3e04b717
(119)

Patch Links:

https://github.com/rigtorp/c2clat/pull/5.patch
https://github.com/rigtorp/c2clat/pull/5.diff

—
Reply to this email directly, view it on GitHub
https://github.com/rigtorp/c2clat/pull/5, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABLO27AAJNHBQAWYSUWY73WPG6WHANCNFSM6AAAAAATJXYZ5I
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

This looks awesome! How did you come up with the palette (I need to make sure it's a free one, looks like the matplotlib ones are CC0)?

Oh I wasn't even aware that there were something like non-free palettes! That's one I designed for another project already (a thermal camera). I needed to have bright enough colors that cover the intuition of cold vs hot and that always permits to draw in green over it. Here we don't need green but we need to write using more readable dark colors so I adjusted it a little bit. It started from what I had done there: https://github.com/wtarreau/Odroid_Go_thermal_IR_camera/commit/a0b9bc792 and https://github.com/wtarreau/Odroid_Go_thermal_IR_camera/commit/695474e3 where I just brightened the cold parts by leaving some cyan in it so that writing in black is always visible. It's just a little bit of trial and error. BTW I didn't know gnuplot's heatmaps and was amazed by how simple it is to use thanks to its ability to interpolate values between these segments.

I'm not sure about the warmup being necessary, the idea is to take the minimum measurement across several samples (nsamples), so that should already cover the warmup.

In fact it depends on the scheduler. Running it on a laptop whose governor is set to "ondemand" makes a big difference for me. I've already faced this on my other tools in the past. Similarly some x86 CPUs running on intel_pstate can take a small time to switch to the higher frequency.

Actually warmup could potentially make freq slower, because turbo frequency gets downclocked due to thermal constraints.

Absolutely, that's also why it's optional.

I expect these latencies to be bounded more by the core interconnects than the core frequencies.

Not for L1 (important between two threads) nor L2 on some CPUs (many ARM CPUs for example can have a few cores share the L2 cache and there the difference is also noticeable due to the narrow bus width of 128 bits sometimes).

I'm not sure this CAS setup is representative if you run this on ARM or RISC-V.

In fact my concern as a developer is that it represents some use cases developers are facing. Sometimes you need a CAS, and being able to show other developers how expensive it can be is important. I've already seen situations where a workload running on two low-power cortex A53 cores executed faster than on two cores from an EPYC just because of monster latencies, and such demos are important to educate developers about the fact that it's not by using a much bigger CPU that latency will be gone, but it's the algorithm that must be adjusted not to depend too much on this latency.
I would also find it interesting to implement the DWCAS. It's not much used in field but there are huge variations between CPU families and again, it can be interesting to know that the small gains on some archs can be totally offset by the huge losses on other ones.

A typical lock would use compare_exchange_weak since the outer lock loop will handle retries anyway and would use acquire semantics for the success case and relaxed semantics for the failure case. It's a good idea to add this benchmark.

That's among the issues I'm dealing with when trying to optimize my code ;-)  At first it surprised me to see that even reads would benefit from exponential back-off, and I quickly realized that it was obvious: by attempting a read access to a cache line you're making the owner lose exclusivity, so on its next access it must obtain it again. So by reading you're slowing down the other one. And it seems that some CPUs with multi-die layouts and multiple L3 caches tend to favor their siblings more than other groups in such scenarios. I've seen situations where a core coudln't get access to a lock for seveal seconds when reading before writing (which makes CAS unusable without EBO, unless you use a first CAS to read).

I really should add support for using rdtsc to get a cleaner measurement.

Yeah it could be nice, but not much portable either. And people will run it in VMs and will report funny values. It should likely be just an option.

Thanks for this!

You're welcome, thanks for your project in the first place, it saves me from having to do it myself ;-)

On Tue, Feb 28, 2023 at 1:09 AM Willy Tarreau @.***>
wrote:

This looks awesome! How did you come up with the palette (I need to make
sure it's a free one, looks like the matplotlib ones are CC0)?
Oh I wasn't even aware that there were something like non-free palettes!

Cool, so I can merge this then.

I'm not sure about the warmup being necessary, the idea is to take the
minimum measurement across several samples (nsamples), so that should
already cover the warmup.
In fact it depends on the scheduler. Running it on a laptop whose governor
is set to "ondemand" makes a big difference for me. I've already faced this
on my other tools in the past. Similarly some x86 CPUs running on
intel_pstate can take a small time to switch to the higher frequency.

Wouldn't increasing nsamples be the same as the warmup, given that measured
value is the minimum latency?

I expect these latencies to be bounded more by the core interconnects than
the core frequencies.
Not for L1 (important between two threads) nor L2 on some CPUs (many ARM
CPUs for example can have a few cores share the L2 cache and there the
difference is also noticeable due to the narrow bus width of 128 bits
sometimes).

Have you measured this? Cap the cpufreq and measure with different caps.
I'm not sure this CAS setup is representative if you run this on ARM or

RISC-V.
In fact my concern as a developer is that it represents some use cases
developers are facing. Sometimes you need a CAS, and being able to show
other developers how expensive it can be is important. I've already seen
situations where a workload running on two low-power cortex A53 cores
executed faster than on two cores from an EPYC just because of monster
latencies, and such demos are important to educate developers about the
fact that it's not by using a much bigger CPU that latency will be gone,
but it's the algorithm that must be adjusted not to depend too much on this
latency.

Yes, but you are using compare_exchange_strong and sequential consistency,
it's the worst case, I don't think any reasonable lock would be implemented
like this.
A typical lock would use compare_exchange_weak since the outer lock loop

will handle retries anyway and would use acquire semantics for the success
case and relaxed semantics for the failure case. It's a good idea to add
this benchmark.
That's among the issues I'm dealing with when trying to optimize my code
;-) At first it surprised me to see that even reads would benefit from
exponential back-off, and I quickly realized that it was obvious: by
attempting a read access to a cache line you're making the owner lose
exclusivity, so on its next access it must obtain it again. So by reading
you're slowing down the other one. And it seems that some CPUs with
multi-die layouts and multiple L3 caches tend to favor their siblings more
than other groups in such scenarios. I've seen situations where a core
coudln't get access to a lock for seveal seconds when reading before
writing (which makes CAS unusable without EBO, unless you use a first CAS
to read).

A good scalable lock like MCS lock (https://lwn.net/Articles/590243/)
transforms the release operation into something close to a SPSC ring buffer
write. So again CAS ping pong with sequential consistency is not a very
realistic scenario.
I really should add support for using rdtsc to get a cleaner measurement.

Yeah it could be nice, but not much portable either. And people will run
it in VMs and will report funny values. It should likely be just an option.

In a typical x86 VM is rdtsc a hypercall? If not there's no issue using it?

Message ID: @.***>

Cool, so I can merge this then.

thanks!

Wouldn't increasing nsamples be the same as the warmup, given that measured value is the minimum latency?

I remember I tried, and it resulted in much longer run times on many-core machines, while the pre-heating allowed CPUs sharing the same governors to start faster. But in the absolute you're right, and ideally we should spend some time after switching between certain pairs of cores. And given that the cpufreq policies and organization change between various archs, it would be a pain to have to rely on them. Overall I think that this can be adjusted by some mini-documentation educating the users to try various approaches depending on the arch they're running on.

I expect these latencies to be bounded more by the core interconnects than the core frequencies.

Not for L1 (important between two threads) nor L2 on some CPUs (many ARM CPUs for example can have a few cores share the L2 cache and there the difference is also noticeable due to the narrow bus width of 128 bits sometimes).

Have you measured this? Cap the cpufreq and measure with different caps.

I remember I did but I must admit I don't have the numbers in mind anymore. For example on RK3399 there's something really odd, the 4 smaller cores seem to have a difficult access to the shared L2 cache. I remember that the performance remains low until at least 2 cores run at full frequency. I couldn't figure why, but there was likely something related to some form of devfreq for power savings. Sorry for not being specific, this was several years ago...

Yes, but you are using compare_exchange_strong and sequential consistency, it's the worst case, I don't think any reasonable lock would be implemented like this.

On x86 you have no other choice. Atomic ops are always sequentially consistent. That's also in part why x86 won where threads matter, they tend to hide software bugs that affect other archs.

A good scalable lock like MCS lock (https://lwn.net/Articles/590243/) transforms the release operation into something close to a SPSC ring buffer write. So again CAS ping pong with sequential consistency is not a very realistic scenario.

Oh thanks for refreshing me with that one, I had read about it back then and completely forgot! But again, it's not just locks. There are plenty of constructs involving CAS, such as queues, rate counters, max, etc. Sometimes you have no other choice and the construct doesn't make it easy. That's also why I'm very interested in exhibiting the poor behavior an architecture can exhibit so as to convince developers (myself included) when things are not going to work.

I really should add support for using rdtsc to get a cleaner measurement.

Yeah it could be nice, but not much portable either. And people will run it in VMs and will report funny values. It should likely be just an option.

In a typical x86 VM is rdtsc a hypercall? If not there's no issue using it?

It will depend on the underlying hardware and how the VM is configured. For a long time we've seen all possible variations of random stuff including clock jumping back and forth just due to the hypervisor reassigning different CPUs with them not supporting a TSC offset by then. Things have improved at least on x86, I don't know the current state on other archs supporting virtualization.

Yes, for sure, feel free to use them! Anyway by appearing here they're already public, so nothing secret there!

@rigtorp just reponding again with you CCed to make sure you received my response, since the PR is closed.

Yeah, I merged your changes!

Yes but you were asking if you could use these images for the README, which I said was OK but not seeing them appear made me suspect you didn't see my response.

I got your response, just didn't have time to update the readme yet. Thanks!
On Tue, Mar 14, 2023 at 10:22 PM Willy Tarreau @.***>
wrote:

Yes but you were asking if you could use these images for the README,
which I said was OK but not seeing them appear made me suspect you didn't
see my response.
—
Reply to this email directly, view it on GitHub
https://github.com/rigtorp/c2clat/pull/5#issuecomment-1469252846, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABLO22YGXKI5CZ6GRXL2LTW4EYY7ANCNFSM6AAAAAATJXYZ5I
.
You are receiving this because you were mentioned.Message ID:
@.***>

OK no pb :-)


It is difficult to write words when exporting, and remote control.

Every time I export, it is difficult to enter text.

Duplicate of #123


[BUG] Cant invoke superuser

env) inventree@Drop:~/src$ invoke superuser
OSError: no library called "cairo-2" was found
no library called "cairo" was found
no library called "libcairo-2" was found
cannot load library 'libcairo.so.2': libcairo.so.2: cannot open shared object file: No such file or directory
cannot load library 'libcairo.2.dylib': libcairo.2.dylib: cannot open shared object file: No such file or directory
cannot load library 'libcairo-2.dll': libcairo-2.dll: cannot open shared object file: No such file or directory
You may require some further system packages to be installed.

Hi there @MIFD670 ! Welcome to InvenTree.
In the future please use the bug report template for reporting - we need the info that is asked for there.

OSError: no library called "cairo-2" was found
no library called "cairo" was found
no library called "libcairo-2" was found

These libraries are missing from your system. As we do not know what system you use I will guess that you use something Debian / ubuntu based. The command to install the packages would be:
sudo apt-get install libcairo2-dev

If you use another distro have a look here for a general overview. For windows or Mac look here.

@MIFD670 did you follow the advice from @matmair above? Any luck?

Closing due to lack of response. @MIFD670 please ping us if you need any further assistance.


how write csv file with support utf-8  when open with excel

What's the issue you are having? Doesn't sound related to this module.

My case is :
I save data sample have text "Président" to csv file.
When I open csv with MS Excel 2010,
It display "PrÃ©sident"
Maybe It 's not issue of this module, but I have not find out way to correct it ?

Are you using the library via the CLI or as a module?

as module only

Can you try through the CLI and see if the same happens?

Also give http://stackoverflow.com/a/6488070/483616 a try.

I have tried and no luck.
Still PrÃ©sident
Simple json I tried
[
  {
    "car": "Audi",
    "price": 40000,
    "color": "Président"
  }, {
    "car": "BMW",
    "price": 35000,
    "color": "black"
  }, {
    "car": "Porsche",
    "price": 60000,
    "color": "green"
  }
]

Then run
json2csv -i input.json -f carModel,price,color -o out.csv

See my other comment above ^

Currently I resolve by iconv command for convert it.
I will try you comment link later.
Thanks

@nvcken Could you share me the solution? I got this problem too...

Response?

I have the same problem, any solution?

+1

+1
the same issue...
from this code:
var csv = json2csv({
    data: fileData,
    del: "\t"
});

fs.writeFileSync(filename + ".xls", csv);

what I found out was that, writing to a .xls file, it opens the with weird chars when opening the file directly in Excel, you would need to import data to actually see the special chars.
if I change to fs.writeFileSync(filename + ".txt", csv); it writes the special chars and as it's a text file, you ca no longer open directly in Excel ...
if you guys have a solution, and I even tried without success using inconv, please let me know... I'll do the same!


Identifiers with arguments for the verbose syntax

#5 implemented substitution identifiers with arguments for the short syntax. As soon as the feature is mature for the short syntax, we should look into implementing the feature for the verbose syntax.

Implemented in 233163b398133bf651cb54bff19090a95c74e722 and awaiting the next release


Fix compiler flags for GCC

This brings IREE closer to beeing build with GCC, see https://github.com/google/iree/issues/6#issuecomment-544168619. -Wno-strict-prototypes is supported by GCC in general, but only for C/Objective-C. Therefore, moving it to the CLANG flags.

Thanks for the review and the quick response. I will sign the CLA asap. However, I thought it might be a good idea to already start the review process on the PR.

@googlebot I signed it!

Hey Marius,
What email did you sign the cla with? The commit and pull request are authored by<EMAIL_ADDRESS>Could you please sign with that email?
Thanks!

Hey Marius,
What email did you sign the cla with? The commit and pull request are authored by<EMAIL_ADDRESS>Could you please sign with that email?
Thanks!

The CLA is signed for that email, but it is a cooperate CLA and not a personal CLA.

Could you please check through the steps for corporate signers posted by googlebot again? It should automatically switch to "cla: yes" once everything is in order, but we still see the CLA as not signed for some reason. Thanks!

I think I have done everything right, but https://cla.developers.google.com/about states "After the agreement is electronically signed, it will need to be reviewed by someone at Google. [..] This normally takes a few days.". So maybe the CLA is not yet processes, since I was informed that it was finally signed from our side last Friday.

@googlebot try again please???

(sorry about the delay on the CLA stuff @marbre - hopefully some lawyercats get to it soon!)

What's the corporate CLA you should be covered under?
I see "Fraunhofer Institute for Open Communication Systems FOKUS". Is that the right one?

@googlebot try again

@GMNGeoffrey
I was told to fill in "Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. for its Fraunhofer Institute for Material Flow and Logistics (IML)" as corporation name, but it might have got signed by the authorized signer with the corporation name "Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V." filled in.

Thanks. Sorry for the bureaucracy :-/ Please let us know when you've got an update. I'm eager to try out the cmake changes :-)

@GMNGeoffrey
No need to apologize :) I will let you know once the CLA is signed again.
Although not everything is implemented yet, I hope the changes boosts the build process with CMake. Can't wait to look deeper into IREE (probably into the C99 HAL or something like this).

@googlebot I signed it!
Finally, the CLA was signed again today :) I think it might still last some days until the CLA was checked from your side. Anyway, the contributions can hopefully land in the repo soon.

Yeah I'm still not seeing it show up on our side. We'll check back next week.
For Googlers, here's a direct link to the page listing the corporate CLAs: http://go/corpclas. Mostly so I don't have to click through a million pages to find it again.

@googlebot Rescan

Ugh sorry I ended up getting credited this commit in master :-( It's because I updated it when I merged in master.

No big deal :) This is more iree-copybara-bot fault, squashing all commits into a single one ;D

Well we're the ones who control what the bot does (mostly). Squashing everything into a single commit is the behavior we want, I think (equivalent to "squash and merge" on GitHub). I didn't realize clicking the nice "update" button on GitHub would confuse the bot. Basically the issue is that GitHub always associates pull requests with your primary email and so to avoid all our commits getting linked to our personal emails instead of our work ones, the bot takes the commit author from the HEAD commit. Anyway, we'll just avoid clicking the update button on GitHub in the future. The FR on GitHub to handle this properly is https://github.com/todogroup/gh-issues/issues/32

Ah okay, I see. Yeah, if squashing all commits of a PR is preferred that is of course okay, but somehow kills git-bisect. Anyway, I am fine with it, if that is the preferred workflow here :) I was just a little confused as I am more used to workflows where explicit squashing is mandatory and where in addition commits are rebased into branches instead of merging branches of pull requests (like in Gentoo Linux's development process).

When you say it kills git-bisect, do you mean just because commits are bigger in the master tree? How is that different than explicit squashing (I assume you mean that a PR has to be squashed before it's sent)? Personally, I'm generally a fan of small commits/PRs for identifying errors, as you mention, as well as for ease of review, merging, rollback, etc. I think it makes sense to move a PR into master as a single commit though. Usually the extra fix-up commits aren't really separate things we want to test, but maybe I'm misunderstanding or you've had a different experience. Especially here, the extra commit wasn't really an extra commit so much as me foolishly clicking a big shiny button in GitHub that I thought would do something I wanted :-D
We're generally trying to make the workflows here easy and understandable for external contributors, and part of that is to use patterns that are well-established. We'd welcome any feedback on how to improve things (including by just documenting them better in CONTRIBUTING.md). We're a little limited by our tooling as we actually have an internal source of truth and what actually happens is that the PR is imported into there, committed, and then migrated back out to GitHub. The effect is basically the same as rebasing the branch onto master (except that it looks much worse on the GitHub UI, which is on my long todo list of OSS infra improvements).
The tool we use for the copying is called copybara. The OSS version is here (I think it's basically the same as the internal one except probably doesn't include the configuration specific to Google's internal mono-repo, but haven't investigated thoroughly): https://github.com/google/copybara

Yes, that is what I essentially mean. Those big commits do not allow easy bisecting nor is it easy to see what happened when looking at the git log. With explicit squashing I don't mean to squash all the commits belonging to a single PR. Instead, small extra fix-up commits should be squashed to those commits they "fix". So either the fix-up commits get reordered and than squashed to the commits they should fix or you rebase interactively, pick the commits and fix the commit directly. Thus you never have those nasty fix-up commits in your repository. This allows you to go ahead with useful, small commits and IMHO those can be safely moved into the master. So more work for the commiter/author of the PR, but allows to profit from small (more or less self consistent) commits ;-D
So anyway, I am absolutely fine with the workflow here. I just wasn't used to squashing all commits to a single one prior to merging. Before, I just got to know the "merge all commits as is"  into the master or "clean your commits, will be merged afterwards" (as described above).

Ah I see, that makes sense. The bot workflow I think has to pull in all the commits in a PR as a single "changelist" (CL), which we then run internal tests on, etc. Also we generally try to test each thing that is committed to master, and PRs only get tested as a whole. Internally, we try to make CLs small (well, I do. Opinions vary). I think a similar goal for PRs would be good, as they're going to map to a single CL. So that means, for example, trying to separate refactoring into a different PR than a functionality change.
Of course that runs into problems if there's any dependency between your PRs. We've got decent support for it internally, but I don't know whether git has good workflows for it. I guess you can branch off your existing PR branch? I'm not sure.
I'll add more detail on all this to the contributing guide, so people know what to expect


#12615: Add queue_id output tensor to slice op, concat_bw

Ticket
Link to Github Issue #12615
Problem description
Add queue_id and optional output tensor to slice op, concat_bw
What's changed
Add queue_id and optional output tensor to slice op, concat_bw
Checklist

[x] Post commit CI https://github.com/tenstorrent/tt-metal/actions/runs/10902612202
[ ] Blackhole Post commit (if applicable)
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[x] New/Existing tests provide coverage for changes

is it just adding queue_id or optional_tensor as well?

is it just adding queue_id or optional_tensor as well?

@tarafdarTT Need both queue_id and optional_tensor.

@tarafdarTT would like a review from the DM team.

I would recommend to update the the code to handle the defaults, so that we don't need to do this... ttnn::slice(0, result, start_index, end_index, std::nullopt, std::nullopt, std::nullopt) and instead we can just have ttnn::slice(result, start_index, end_index)

sure @eyonland will do. updating the struct in slice.hpp with default values will achieve that.


Generate bindings for WABT

[x] Update the wabt project from using the old *.wtix format to a newer wit-bindgen and the *.wit format (https://github.com/wapm-packages/wabt/pull/1)
[ ] Update the vscode-wasm plugin to use wabt bindings from WAPM instead of the NPM package

This is already done.


Update WildFly Core to 2.0.0.Alpha6

Windows Build 1955 is now running using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854

Linux Build 6880 is now running using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854

Linux with security manager Build 1777 is now running using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854

Linux with security manager Build 1777 outcome was SUCCESS using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854
Summary: Tests passed: 864, ignored: 311 Build time: 0:07:57

Linux Build 6880 outcome was FAILURE using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854
Summary: Tests failed: 1 (1 new), passed: 2927, ignored: 367, muted: 1 Build time: 0:49:04
Build problems:
Failed tests detected
Failed tests
org.jboss.as.test.integration.messaging.mgmt.ConnectionFactoryManagementTestCase.testWriteDiscoveryGroupAttributeWhenConnectorIsAlreadyDefined: <no details avaliable>

Windows Build 1955 outcome was SUCCESS using a merge of 0dbec2523d4bfb06b1c7c1f223fefe6d5bb2c854
Summary: Tests passed: 2929, ignored: 367 Build time: 0:49:50


Get [panic:runtime error: index out of range] error after few days running

Steps to reproduce:
This issue seems difficult to reproduce as it might be related to data we have inside database and it appear after few days only.
I'll describe what is happening.

InfluxDB is started
We perform regular queries like this one: SELECT last("users") FROM product.r_90d.usage WHERE time > now() - 380d GROUP BY time(1d), id fill(none)
After few days running, the above query result in an error [panic:runtime error: index out of range] each time it is run.

Expected behavior:
A result is returned to the query
Actual behavior:
The following error is returned: [panic:runtime error: index out of range]
Environment info:

System info: Linux 4.15.0-50-generic x86_64
InfluxDB version: InfluxDB v1.7.10 (git: 1.7 f46f63d4e2d9684a2dd716594ab609ccd32f0a5b)
Other relevant environment details: Container runtime, disk info, etc:

Running in Kubernetes cluster, using Docker 18.06.1-ce as runtime engine.
Using official docker image from https://hub.docker.com/_/influxdb
Distribution is Ubuntu 18.04

Config:
reporting-disabled = true
bind-address = ":8088"

[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  engine = "tsm1"
  wal-dir = "/var/lib/influxdb/wal"
  max-values-per-tag = 0
  max-series-per-database = 3000000
  index-version = "tsi1"

Performance:
I have generated a pprof, but I couldn't generate it when the bug is actually happening. I don't know if my pprof will be usefull then.
I'm ready to provide any additional information. The bug will certainly happen again in few days. Let me know if there are commands to run when it happens.

@couloum thank you for the issue. If it's hard for you to reproduce, we might also have a hard time. You are running in k8s, do you know if your persistent storage is backed by ssd?

Hi @russorat . I'm using GlusterFS as storage layer. I have other InfluxDB instances using GlusterFS with no such issues. Also, this InfluxDB instance is not heavily used at all (few queries per minute maximum).
Is there any command I could run when the issue happen so that it's easier to debug?
Thanks


keystrokes on Mac

not sure if it's a bug, UX enhancement proposal or irrelevant... @oseppala @jsorva
at this point, all the environments (Win/Ubuntu/Mac) use the same key combinations for actions like "start a REPL", "execute a command in REPL" etc. and those contain "Ctrl" key. this is fine, as Mac has the "Ctrl" key too, thus I'm unsure, if that's a conventional use of it, and whether "Cmd" should be used on Mac for those key combinations. On the other hand, if those are mentioned in the course material, it might be some extra burden...

some issues with re-running the REPL discovered (on Mac): https://youtrack.jetbrains.com/issue/IDEA-137590

some issues with re-running the REPL discovered (on Mac): https://youtrack.jetbrains.com/issue/IDEA-137590

this is not actually exactly the thing, will try to solve it with a workaround (custom set keymap in settings)

the settings file lives here: https://version.aalto.fi/gitlab/denissn2/intellij_o1_keymap/

for non-standard keymaps use plugins: https://plugins.jetbrains.com/plugin/13258-macos-keymap/

for non-standard keymaps use plugins: https://plugins.jetbrains.com/plugin/13258-macos-keymap/

not all the keymaps are now fetched by default, some come as separate plugins
some keymaps might be removed from delivered settings as they the same as defaults
also left several questions-comments into the file (there are a couple of potential usage conflicts resulting from the O1 changes)
for Mac, well, I do not really have any good ideas how to replace all those 15+ combinations that have "ctrl" in them with "cmd" (meta) key, as they might a) break the UX flow, b) just be reserved to either IJ's other features or simply OS-things. So not sure, if this is really needed, as it's a very easily customizable thing. Also, some default keys for Mac contain "ctrl" too.

so now there is an issue about Scala REPL not restarting on Mac properly: https://youtrack.jetbrains.com/issue/SCL-18072
what comes to the addition of the keys as show in the O1 settings file - that worked!


Exception getting lost when calling iterator.throw on a generator that doesn't define a correponding try/catch

Uncaught exceptions are logged to console in the next method in proc.js, but are a bit difficult to track down as the logged exception isn't the original exception but the one thrown by regenerator when it tries to call throw which doesn't exist.
proc.js#L135-L154
try {
  // calling iterator.throw on a generator that doesn't define a correponding try/Catch
  // will throw an exception and jump to the catch block below
  const result = error ? iterator.throw(error) : iterator.next(arg)
  if(!result.done) {
     runEffect(result.value, parentEffectId, '', next)
  } else {
    end(result.value)
  }
} catch(error) {
  if(error instanceof SagaCancellationException) {
    end(error, false, true)
    if(isDev) {
      log('warn', `${name}: uncaught`, error)
    }
  } else {
    end(error, true)
    log('error', `${name}: uncaught`, error)
  }
}

Console displays:
proc.js:123 sagaName: uncaught TypeError: Cannot read property 'call' of undefined(…)
If I stick a breakpoint in proc.js I can see the correct exception in the error variable in the try block but iterator.throw(error) hides the exception.
I am using babel-polyfill and I believe the logged exception originates here.

 In my local tests I can see the same error thrown inside the Saga logged into the console.

Sorry for the delay. I've figured out the cause but I'm not sure if this is supported usage.
function* error() {
  throw new Error('this error is not logged correctly')
}

function* saga() {
  yield* [
    error()
  ]
}

@danwad if you call error() like that the generator is not called it's just a regular function call then. You should rather use:
function* saga() {
  yield* [
    call(error)
  ]
}

@slorber Oops - thanks. This also hides the exception.
function* error() {
  throw new Error('this error is not logged correctly')
}

function* saga() {
  yield* [
    call(error)
  ]
}

have you imported call?
Also I'm not sure but is yield* [] with an array something that actually works? (I mean with the asterisk)

yield* is used to yield to an iterable object. An array is iterable and so yield* [] is valid.
Yes, redux-saga's call is imported.

This issue is fixed in master.


Syntax highlighting with static properties in JavaScript/JSX

First of all, thanks a lot for all the time you spent working on this wonderful project.
Now here is the problem I encountered: as soon as I add a few static properties to the class body, the syntax isn't highlighted correctly. I tried using the following code and both javascript/jsx modes:
import React, { Component } from 'react';

class App extends Component {
  static propTypes = {
    title: PropTypes.string
  };

  static a = {};

  static hello() {
    return "Hello";
  }
}

You can see how the editor highlighted this code in codemirror here.
That pretty sums up the issue here, hope the mistake can easily be fixed.
Thanks again.

Class properties aren't part of the language yet, and aren't supported in the JavaScript mode. A pull request that implements the current proposal would be welcome, but I'm not going to implement them before they reach stage 4.


Import files in other folders/paths

Is there a way to import a file that is located in other path which is not the "src" folder when using testImport()? It would be very useful to edit VBA files remotely

i don't think this is possible considering the actual version of the code but it would be great


android.clients.google.com breaks ingress login

If you want to login to play Ingress, this domain is necessary to login.
Probably other applications and android will need it as well.

Hmm, it's interesting. This subdomain is blocked long time ago, and no one app was blocked. As far as I know, of course. Will try Ingress shortly :-)

Hmm, it seems, Ingress is blocked on Russia and Belarus grounds. So, the testing will be not so shortly.


fix: fix dataset select list

SUMMARY
The chart listview dataset select list before would only load the first page of datasets and would not paginate. This pr fixes the issue to allow the dataset select list to paginate on scroll
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
before
https://user-images.githubusercontent.com/17326228/122298685-eb786000-ceb1-11eb-84be-0fff360d448c.mov
after
https://user-images.githubusercontent.com/17326228/122298529-bff57580-ceb1-11eb-8e51-4b00296bf97a.mov
TESTING INSTRUCTIONS
Go to chart listview and click on dataset select and ensure that it loads all datasets.
ADDITIONAL INFORMATION

[ x] Has associated issue: https://github.com/apache/superset/issues/15131
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

fixes https://github.com/apache/superset/issues/15131

/testenv up


nrfx: add 380kbps to nrf52840_bitfields.c for twi and twim

nrfx used to be customized with this change - support for TWI/TWIM 380kbps.  This change was lost when nrfx was recently upgraded.
This PR adds the change back to nrfx.
We should find a better long term solution, but for now, let's just get the change back in.

LGTM

LGTM

any reasons to not upstream this?


Question about new build system

Hi!
I'm trying to build on macOS and looking at debian instructions I can't seems to figure out how to apply patches.

All build scripts follow the general pattern:

Setup some or all of the source tree (as necessary)
Apply patches
Build GN via tools/gn/bootstrap/bootstrap.py
Run gn gen with the GN flags
Build Chromium via ninja

In a previous build system there was apply_patches step which called quilt. I see that you are using dpkg to create debian package and I guess you are applying patches somewhere inside it, right? I would appreciate any help.
Thank you.

I see that you are using dpkg to create debian package and I guess you are applying patches somewhere inside it, right?

Yes, that's correct. The Debian build system uses quilt to apply patches. The build files generator just places the files in correct locations for Debian to use them.
To get the full set of patches that are domain-substituted, you will need to use export_resources.py. Then you can configure quilt to use the exported patch order file and patches directory, or apply the patches using any other tool you choose.

Got it, thank you!
I have a lot of ninja errors because of the missing files (those that were removed when cleaning sources) how do you run build without those files? I think they are mostly required by unittests, can we build without them?

Nevermind. Do you happen to know why it would fail on:
FAILED SOLINK libtrknotify.dylib libtrknotify.dylib.TOC

Disabled "//iridium:trknotify", and finally built Chromium 57 for macOS :)

Interesting. Do you have the error message for linking that library?

I got this one too. The problem is with is_component_build = true that I set to speed up builds. Without it - no errors.
I put up my macOS instructions in my fork so if you want I can submit a pull request.

The problem was with is_component_build = true that I set to speed up builds

Ah ok. That makes sense

I put up my macOS instructions in my fork so if you want I can submit a pull request.

Sure. I'm making some minor revisions to utilikit right now to reduce code duplication so once you submit your pull request I can make some additional changes.

Closing since macOS support is in develop


handleDownloads waits for first "download" event before creating the downloadPath folder

I posted the requested information below but I did spend some time looking into this issue and stepping through the code. The issue appears to be that the downloadPath folder is not created until the "download" event is received. That means that when I.amInPath is called, we are pointing at a non existent directory, then when I.waitForFile is called an unhanded exception occurs (which is caught) when we attempt to access this location on the file system.
Note that this is not normally an issue whenever you have a download that starts, the event seems to beat the rest of codeceptjs, so its more of a race condition. This is easy to reproduce if you comment out the call to I.click (never initiating the download). This will cause waitForFile to throw an error every time, this error is being caught at line 81 in FileSystem.js but it is not an expected error case. Once this is thrown codecept closes out the browser which then causes the waitForEvent to throw an UnhandledPromiseRejectionWarning because the page it has a handle on was closed prematurely.
Suggestion: I.handleDownloads should create the folder immediately.
What are you trying to achieve?
Test should wait for file to download and fail when it does not occur.
What do you get instead?
UnhandledPromiseRejectionWarning is thrown, the browser page is close, and codeceptjs exits after the timeout completes.

Provide console output if related. Use --verbose mode for more details.

# paste output here

Note: Swapped local path to "localpath"
(node:16972) UnhandledPromiseRejectionWarning: Error: Page closed
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.
at Page.waitForEvent (localpath\node_modules\playwright\lib\client\page.js:293:68)
at Playwright.handleDownloads (localpath\node_modules\codeceptjs\lib\helper\Playwright.js:1040:15)
at Step.run (localpath\node_modules\codeceptjs\lib\step.js:66:47)
at localpath\node_modules\codeceptjs\lib\actor.js:117:23

Provide test source code if related
Code follows the examples

I.handleDownloads('downloads/avatar.jpg');
I.click('Download Avatar');
I.amInPath('output/downloads');
I.waitForFile('downloads/avatar.jpg', 5);

// paste test

Details

CodeceptJS version: 3.0.4
NodeJS Version: 14.2.0
Operating System: Windows 10
Playwright
Configuration file:

# paste config here

+1

Be honest, did you even read the issue?
I broke it down for you in detail, and I did provide a code sample…


Configure default PolicyGroups

When you want to evaluate a group of policies, you need to first define the PolicyGroup
group = Strategize::PolicyGroup.new
group.add(Policy)
group.add(Policy)

This is too cumbersome for developers. The ability to define policy groups in one place that will be used throughout your app would greatly reduce tediousness of using the library. This should be done through the configuration of the gem.
Strategize.configure do |config|
  config.define_policy_group(:group_name) do
    policy PolicyClass
    policy PolicyClass
  end
end

When using the gem, you could get a policy group by:
Strategize.policy_group(:group_name) #=> PolicyGroup

This feature is available in version 0.0.4


Is there a way to compile Incrementally

I am working on dreamview, and when I compiled dreamview source code, it would take a long time.
Is there a way to compile Incrementally.
For example, dreamview depends on map module, so when I modify /modules/map/hdmap/hdmap_common.h, what I expected was the compiler only compile the modified part, instead of compiling the whole module and it's dependence.
It's convenient to debug. Thanks!

This is actually a limitation of bazel. When you compile a target, such as the target in the map module you mentioned, bazel will recompile all the targets that depend on that target.

I also meet the same issue , what a pity

The current compilation is indeed cached! It will only compile the modified part, so I am not sure whether your problem is like this


Restore not honoring original stream's replica

Apparently stream restore is not honoring stream's replica.
I did a stream backup from a cluster and restored into another, original stream was R3, destination stream was created R1 by natscli.
nats-cli 0.0.35

I can't seem to replicate this, can you show me backup/backup.json of your backup?

I apologize, I was really convinced that original stream was R3, it turned out to be R1.
I'm closing this issue.


[RFC] Implement Nested in the V2 Engine

Description
This proposal will outline a suggestion for the scope and implementation of the nested function in the V2 engine. This Draft PR is only meant for discussion and not to be merged.
Requirements
nested in the new engine needs to support all functionality of nested in the legacy engine. This includes:

nested in SELECT, WHERE, GROUP BY. example queries:

SELECT nested(message.info) FROM multi_nested

SELECT nested(message.*) FROM multi_nested

SELECT nested(message.info) FROM multi_nested WHERE nested('message', message.info = 'a' OR message.info = 'a')

SELECT * FROM opensearch-sql_test_index_nested_type WHERE nested(message.info)='b';

SELECT nested(message.info) FROM multi_nested GROUP BY nested(message.info)

SELECT nested(message.info) FROM multi_nested GROUP BY message.info

alias of the result should be the argument of the nested function

nested in function calls like SELECT nested(message.info) FROM multi_nested WHERE wildcardquery(nested(message.info), 'z*')

multilevel nested field queries like SELECT nested(message.author.address.street) FROM opensearch-sql_test_index_multi_nested

multiple nested functions in one query which is a limitation on elasticsearch like SELECT nested(message.author.address.street), nested(message.info), nested(message.dayOfWeek) FROM opensearch-sql_test_index_multi_nested

nested accepts strings as arguments for identifiers (This still needs to be investigated if it should not be supported). Example queries are:

SELECT nested("field.subield") FROM test;

SELECT nested("field"."subield") FROM test;

SELECT nested('field.subield') FROM test;

SELECT nested('field'.'subield') FROM test;

Limitations
If we decide to not support strings as argument, then this will become a breaking change for 3.x
Implementation
The new SQL engine implementation of nested function will be similar to the implementation of nested function in legacy. The logical plan builder implementation will be the same. Legacy’s logical plan builder is a part of NestedFieldProjection while the new engine will contain it in OpenSearchRequestBuilder. This is currently implemented in this PoC for nested in only the SELECT clause. Implementation for nested in WHERE and GROUP BY may be different.
Proof of Concept
The PoC does not satisfy all requirements for nested in SELECT. It only supports an end to end implementation of SELECT nested(message.info) FROM opensearch-sql_test_index_nested_type;. Further testing and support for more cases will be added once we are in agreement of the PoC.
SQL queries to support for each stage of implementation
// Support nested in SELECT (POC)
SELECT nested(message.info), someField FROM opensearch-sql_test_index_nested_type;

// Support for *
SELECT nested(message.*)
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@GumpacG Thanks for working on this PoC! A high level question is: can/should we implement this in some way which can  support standard PartiQL/SQL++ as well?
As you many know, nested function is legacy stuff from NLPChina codebase. PartiQL/SQL++ has more standard and clear definition in their spec. So I assume the latter is our long term goal for better user adoption and better interaction with other systems.
Here is my random thoughts on how to support PartiQL and then we can discuss if we can do the same for nested function. Take queries in your WHERE Support section for example. They will be as similar form as below by following https://partiql.org/tutorial.html#unnesting-a-nested-collection:
SELECT e.name, p.name
FROM employees AS e,
e.projects AS p
WHERE p.name LIKE '%security%'

First, AST will look like:
Project(e.name, p.name)
- Filter(Like(p.name, '%security'))
 - Join(on=true)
  - Relation(employees, e)
  - Relation(e.projects, p)

After analyzed/planned: (Q for QualifedName after analyzed. New Unnest operator is a special version of Join which represents correlation between index and its nested field):
Project(Q(e.name, STRING), Q(p.name, STRING))
- Filter(Like(Q(p.name, STRING))
 - Unnest(Q(e.projects, ARRAY), p)
  - Relation(Table(employees), e)

After optimized with both Unnest and Filter operator pushed down: (If impossible to push down, both will run in post-processing, ex. Unnest operator flat each row itself (tupleValue) by Cartesian Product e.projects (array value).)
Project(Q(e.name, STRING), Q(p.name, STRING))
- OpenSearchIndexScan(employees,
   dsl='
   "query": {
    "nested": {
      "path": "projects",
      "query": {
        "bool": {
          "must": [
            { "wildcard": { "projects.name": "%security" }}
          ]
        }
      }
    }
   }',
   output={
    e.name: stringValue(name),
    p.name: stringValue(projects.name)
   }
 )

If some solution like above makes some sense, the question is: can we treat nested function just difference in language grammar?
SELECT e.name, NESTED(projects.name)
FROM employees e,
WHERE NESTED(projects.name) LIKE '%security%'

Otherwise, I'm afraid that:

Our engine will have totally different logic for both
nested function can only work with OpenSearch because of missing clear abstraction for in-memory implementation

@dai-chen thanks for your comments. It makes sense to add PartiQL support however, it may be out of the scope of legacy deprecation. I think it can be an enhancement for after legacy deprecation. I created an issue to track this. https://github.com/opensearch-project/sql/issues/1218
Please let me know if you have any thoughts.

@dai-chen thanks for your comments. It makes sense to add PartiQL support however, it may be out of the scope of legacy deprecation. I think it can be an enhancement for after legacy deprecation. I created an issue to track this. #1218
Please let me know if you have any thoughts.

@GumpacG Sure. I was thinking it would be great if we can have single code path after logical plan. I also don't want to make it complicated at this point, so I'm good if the code changes are not too much and easy to refactor when introducing PartQL support in V2 in future.
Btw, here are 2 sample queries previously we have limitation. Could you confirm if any chance to support in current approach in V2?

Deep nested field: SELECT * FROM transaction WHERE nested(segments.userData.userZip) = '1234'
Nested field in aggregation: SELECT avg(segments.segmentTime) FROM transactions

Thanks!

We are still pretty early on in the implementation of the nested function in SELECT and do not have those limitations implemented yet but we have it in our tasks to support these queries.

We have added some logic for a naive implementation of nested in a GROUP BY clause. Revisions will be needed for nested aggregation to work alongside composite-aggregation. This should demonstrate the basic query sequence which we can accomplish the push down operation for non-aggregation nested queries and nested aggregation queries.
sequenceDiagram
    participant SQLService
    participant ParserBaseRoot
    participant AstExpressionBuilder

    participant QueryService
    participant Analyzer
    participant Unnested
    participant ExpressionAnalyzer
    participant Planner
    participant DefaultImplementor
    participant AggregationQueryBuilder

SQLService->>ParserBaseRoot:visitRoot
  ParserBaseRoot->>AstExpressionBuilder:visitScalarFunction
  AstExpressionBuilder-->>ParserBaseRoot:Function
ParserBaseRoot-->>SQLService:UnresolvedPlan

SQLService->>QueryService:analyze
  QueryService->>Analyzer:visitProject
    Analyzer->>Unnested:accept
      Unnested->>ExpressionAnalyzer:visitFunction
        ExpressionAnalyzer-->>Unnested:Function
      Unnested-->>Analyzer:LogicalNested
    Analyzer-->>QueryService:UnresolvedPlan
  QueryService->>Planner:plan
    Planner->>DefaultImplementor:visitProject
      DefaultImplementor->>AggregationQueryBuilder:buildAggregationBuilder
        AggregationQueryBuilder-->>DefaultImplementor:NestedAggregationBuilder:MetricParser
      DefaultImplementor->>OpenSearchRequestBuilder:pushDownAggregation
    DefaultImplementor-->>Planner:PhysicalPlan
  Planner-->>QueryService:PhysicalPlan
QueryService-->>SQLService:PhysicalPlan

@GumpacG Thanks for working on this PoC! A high level question is: can/should we implement this in some way which can support standard PartiQL/SQL++ as well?
As you many know, nested function is legacy stuff from NLPChina codebase. PartiQL/SQL++ has more standard and clear definition in their spec. So I assume the latter is our long term goal for better user adoption and better interaction with other systems.
Here is my random thoughts on how to support PartiQL and then we can discuss if we can do the same for nested function. Take queries in your WHERE Support section for example. They will be as similar form as below by following https://partiql.org/tutorial.html#unnesting-a-nested-collection:
SELECT e.name, p.name
FROM employees AS e,
e.projects AS p
WHERE p.name LIKE '%security%'

First, AST will look like:
Project(e.name, p.name)
- Filter(Like(p.name, '%security'))
 - Join(on=true)
  - Relation(employees, e)
  - Relation(e.projects, p)

After analyzed/planned: (Q for QualifedName after analyzed. New Unnest operator is a special version of Join which represents correlation between index and its nested field):
Project(Q(e.name, STRING), Q(p.name, STRING))
- Filter(Like(Q(p.name, STRING))
 - Unnest(Q(e.projects, ARRAY), p)
  - Relation(Table(employees), e)

After optimized with Filter operator pushed down: (During execution, Unnest operator flat each row itself (tupleValue) by Cartesian Product e.projects (array value))
Project(Q(e.name, STRING), Q(p.name, STRING))
- Filter(Like(Q(p.name, STRING)) -- Need to filter again because doc with any nested value match will return
 - Unnest(Q(e.projects, ARRAY), p) -- OpenSearch does not flatten on inner hits
  - OpenSearchIndexScan(employees,
   dsl='
   "query": {
    "nested": {
      "path": "projects",
      "query": {
        "bool": {
          "must": [
            { "wildcard": { "projects.name": "%security" }}
          ]
        }
      }
    }
   }',
   output={
    e.name: stringValue(name),
    e.projects: arrayValue(projects)
   }
  )

If some solution like above makes sense, the question is: can we treat nested function just difference in language grammar?
SELECT e.name, NESTED(projects.name)
FROM employees e,
WHERE NESTED(projects.name) LIKE '%security%'

Otherwise, I'm afraid that:

Our engine will have totally different logic for both
nested function can only work with OpenSearch because of missing clear abstraction for in-memory implementation

Thanks for this in-depth comment for an implementation that can work with PartiQL. We have updated the code with this implementation in mind. Further down the road we should be able to share this implementation with PartiQL. Please let us know what you think of the new implementation and if it was more what you had in mind.

Codecov Report

Merging #1210 (87d836a) into main (af188a3) will decrease coverage by 2.11%.
The diff coverage is 32.07%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #1210      +/-   ##
============================================
- Coverage     98.36%   96.26%   -2.11%
- Complexity     3643     3644       +1
============================================
  Files           343      348       +5
  Lines          9017     9227     +210
  Branches        585      605      +20
============================================
+ Hits           8870     8882      +12
- Misses          142      334     +192
- Partials          5       11       +6

Flag
Coverage Δ

sql-engine
96.26% <32.07%> (-2.11%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ain/java/org/opensearch/sql/analysis/Analyzer.java
98.74% <0.00%> (-1.26%)
:arrow_down:

...c/main/java/org/opensearch/sql/expression/DSL.java
99.49% <0.00%> (-0.51%)
:arrow_down:

...opensearch/sql/expression/ReferenceExpression.java
82.35% <0.00%> (-17.65%)
:arrow_down:

...org/opensearch/sql/planner/DefaultImplementor.java
96.96% <0.00%> (-3.04%)
:arrow_down:

.../opensearch/sql/planner/logical/LogicalNested.java
0.00% <0.00%> (ø)

...ch/sql/planner/logical/LogicalPlanNodeVisitor.java
95.65% <0.00%> (-4.35%)
:arrow_down:

...ch/sql/planner/optimizer/LogicalPlanOptimizer.java
100.00% <ø> (ø)

.../sql/planner/physical/PhysicalPlanNodeVisitor.java
95.00% <0.00%> (-5.00%)
:arrow_down:

...pensearch/sql/planner/physical/UnnestOperator.java
0.00% <0.00%> (ø)

.../opensearch/sql/storage/read/TableScanBuilder.java
90.00% <0.00%> (-10.00%)
:arrow_down:

... and 23 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@forestmvey Thanks for the update! Will take another look. Meanwhile I dived into the relation between PartiQL and SQL Lateral Join. Will share my thoughts for your reference shortly.

@forestmvey Thanks for the update! Will take another look. Meanwhile I dived into the relation between PartiQL and SQL Lateral Join. Will share my thoughts for your reference shortly.

https://github.com/opensearch-project/sql/issues/1104#issuecomment-1421790837

@forestmvey @MaxKsyunz I shared my recent finding for how to better support PartiQL in V2 engine here for your future reference: https://github.com/opensearch-project/sql/issues/1104#issuecomment-1421790837.
According to my investigation, considering PartiQL unnest grammar as a special case of ANSI SQL Lateral Join seems a good solution. However, one of the blocker in v2 engine is currently we only support symbol resolving from a single table in Analyzer. In the short term, I think we're good to migrate nested function in v2 first in this PR. We can discuss further later. Thanks!

@dai-chen as discussed regarding arrays of objects, we want to follow PartiQL behaviour. However, it is still unclear if we should flatten or not. With the examples below, would you be able let us know which direction we should go?
Legacy Behaviour
Legacy behaviour for the query SELECT nested(message.info) FROM nested_simple which flattens and separates values of arrays into different rows (more details in https://github.com/opensearch-project/sql/issues/1305) has the output:
"total": 6,
    "datarows": [
        [
            "a"
        ],
        [
            "b"
        ],
        [
            "c"
        ],
        [
            "c"
        ],
        [
            "a"
        ],
        [
            "zz"
        ]
    ],

With data (message is of type nested):
{"index":{"_id":"1"}}
{"message":{"info":"a","author":"e","dayOfWeek":1},"comment":{"data":"ab","likes":3},"myNum":1,"someField":"b"}
{"index":{"_id":"2"}}
{"message":{"info":"b","author":"f","dayOfWeek":2},"comment":{"data":"aa","likes":2},"myNum":2,"someField":"a"}
{"index":{"_id":"3"}}
{"message":{"info":"c","author":"g","dayOfWeek":1},"comment":{"data":"aa","likes":3},"myNum":3,"someField":"a"}
{"index":{"_id":"4"}}
{"message":[{"info":"c","author":"h","dayOfWeek":4},{"info":"a","author":"i","dayOfWeek":5}],"comment":{"data":"ab","likes":1},"myNum":4,"someField":"b"}
{"index":{"_id":"5"}}
{"message": [{"info":"zz","author":"zz","dayOfWeek":6}],"comment":{"data":["aa","bb"],"likes":10},"myNum":[3,4],"someField":"a"}

PartiQL behaviour
Test data:
{
    'test': {
        'message': <<
            {
                'info': 'a'
            },
            {
                'info': 'b'
            },
            {
                'info': 'c'
            },
            [
                {
                    'info': 'c'
                },
                {
                    'info': 'a'
                }
            ],
            [
                {
                    'info': 'zz'
                }
            ]
        >>
    }
}

Query: SELECT message.info FROM test.message as message;
Output:
<<
  {
    'info': 'a'
  },
  {
    'info': 'b'
  },
  {
    'info': 'c'
  },
  {},
  {}
>>

Query: SELECT message.* FROM test.message as message;
Output:
<<
  {
    'info': 'a'
  },
  {
    'info': 'b'
  },
  {
    'info': 'c'
  },
  {
    '_1': [
      {
        'info': 'c'
      },
      {
        'info': 'a'
      }
    ]
  },
  {
    '_1': [
      {
        'info': 'zz'
      }
    ]
  }
>>

@GumpacG Thanks for the context! I will double check.

@GumpacG I think there are problems in your data and query

Test data: In your table test.message, some rows are object while the others are array. I'm not sure if you're using catalog name test as table? Please compare with the test data provided by PartiQL.
Test query: your query is accessing each row directly, that's why it's not flatten. Please check my test below. I think our nested function is equivalent to PartiQL comma join which indicates unnest implicitly.

$ cat Tutorial/code/q2.env
{
  'hr': {
      'employeesNest': <<
         {
          'id': 3,
          'name': 'Bob Smith',
          'title': null,
          'projects': [ { 'name': 'AWS Redshift Spectrum querying' },
                        { 'name': 'AWS Redshift security' },
                        { 'name': 'AWS Aurora security' }
                      ]
          },
          {
              'id': 4,
              'name': 'Susan Smith',
              'title': 'Dev Mgr',
              'projects': []
          },
          {
              'id': 6,
              'name': 'Jane Smith',
              'title': 'Software Eng 2',
              'projects': [ { 'name': 'AWS Redshift security' } ]
          }
      >>
    }
}

$ ./bin/partiql -e Tutorial/code/q2.env

PartiQL> SELECT p.name FROM hr.employeesNest n, n.projects p;
==='
<<
  {
    'name': 'AWS Redshift Spectrum querying'
  },
  {
    'name': 'AWS Redshift security'
  },
  {
    'name': 'AWS Aurora security'
  },
  {
    'name': 'AWS Redshift security'
  }
>>
---
OK!

 I modified my PartiQL data to reflect our test data and V2 nested implementation should include flattening. Including an example here for reference. As a result https://github.com/opensearch-project/sql/issues/1305 should be closed as legacy behaviour is expected.
PartiQL data:
{
    'root': {
        'test': <<
        {
            'message': {
                'info': 'a'
            },
            'comment': {
                'numComments': 1
            }
        },
        {
            'message':  {
                'info': 'b'
            },
            'comment': {
                'numComments': 2
            }
        },
        {
            'message':  {
                'info': 'c'
            },
            'comment': {
                'numComments': 3
            }
        },
        {
            'message':  [
                {
                    'info': 'c'
                },
                {
                    'info': 'a'
                }
            ],
            'comment': {
                'numComments': 4
            }
        },
        {
            'message':  [
                {
                    'info': 'zz'
                }
            ],
            'comment': {
                'numComments': [5, 6]
            }
        }
        >>
    }
}

PartiQL query and result:
PartiQL> SELECT message.info, comment.numComments FROM root.test as a, a.message as message, a.comment as comment;
==='
<<
  {
    'info': 'a',
    'numComments': 1
  },
  {
    'info': 'b',
    'numComments': 2
  },
  {
    'info': 'c',
    'numComments': 3
  },
  {
    'info': 'c',
    'numComments': 4
  },
  {
    'info': 'a',
    'numComments': 4
  },
  {
    'info': 'zz',
    'numComments': [
      5,
      6
    ]
  }
>>
---
OK!

Closing this as the spike and POC are done.


feat(ui-shell): add platform header

Closes IBM/carbon-components-react#1412
Changelog
New

Add in initial selection of Header* components!

Changed
Removed

@asudoh saw your comment, I left a TODO block when I made this WIP PR that I think addresses most of your notes 👍

@alisonjoseph yeah! Can see the behavior described here: https://www.w3.org/TR/wai-aria-practices/examples/menubar/menubar-1/menubar-1.html#kbd1_label
I believe the reason is that if all nav items were tabbable then there would be a lot of key strokes to go through to get through that navigation. This is one of the focus algorithms that they suggested as a result. Another tiny detail is that whatever the last focus item was in the navigation will have the active tab index, as well, so when revisiting the menu the tab should be on the last focused item.

Thanks for the link! Totally makes sense especially for longer nav lists. and I noticed that it knows where you were before when you tab back! nice!


Deactivate GNSS watcher in field navigation

It is not needed for the field navigation with plants and could be the cause for an automation freezing issue

I'm not so sure about this change. The idea of the watcher is to ensure a geo fence. Why do you think it's not needed for the field navigation?

@rodja No, you mean the field_watcher. That is still active.
The gnss_watcher pauses the automation, when the gnss signal is lost, because we thought it would help with the field navigation, but it caused problems when it unpaused


added Cesium map; changed layout to match figma updated design 2

@ankyAgrawal Shall I merge it?

@ankyAgrawal I am closing this PR without merging. As we discussed earlier, we will not merge frontend-updates into SADE-UI-main as of now. We will wait until the app is in good shape and stable to merge and then create the PR again.


Create command handler

The command handler is finished. Although for some reason GitHub won't let me commit. Changes will be committed in the next few days.


Provided a simple example in the documentation to orient new users.

@natefaubion provided me with a nice neat example of using codecs that made it easy for me to get my head around the module.  I think the example might help to orient new users of the module so I've added it to the README.

This looks good; would it possible to get this change in? I too was confused when I first started looking at the Argonaut pieces (having never used the Scala original or Haskell port).

@AidanDelaney I'm sorry that this is the first time we're getting back to this -- contributors have been backlogged for quite some time, but I'm going through and taking care of older issues and PRs.
I think this is a valuable addition to the library. Quick note: the code snippet will need to be updated for 0.12, where StrMap has been deprecated in favor of a more accurate Foreign.Object type.
I'm going to leave this issue live for another week. If there hasn't been activity, then I'll merge and then perform the update to 0.12.
Thanks!

I've actually just gone ahead and updated this for 0.12 and fixed the conflicts. Thanks for adding this to the documentation!