Matter Device Type Features page

A New Features Page
Requirements:
Phase 1:

Create a new page for displaying features per Device Type.
Update Matter XML to include relevant feature data
Design the ZAP SQL database accordingly.
Translate conformance from the Matter spec to the ZAP SQL database and then to the Vue front end.

Description:
When creating Matter device types in ZAP, we propose adding a new page called "Features." This page will display all the features listed in the Element Requirements section of the Device Type Matter spec. It will also include data from the Element Requirements section, such as Conformance. This data must be extracted from the Matter data model XML, translated, stored in the ZAP SQL database, and retrieved by the front end via a REST API.
Use Case:
As a ZAP user creating a Matter application, I want to select a device type and activate features. By selecting "Features," I should see all possible features supported by the Device Type. This will provide a convenient reference for setting the featureMap attribute bit when finalizing my application.
Requirements:
Phase 2:

Implement the ability to toggle optional features on or off.
When toggling a feature, update the associated elements to correct conformance, including attributes, commands, events, features, and automatically update the feature map attribute according to the configured features.
When toggling an element, send a warning if it does not conform
Update Matter XML to include conformance of attributes and commands in CHIP repo

Description:
This page should be configurable like Attributes, Commands, and Events. It will list the features supported by each Device Type. Mandatory features should always be enabled in the GUI. Deprecated, disallowed, or provisional features should always be disabled. Optional features should be toggled on or off based on the user's application needs.
Use Case:
As a ZAP user creating a Matter application, I want to choose a device type and activate features. Selecting "Features" should show all possible features for the Device Type and allow me to toggle features. The system should automatically set the corresponding bit in the featureMap attribute, and adjust associated Attributes, Commands, Feature, etc. to ensure conformance, eliminating the need for manual configuration.
If a device type feature has complex conformance and doesn't comply after toggling, I should receive a warning and my changes will be disabled, instead of auto-updating elements for me. Changes to the same feature within a cluster and endpoint should sync across multiple device types. If I modify an attribute, command, or feature that causes incorrect conformance, I should receive warnings.
Requirements:
Phase 3:

Feature Page also include features from cluster XMLs that are not present in the device type XML. In phase 2, it only supports features defined in the device type XML.
Move the feature page UI into a per cluster view in parallel with attribute, command, and event page
Add Vue tutorial and documentation

Description:
The functionality remains the same as in Phase 2 but is now extended to include all features, not just device type features. The UI has been redesigned into a more intuitive cluster view, making it easier for users to locate, use, and compare features alongside cluster elements. Additionally, a tutorial will be provided to guide users in understanding and utilizing the project's features effectively.
Use Case:
Building on Phase 2, a ZAP user can now toggle any feature defined in the Matter specification, regardless of whether it is directly relevant to a specific device type. Additionally, the feature page has been integrated into the cluster view. User can conveniently check and compare the elements updated by toggling a feature with a single click.
Technical Details:
Database Design:
New Tables:

Create a table for Features, including fields for Feature ID, name, description, and conformance.
Create a table to link Device Types to Features, with fields for Device Type ID, Feature ID, and conformance.

Data Generation:

Alchemy Generation: Utilized the automated tool Alchemy to generate conformance data for all attributes, commands, events, and features in CHIP repo and manually fix incompatible places.

Data Transformation:

XML Extraction: Developed a parser to extract and translate conformance data from XML into conformance expressions as defined in the Matter specification.
Conformance Evaluation: Created a parser to evaluate the value of a conformance expression based on the state of its terms.

Front End Integration:

Developed Vue components to create the features page, including tables for displaying features.
Implemented toggles and controls to manage features, with real-time updates reflecting user actions.
Stored real-time element conformance data to display warnings if feature or element toggles result in incorrect conformance.
Integrated REST API calls to calculate conformance in the backend and update elements that require changes in the frontend.

Matter SDK Version Considerations:

Older Matter SDK versions has outdated XML files and lack conformance data, which could cause the feature page non-functional.
To address this, we only display the feature page and related functionalities for newer SDK versions where conformance data exist

Frontend UI of the Device Type Feature Page

Phase 2 Example Use Cases:
1. Enable HS feature in an endpoint with Extended Color Light device type

2. After enabling HS, disabling attribute CurrentHue that conforms to HS will cause conformance warnings


Agent isn't setting replica label by default

Having recently switched to this helm chart as we're building out a Mimir setup, I think I've run into an issue.
When setting the agent statefulset to run multiple replicas for HA mode, I get lots of rejected duplicate samples in the logs and the Mimir HA dashboard shows it's not recognised as a HA agent setup.
I believe Mimir by default expects the replica label to be set on all incoming metrics but that this isn't being set by default in this chart.
I did try setting __replica__: ${POD_NAME} in the externalLabels section of the chart, but this didn't seem to register correctly (it ended up sending ${POD_NAME} as the replica name which of course didn't work).
Am I missing something in the chart values for this to be setup correctly as I think it should be the default for the agent when running it in HA mode?

Hi Rob!
https://github.com/grafana/k8s-monitoring-helm/releases/tag/v0.1.15 included some fixes to the Agent clustering behavior. The statefulset that gathers metrics should properly distribute scrape jobs across the different instances, resulting in less duplicate metrics being sent to Mimir. Try with the latest release and see if you're still seeing issues.

Rolling that out and things seem a lot better!
Also very happy to see the HA side being handled by the agent itself rather than the HA reconciler needed in Mimir (that initially confused me not seeing the agent show up in the HA list on the Ingester). Great stuff!


Complying to #65 - reading parameters of different bulbs types

I went with [hex|xy]_color[_raw] and kelvin_color. Advantages:

if user knows that he wants raw value, he explicitely asks for it.
if user just want the value, we're doing the best we can; even if lightbulb has one fixed color, this color has some temperature, although it doesn't inform about it via API
we're backward compatible

Guess we're good to merge here? @balloob

Merged and pushed to pypi as 3.0.1. Thanks @matemaciek for your contribution!


Add HealthChecks for IHealthChecksBuilder

Add IHealthCheck on top of IHealthChecksBuilder. Here are some options that this could have:

YARP is healthy if all endpoints of all backends are healthy.
YARP is healthy if at least one endpoint of each backend is healthy.
Determine which backends to do the check for.

What kinds of things do you think would consume this data?
I think a lot more design would be needed to make this actionable, because you'd actually want to return a more sophisticated data payload than just pass/fail/warning.
There's a big philosophical difference between health check usage by a monitoring/alerting system VS an orchestrator:

monitors usually want some data in addition to health
orchestrators will restart your service if it's unhealthy

So the kind of health described here doesn't seem useful for an orchestrator, because it would reboot your proxy if anything goes down 😆
I can see this being useful for the monitor case, because the proxy is the keeper of knowledge about what backends exist. Determining health from the proxy's point of view would be useful to a monitoring/alerting system 👍

+1 to @rynowak's comments. I would suggest rather than having a pass/fail, we focus on what stats would a dashboard or monitoring agent want from the proxy.

RPS
Latency info

Median
Mean
P95 / P99
Queue sizes

Failure rates (something went wrong with proxying a request/response
Throughput B/s
Same for each Route & Backend

Ideally this can be pulled by listening to Eventsoure / EventCounters internally to the proxy as well as publishing the events to EventPipe for other monitoring tools.

I’d suggest looking at I.M. how to integrate with existing systems for metrics (like promethues)

Duplicate of #125

Why is this a duplicate of #125? as @rynowak said this could be useful for monitoring.

I can see this being useful for the monitor case, because the proxy is the keeper of knowledge about what backends exist. Determining health from the proxy's point of view would be useful to a monitoring/alerting system

#125 is about making sure the proxy infrastructure exposes the right APIs and data to monitor health state. That should provide all of the information needed to build something for IHealthChecksBuilder, and we'll decide at that point if a IHealthChecksBuilder based implementation should be included in-box.

Duplicate was probably the wrong term; subsumed into #125 would probably be better phrasing.

Metrics would be the most useful way to get status information (ingest to prom, w/e and create alerts off of Routes with no healthy Destinations). The same information that drives #125 would drive these metrics.
Health checks are separate from status monitoring and useful for orchestration (liveness/restart instance or readiness/route traffic) or for a quick post-deployment smoke check.
Having backends down should neither restart the proxy nor remove it from an upstream LB (if you're running multiple proxies). It should send traffic to the proxy so the proxy can return the appropriate response.

YARP is healthy if it has valid configuration and it is able to respond to requests as configured.

YARP is healthy if it has valid configuration and it is able to respond to requests as configured.

Metrics would be the most useful way to get status information (ingest to prom, w/e and create alerts off of Routes with no healthy Destinations). The same information that drives #125 would drive these metrics.
Health checks are separate from status monitoring and useful for orchestration (liveness/restart instance or readiness/route traffic) or for a quick post-deployment smoke check.
Having backends down should neither restart the proxy nor remove it from an upstream LB (if you're running multiple proxies). It should send traffic to the proxy so the proxy can return the appropriate response.

YARP is healthy if it has valid configuration and it is able to respond to requests as configured.

Is there a method where I indicate the status of the proxy? With that, we can implement an IHealthCheck that tells us if it is working or not, or if it is degraded.

What status? You can get the list of clusters and destinations, enumerate their health state, and report based on that. See IProxyStateLookup https://github.com/microsoft/reverse-proxy/blob/main/src/ReverseProxy/Management/IProxyStateLookup.cs


The prediction accuracy began to hover around 60%, and even slowly declined

Hello, I tested on my own binary data set. After simclr (batch_size=256) feature extraction, I found that my training accuracy has been hovering around 60%, and finally it will even be predicted that it is all in the same category. Is it me_ Feats error or something else, and bag_ The prediction values are very close, both in the 0.1 attachment .please point it out, thank you

hello, has this problem been solved? I also have the same problem

#44 Maybe these two threads should be put together?

I incorporated the training/testing into the same pipeline in the latest commit. You can set --eval_scheme=5-fold-cv-standalone-test which will perform a train/valid/test like this:

A standalone test set consisting of 20% samples is reserved, remaining 80% of samples are used to construct a 5-fold cross-validation.
For each fold, the best model and corresponding threshold are saved.
After the 5-fold cross-validation, 5 best models along with the corresponding optimal thresholds are obtained which are used to perform inference on the reserved test set. A final prediction for a test sample is the majority vote of the 5 models.
For a binary classification, accuracy and balanced accuracy scores are computed. For a multi-label classification, hamming loss (smaller the better) and subset accuracy are computed.

You can also simply run a 5-fold cv --eval_scheme=5-fold-cv
There were some issues with the testing script when loading pretrained weights (i.e., sometimes the weights are not fully loaded or there are missing weights, setting strict=False can reveal the problems.). The purpose of the testing script is to generate the heatmap, you should now read the performance directly from the training script. I will fix the issues in a couple of days.


Operator indefinitely reconciling CR with Deployed Helm Release due to Basic String Comparison

Bug Report
What did you do?
We have a chart which is dependent on other 4 helm charts. We have deployed all the charts with helm operator.
What did you expect to see?
The operator deployed our application well and it is working well. Also, reconciling is working well.
What did you see instead? Under which circumstances?
However, we have noticed that the operator is constantly updating our CR and some resources (.e.g secrets, network policies). We had huge secrets with our helm charts which lead to huge CR (1mb). Indefinite update of the CR brought our ETCD down, where ETCD duplicates resources if a new version of a resource comes.
Are you writing your operator in ansible, helm, or go?
helm
Possible Solution
It can be solved by implementiong deep equal for the line https://github.com/operator-framework/operator-sdk/blob/master/pkg/helm/release/manager.go#L139
Additional context
I have seen in the source code that basic string comparison is done between candidate and deployed release manifests strings. However, 'helm install --dry-run' is generating different texts where content is the same, but the order of some resources are changed. Here is comparison line https://github.com/operator-framework/operator-sdk/blob/master/pkg/helm/release/manager.go#L139

@muralov Just checking. Do your templates use any functions that generate random data?
Are you able to show some examples of this behavior from the operator logs? I was under the impression that the Helm libraries had a deterministic sort of the resources in the manifest, but I could be wrong on that.

@joelanford Our templates don't use any random data.
I have tried our helm charts with helm install --dry-run and got most of the time diffs, but sometimes not. I run 'helm template' command and I got always the same string. helm template seems to consider ordering of the resources.

@muralov Thanks for the info. This is something we should be able to fix. So just to make sure I understand, is it the order of the resources that change (e.g. sometimes a secret comes before a network policy and sometimes that same network policy comes before the secret)?

@joelanford yes, that is correct.

@muralov I started looking at the call stack and if I'm not mistaken, both helm template and the helm-operator end up using the same sorting interface implementation to sort the manifests no matter whether it's an install, update, or update dry-run.
https://github.com/helm/helm/blob/618447cbf203d147601b4b9bd7f8c37a5d39fbb4/pkg/tiller/kind_sorter.go#L122-L150
Is that non-deterministic for your chart's resources? Are you able to make a sharable example that reproduces the problem consistently?
Call stack for helm template

https://github.com/helm/helm/blob/v2.13.1/cmd/helm/template.go#L229
https://github.com/helm/helm/blob/618447cbf203d147601b4b9bd7f8c37a5d39fbb4/pkg/tiller/kind_sorter.go#L155-L156

Call stack for helm-operator (install)

https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/pkg/helm/release/manager.go#L224
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/release_install.go#L36
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/release_install.go#L87
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/release_server.go#L330
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/hooks.go#L106
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/kind_sorter.go#L96-L97

Call stack for helm-operator (update)

https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/pkg/helm/release/manager.go#L264
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/release_update.go#L39
https://github.com/operator-framework/operator-sdk/blob/c2b49e8f6a0e1bb310f59be4447211332df292fd/vendor/k8s.io/helm/pkg/tiller/release_update.go#L116

And the rest is the same as the install stack from 4-6.

@joelanford sorry for the delay. Unfortunately, I cannot share our charts because I am not allowed to share. Today I have run 'helm install --dry-run' and 'helm template' again and I have got different strings again. As I said, with dry-run the results almost each time different, but with helm template the results are always the same.
We have fixed this diff problem by splitting the whole text with '---' and compare each k8s object manifest string independently. After applying this change, the Helm Operator indefinite reconcile is stopped. Hence, the problem is really with the basic string diff.

@muralov Hmm. And you're not able to create a sharable chart that reproduces the issue?
A few more questions to see if I can narrow this down:

What version of helm are you using on the command line when you run helm install --dry-run and helm template
What version of the helm-operator are you using?
For your diff where you split and compare, what are you doing to ensure that the each list of resources is in the same order?

@joelanford I will try to create such a chart, please give me some time.
Here are my answers to your questions:

helm version is v2.12.3. Yes, I am using command line. I am also using command line diff comand.
We are using operator-sdk version: v0.8.1-dirty, commit: 33b3bfe10176f8647f5354516fff29dea42b6342
Here is code snippet of our solution, this is our fast solution:

if deployedRelease.GetManifest() != candidateRelease.GetManifest() {
  m.isUpdateRequired = true
  manifestSeparator := "---"
  candidateManifests := strings.Split(candidateRelease.GetManifest(), manifestSeparator)
  deployedManifests := strings.Split(deployedRelease.GetManifest(), manifestSeparator)
  hasDifferentManifestCounts := len(candidateManifests) != len(deployedManifests)
  m.isUpdateRequired = hasDifferentManifestCounts
  if (!hasDifferentManifestCounts) {
    for i := 0; !m.isUpdateRequired && i < len(candidateManifests); i++ {
      indexOfMatch := indexOf(candidateManifests[i], deployedManifests)
      hasNoMatch := indexOfMatch < 0
      if hasNoMatch {
        m.isUpdateRequired = true
      }
    }
}

func isEqualBitwise(text1 string, text2 string) (bool) {
  hasDifferentCharCounts := len(text1) != len(text2)
  foundDiff := hasDifferentCharCounts
  if (foundDiff) {
    return false
  }
  for pos := 0; !foundDiff && pos < len(text2); pos++ {
    hasDifferentValues := text2[pos] != text1[pos]
    if (hasDifferentValues) {
      foundDiff = true
    }
  }
  return !foundDiff
}

func indexOf(search string, texts []string) (int) {
  trimChars := "\n\r \t"
  for key, text := range texts {
    text = strings.Trim(text, trimChars)
    search = strings.Trim(search, trimChars)
    if isEqualBitwise(search, text) {
      return key
    }
  }
  return -1
}

@muralov Just wanted to follow up and see if you've been able to reproduce this with a sharable chart?

Today I have run 'helm install --dry-run' and 'helm template' again and I have got different strings again.

I didn't catch this comment of yours last time, but if you're seeing similar differences with the Helm CLI tools, then I'd suggest opening an issue against the Helm repo since that's where the fix would ultimately need to come from.

@joelanford sorry, it is not ready yet. Please give me some time.

What I mean is that based on my perusal of helm's code, I would expect helm --dry-run and helm template to produce identical release manifests. So if you're seeing a difference between those, it would explain why you're seeing the same behavior in the helm operator. The helm operator is using the same underlying libraries as the Helm CLI to generate the manifest.
So my stance on this issue is that:

If helm --dry-run and helm template are supposed to produce identical release manifests, and they aren't, that's a bug in Helm that we should work to resolve upstream.
If there is no guarantee that helm --dry-run and helm template will produce identical release manifests, that's a bug in helm-operator that should be resolved here.

If you're able to reproduce this with a shareable example, I think we should open an issue upstream to see what the helm maintainers think. WDYT?

@joelanford sorry for delay! Yes, I agree with you. I will do that soon.

@joelanford @muralov  I've noticed something similar/related as well but in my case it was due to my usage of .Release.IsInstall to run a Job only at chart install time. This caused the helm operator to see a diff with the dry run which gets templated as IsUpgrade triggering a reconciliation and removing the job before it completes.

If there is no guarantee that helm --dry-run and helm template will produce identical release manifests, that's a bug in helm-operator that should be resolved here

I would say that there is no real guarantee that both of these commands will produce identical releases at the very least because of how Helm templating can alter the manifests simply based off the types of operation. In my case for example helm template can show the same as helm install --dry-run but not helm upgrade --dry-run
Here's the chart I'm using although I had to remove the .Release.IsInstall check for compatibility with the Helm Operator. While this allows my chart to work in the interim, any changes to the job itself causes failed upgrades by the Operator since it tries to update an immutable job spec.
https://github.com/scality/Zenko/blob/development/1.1/kubernetes/cosmos/templates/rclone-cronjob.yaml#L86

@ssalaues Thanks for bringing this up. It isn't something we had considered. In the non-operator case, what would happen with your chart if it was installed, then quickly upgraded while the Job was still running (say the installer noticed a typo'd value and wanted to quickly fix it)?
Would the Job get deleted in that case?
I agree that this is a slightly different problem than the one mentioned in this issue, which has the expectation that installs and upgrades will have the same outputs given the same inputs.
What you're describing is that you expect the install and upgrade manifest to be different even when inputs are identical (based on templates that use .Release.IsInstall and .Release.IsUpgrade).
Can you think of a way to make such a scenario work with the Helm operator? It seems like those template conditionals effectively make the chart non-idempotent and therefore not really compatible with the idea of Kubernetes idempotent reconciliations.

In the non-operator case, what would happen with your chart if it was installed, then quickly upgraded while the Job was still running (say the installer noticed a typo'd value and wanted to quickly fix it)?

It would be the same behavior as you describe. I've tried to use helm hooks so it doesn't delete the resource which seemed more appropriate since there are post-install hooks. However Helm blocks/waits until all helm hooks are done to report a successful release. So long running post-install jobs just end up causing the helm operator to wait until a job finishes therefore blocking any other reconciliations.
https://github.com/helm/helm/issues/5891
But I digress.

Can you think of a way to make such a scenario work with the Helm operator?

Yeah my solution ultimately was to switch the post-install job to simply a post-install pod definition that runs to completion. This doesn't block any other Helm operations and allows the process to run only at install time.

@joelanford I see that @ssalaues have this issue too. Do you still need sample helm chart from me?

@muralov I think @ssalaues's issue is different because the release manifest changes depending on whether it's an install or an upgrade. That breaks the helm-operator's assumption that the chart is idempotent. If I understand correctly, @ssalaues did not have an issue with non-deterministic release manifest sorting.
If you have a reproducer, that would be super helpful.
As an aside, we're working on making the switch to Helm v3. I haven't seen anything about this specifically being fixed, but it's possible that the issue goes away with Helm v3. It might be worth checking to see if you can reproduce this with the Helm v3 CLI or with a build of the SDK from the branch in https://github.com/operator-framework/operator-sdk/pull/2080

@joelanford thanks for your reply! It was for me a bit difficult to reproduce, then somehow I gave up, however, I will try once more.
Let me try with Helm v3 CLI for helm --dry-run.  I will aslo try with a build of the branch 2080.

@muralov I'm going to close this for now since it's pretty stale and I haven't been able to reproduce it, but if you are able to provide a reproducer, please ping me and we can re-open!
Also, BTW, the Helm 3 PR has been merged to master and will be in the next release (probably this week).

Re-opening. We have a reproducer in #2815.
I want to investigate Helm's sorting logic to make sure it isn't inadvertently introducing the non-deterministic sorting in release manifests. If it is, we should make an upstream fix.
It is possible that a chart template could introduce the non-determinism (e.g. by ranging over a map to produce manifests or the contents of individual manifests). So a solution that compares individual manifests still won't solve this in all cases because even those can be different if the template is introducing non-determinism.
So step 1: Let's investigate Helm's sorting logic to see if there's a fix to be made there.

It looks like there was an issue in Helm's sorting logic that was causing this, but that issue was fixed in Helm v3.1.0 via helm/helm#6842.
The latest release of Operator SDK vendors Helm v3.1.2, so this issue should be fixed starting with Operator SDK v0.17.0.
To double check the fix, I used the Kasten K10 chart from #2815 and ran helm template about 10 times each with v3.0.2 (from SDK <v0.17.0) and v3.1.2 (from SDK v0.17.0) and saw that the order was inconsistent with v3.0.2 and consistent with v3.1.2.
Re-closing since this seems to be finally fixed both upstream and in Operator SDK.
The fix for this issue is to upgrade your helm-operator base image to v0.17.0.


Add accelerated f64 kernels

Implement Wasm/Neon kernels for f64

I don't see much difference in performance between Generic kernels and WasmSimd128Kernel, so it will probably be deprecated.
But NeonKernel<f64> can be implemented, for sure.


JAD extraction

JAD is plaintext
JAD always contains the word MIDlet
JAD is boring
I'll skip JAD

Ended up writing up anyway... that's what happens when ideas are not dropped in time.


stores defined as object

Instead of defining stores as
const stores = {
  storeOne: [
    ['someCollection', 'someKey'],
    ['someOtherCollection', 'someOtherKey'],
  ],
  storeTwo: [
    ['someThirdCollection', 'someThirdKey'],
  ],
};

why not something like
const stores = {
  storeOne: {
    someCollection: 'someKey',
    someOtherCollection: 'someOtherKey',
  },
  storeTwo: {
    someThirdCollection: 'someThirdKey',
  },
}

Hi, @wtgtybhertgeghgtwtg . store.createIndex() accepts the third argument as "optionalObjectParameters". How can you design a cleaner API for that?
I guess that's why ;)

const stores = {
  storeOne: {
    // objectIndexName will be 'someCollection', objectKeyPath will be 'someKey'
    someCollection: 'someKey',
    // objectIndexName will be 'someOtherCollection', objectKeyPath will be 'someOtherKey', optionalObjectParamaters will be 'optionalParameters'
    someOtherCollection: {
      objectKeyPath: 'someOtherKey',
      optionalObjectParameters: 'optionalParameters',
  },
}

Thank you for the concrete example. That'd be the answer as a matter of fact. And I understand it'd be more kind to people who doesn't know the native IDBObjectStore.createIndex() interface.
The reason I prefer the current API is that I want SyncedDB to be "raw" as much as possible. The option (['someCollection', 'someKey']) goes to store.createIndex.apply(store, option) and I feel it's simple, fast and enough flexible.
I admit the object argument is not understandable at first. Here why don't we just modify API.md, instead of starting if (typeof storeOpts[indexName] === 'string')?

As you will.

Yes. What @piglovesyou wrote is excactly the reason why I originally went with the current design.
I also considered teh same as your proposal @wtgtybhertgeghgtwtg. But while it looks simpler at first it introduces an edge case and one will have to design a workaround as you did. In my opinion this ends up making the API less simple compared to the current where the third argument is handled seamlessly.


Crash occurs when the OR operator is used in the WHERE clause of the MATCH query.

Describe the bug
I ran the query below with the console, and there was a crash.
(root@nebula) [test]> MATCH (s:user)-[:follow]->(t:user) \
                   -> WHERE s.id == "u00000000000000000000000000000001" OR s.id == "u00000000000000000000000000000002" \
                   -> RETURN s.id AS sid, t.id AS tid
2021/04/13 22:41:40 [ERROR] Failed to reconnect, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused
2021/04/13 22:41:40 Loop error, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused

Bye root!
Tue, 13 Apr 2021 22:41:40 JST

panic: Loop error, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused

goroutine 1 [running]:
log.Panicf(0x624b08, 0xe, 0xc000047e78, 0x1, 0x1)
        /usr/local/go/src/log/log.go:361 +0xc5
main.main()
        /home1/irteamsu/nebula-console/main.go:386 +0x4f3

Environments

OS

Linux nebula001-dev 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Nebula Version: Nebula RPM package

nebula-graph-2.0.0.el7.x86_64.rpm

CPU Info

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
...
Model name:            Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
...

Service Architecture

metad: 3ea
graphd: 3ea
storaged: 3ea

How To Reproduce

Create tag & edge type

CREATE TAG user(id fixed_string(33), type int, region fixed_string(2), crtm timestamp);
CREATE TAG INDEX user_idx_id on user(id);
REBUILD TAG INDEX user_idx_id

CREATE EDGE follow(crtm int);

Insert vertex & edge

INSERT VERTEX user (id, type, region, crtm) VALUES "u00000000000000000000000000000001":("u00000000000000000000000000000001", 1, "JP", now()); /* 1 million data inserts */
INSERT EDGE follow(crtm) VALUES "u00000000000000000000000000000001"->"u00000000000000000000000000000002":(now()) /* 1 million data inserts */

Execute Match Query

MATCH (s:user)-[:follow]->(t:user) \
WHERE s.id == "u00000000000000000000000000000001" OR s.id == "u10002" OR s.id == "u00000000000000000000000000000002" \
RETURN s.id AS sid, t.id AS tid

Expected behavior
The results of the query shall be as follows:
+-------------------------------------+-------------------------------------+
| sid                                 | tid                                 |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000002" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000003" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000004" |
+-------------------------------------+-------------------------------------+
... more ...

Sorry for the trouble, but i didn't reproduce this problem. Would please provide more informations? such as the logs, core file.

The crash log is not found in the error log files(nebula-storaged.ERROR, storaged-stderr.log).
I can't find where the core file is, so can I know what the core file path is?

Did you find anything in nebula-graph logs? And you might find the core files by following the path defined in /proc/sys/kernel/core_pattern.

According to your description, the problem might occurs at one of the three nebula-graphd, so you have to check which one had crashed and get the logs.

Nothing found in nebula-graph logs.
And I looked it up as you said, but the core file was not created either.

That's weird, is this a case that could be reproduced in your environment? And would you please make sure that the daemons (graphd, metad, storage) are all healthy? And make sure you console is updated to the corresponding version.

I didn‘t reproduce the error either, this is my local test:
(czp@nebula) [test]> DROP space test;
Execution succeeded (time spent 3216/3821 us)

(czp@nebula) [(none)]> CREATE SPACE test(VID_TYPE=FIXED_STRING(40));
Execution succeeded (time spent 2620/3009 us)

(czp@nebula) [(none)]> use test;
Execution succeeded (time spent 1610/2199 us)

(czp@nebula) [test]> CREATE TAG user(id fixed_string(33), type int, region fixed_string(2), crtm timestamp);
Execution succeeded (time spent 1918/2207 us)

(czp@nebula) [test]> CREATE TAG INDEX user_idx_id on user(id);
Execution succeeded (time spent 1459/1613 us)

(czp@nebula) [test]> CREATE EDGE follow(crtm int);
Execution succeeded (time spent 2575/3113 us)

(czp@nebula) [test]> REBUILD TAG INDEX user_idx_id;
+------------+
| New Job Id |
+------------+
| 46         |
+------------+
Got 1 rows (time spent 2206/2769 us)

(czp@nebula) [test]> SHOW JOBS;
+--------+---------------------+------------+-------------------------+-------------------------+
| Job Id | Command             | Status     | Start Time              | Stop Time               |
+--------+---------------------+------------+-------------------------+-------------------------+
| 46     | "REBUILD_TAG_INDEX" | "FINISHED" | 2021-04-17T03:24:42.000 | 2021-04-17T03:24:42.000 |
+--------+---------------------+------------+-------------------------+-------------------------+

(czp@nebula) [test]> INSERT VERTEX user (id, type, region, crtm) VALUES "u00000000000000000000000000000001":("u00000000000000000000000000000001", 1, "JP", now()); /* 1 million data inserts */
Execution succeeded (time spent 2150/2417 us)

(czp@nebula) [test]> INSERT VERTEX user (id, type, region, crtm) VALUES "u00000000000000000000000000000002":("u00000000000000000000000000000002", 1, "JP", now()); /* 1 million data inserts */
Execution succeeded (time spent 693/848 us)

(czp@nebula) [test]> INSERT EDGE follow(crtm) VALUES "u00000000000000000000000000000001"->"u00000000000000000000000000000002":(now()) /* 1 million data inserts */
Execution succeeded (time spent 2247/2850 us)

(czp@nebula) [test]> MATCH (s:user)-[:follow]->(t:user) \
                  -> WHERE s.id == "u00000000000000000000000000000001" OR s.id == "u10002" OR s.id == "u00000000000000000000000000000002" \
                  -> RETURN s.id AS sid, t.id AS tid;
+-------------------------------------+-------------------------------------+
| sid                                 | tid                                 |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000002" |
+-------------------------------------+-------------------------------------+
Got 1 rows (time spent 8680/9344 us)

One thing you need to know is that you created a null-point when you insert an edge that end point didn't exist, the tag of the null-point is not user.

That's weird, is this a case that could be reproduced in your environment? And would you please make sure that the daemons (graphd, metad, storage) are all healthy? And make sure you console is updated to the corresponding version.

I agree with that, plz ensure the above conditions. @RenmanKang  We will be looking forward to your feedback.

I created a new graph space and tested it again.
It generated 1 million user vertices(vid: u00000000000000000000000000000001 ~ u00000000000000000000000001000000) and about 500,000 follow edges.
Then i ran the following query.
MATCH (s:user)-[:follow]->(t:user) \
  WHERE s.id == "u00000000000000000000000000000001" \
  OR s.id == "u00000000000000000000000000000002" \
  OR s.id == "u00000000000000000000000000000003" \
  RETURN s.id AS sid, t.id AS tid;

+-------------------------------------+-------------------------------------+
| sid                                 | tid                                 |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000001" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000002" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000002" | "u00000000000000000000000000000010" |
+-------------------------------------+-------------------------------------+
... more ...

Got 662 rows (time spent 383615745/383511260 us)

There was no crash this time. But the query took too long.
It takes 383,615,745us(383sec) even though user.id has index.
I don't understand why it takes so long.
Anyway, since then, I have added about 500,000 more follow edges(total 1 million edges) and run the same query. Then a crash occurred.
MATCH (s:user)-[:follow]->(t:user)   WHERE s.id == "u00000000000000000000000000000001"   OR s.id == "u00000000000000000000000000000002"   OR s.id == "u00000000000000000000000000000003"   RETURN s.id AS sid, t.id AS tid;
2021/04/18 21:06:26 [ERROR] Failed to reconnect, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused
2021/04/18 21:06:26 Loop error, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused

Bye root!
Sun, 18 Apr 2021 21:06:26 JST

panic: Loop error, Failed to open transport, error: dial tcp <IP_ADDRESS>:9669: connect: connection refused

goroutine 1 [running]:
log.Panicf(0x624b08, 0xe, 0xc000093e78, 0x1, 0x1)
        /usr/local/go/src/log/log.go:361 +0xc5
main.main()
        /home1/irteamsu/nebula-console/main.go:386 +0x4f3

Given the above circumstances, this seems to be a memory problem(All three test servers are using 8GB of memory).
If the number of edges is large, there seems to be a problem with using the OR operator in the MATCH query... Is this a normal situation? Is there any solution to this?
On the other hand, querying with vid takes much shorter.
MATCH (s:user)-[:follow]->(t:user) \
  WHERE id(s) == "u00000000000000000000000000000001" \
  OR id(s) == "u00000000000000000000000000000002" \
  OR id(s) == "u00000000000000000000000000000003" \
  RETURN s.id AS sid, t.id AS tid;

+-------------------------------------+-------------------------------------+
| sid                                 | tid                                 |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000001" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000001" | "u00000000000000000000000000000002" |
+-------------------------------------+-------------------------------------+
| "u00000000000000000000000000000002" | "u00000000000000000000000000000010" |
+-------------------------------------+-------------------------------------+
... more ...

Got 662 rows (time spent 290154/291801 us)

Bcz s.id is an attribute filter and needs to scan all data to find the starting points, it is not necessary to query with vid directly. You can profile the query to get the execution cost.

We had found the problem due to your detailed description, it is because we did not provide an efficient execution plan which would lead to full tag scan which finally caused the OOM on your environment. We will improve it as soon as possible. For now, you could turn to the alternative usage which you had already found:
MATCH (s:user)-[:follow]->(t:user) \
  WHERE id(s) == "u00000000000000000000000000000001" \
  OR id(s) == "u00000000000000000000000000000002" \
  OR id(s) == "u00000000000000000000000000000003" \
  RETURN s.id AS sid, t.id AS tid;

Again, sorry for the trouble.


backup_nhg_test.go : defaultNetworkInstance = "default"

https://github.com/openconfig/featureprofiles/blob/c5eb74180a2d49a30786123b08518b42af4f9415/feature/experimental/backup_nhg/ate_tests/backup_nhg_test/backup_nhg_test.go#L40
Same context as issues #333 and #329 . Shouldn't the NetworkInstance name be "DEFAULT" instead of "default" here?

Thanks for reporting this issue, we will address via #329


updated v1.5.0

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

That seems to have done the job! 🎉

That seems to have done the job! 🎉

Ping @marcelotrevisani! Would be great if you've got time to merge this. I'm happy to lend a hand going forward...

Ping @marcelotrevisani! Would be great if you've got time to merge this. I'm happy to lend a hand going forward...

Ping @marcelotrevisani! Would be great if you've got time to merge this. I'm happy to lend a hand going forward...

Sorry for my delay, I got sick in the past days and also because of the holidays it got delayed

Ping @marcelotrevisani! Would be great if you've got time to merge this. I'm happy to lend a hand going forward...

Sorry for my delay, I got sick in the past days and also because of the holidays it got delayed

Sorry for my delay

Not a problem - being merged in under a day is very fast! So, thanks for the merge. I took the opportunity to update a few other repos I'm using too so I can help out in future...

Sorry for my delay

Not a problem - being merged in under a day is very fast! So, thanks for the merge. I took the opportunity to update a few other repos I'm using too so I can help out in future...


Log4j log correlation: Capitalize "opencensus" as one word in context key names.

For example, this commit renames "openCensusTraceId" to "opencensusTraceId".

Codecov Report

Merging #1414 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1414   +/-   ##
=========================================
  Coverage     83.71%   83.71%
  Complexity     1522     1522
=========================================
  Files           233      233
  Lines          6993     6993
  Branches        661      661
=========================================
  Hits           5854     5854
  Misses          939      939
  Partials        200      200

Impacted Files
Coverage Δ
Complexity Δ

...ion/log4j2/OpenCensusTraceContextDataInjector.java
92.5% <ø> (ø)
19 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 18a2bbd...9ee967d. Read the comment docs.

Thanks!


Failed to connect to the cardano-rest explorer api

Context

Information
-

Version
2.1.0

Platform
Docker

Installation
From Github Release

Steps to Reproduce

pull the latest release from github
rm .example of the files in the folder config
NETWORK=mainnet docker-compose up -d --build && docker-compose logs -f
After several hours of synching, run curl http://localhost:8100/api/txs/last

Expected behavior
json response
Actual behavior
curl: (56) Recv failure: Connection reset by peer
When I checked the log from the  explorer api
docker-compose logs -f cardano-explorer-api
the log shows

`Attaching to cardano-rest_cardano-explorer-api_1
cardano-explorer-api_1  | Generating PGPASS file
cardano-explorer-api_1  | [Info] Connecting to database at postgres:5432
cardano-explorer-api_1  | cardano-explorer-api: libpq: failed (could not send data to server: No route to host
cardano-explorer-api_1  | could not send SSL negotiation packet: No route to host
cardano-explorer-api_1  | )

Please try the latest version and reopen if still experiencing issues


hpack doesn't see dependency

I'm testing hpack with stack.
I have standard project: library + executable that use this library. But when I specify dependency from this library, stack build gives me an error:
Main.hs:17:1: error:
    Failed to load interface for    BreadU
    It is a member of the hidden package    breadu-0.3.1   .
    Perhaps you need to add    breadu    to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.

It's really strange, because I have that dependency in my package.yaml and even see that dependency in generated .cabal-file!
library:
    source-dirs:      src/lib
    exposed-modules:
        - BreadU
    default-extensions:
        - RecordWildCards
        - OverloadedStrings
        - MultiWayIf
        - ViewPatterns
    ghc-options:      -Wall -Werror -Wno-orphans -O

executables:
    breadu-exe:
        main:           Main.hs
        source-dirs:    src/app
        ghc-options:    -threaded -rtsopts -with-rtsopts=-N -Wall -Werror -O
        dependencies:
            - breadu

How can I fix it? I'm using these tools:

stack 1.4.0,
LTS Haskell 8.18,
Ubuntu 16.04.2.

Hey!  Thanks for taking the time to open this issue.
I don't see anything wrong with the package.yam you provide.
Can you push what you have to some branch so that I can take a look at it?

https://github.com/denisshevchenko/breadu.info/blob/Hpack/package.yaml

@denisshevchenko Aren't you just missing the breadu dependency for the update-breadu-server executable?
(Alternatively, perhaps you misspelled main or source-dirs for update-breadu-server, as it seems strange to build the same binary twice.)

@denisshevchenko Aren't you just missing the breadu dependency for the update-breadu-server executable?

@liskin Of course, because update-breadu-server doesn't use breadu. :-) Only breadu-exe uses breadu lirary.

@denisshevchenko It's the same source file as breadu-exe. How come one doesn't use breadu and the other does? It makes no sense. :-(

Wait. I see this:
executables:
    # BreadU server.
    breadu-exe:
        main:           Main.hs
        source-dirs:    src/app
        ghc-options:    -threaded -rtsopts -with-rtsopts=-N -Wall -Werror -O
        dependencies:
            - breadu
    # BreadU server updater (using during CI).
    update-breadu-server:
        main:           Main.hs
        source-dirs:    src/app
        ghc-options:    -threaded -rtsopts -with-rtsopts=-N -Wall -Werror -O

executables, not executable, right? I'm trying to define two different executables here. Is it possible? Or I have to change it like this?
executables:
    # BreadU server.
    breadu-exe:
        main:           Main.hs
        source-dirs:    src/app
        ghc-options:    -threaded -rtsopts -with-rtsopts=-N -Wall -Werror -O
        dependencies:
            - breadu

executables:
    # BreadU server updater (using during CI).
    update-breadu-server:
        main:           Main.hs
        source-dirs:    src/app
        ghc-options:    -threaded -rtsopts -with-rtsopts=-N -Wall -Werror -O

Nope, it doesn't work too. :-(

As I said, I have to build two different executables, and only one of them depends on breadu library.

@denisshevchenko As @liskin pointed out, the executable section for update-breadu-server is most likely the issue.  You either have to add dependencies: breadu to it OR change main: to something different than Main.hs.  Currently breadu-exe and update-breadu-server are *exactly the same, exception for update-breadu-server the dependency on breadu.

Aaaaaah... I copied source-dirs:    src/app instead of source-dirs:    src/update. Sorry, guys.


imod.evaluate.flow_velocity requires porosity but doesnt use it

In GitLab by @jdelsman on Dec 17, 2020, 10:02
@Huite what was your intent for this function? It does not divide by porosity, as I would expect it to do. Shouldn't it output flow velocity, as the name implies?

In GitLab by @Huite on Nov 4, 2022, 10:56
This indeed looks like a clumsy mistakes, it should divide by porosity.


Drawing lines across 180º longitude wraps them the wrong way

Another issue coming out of the ZIPScribble map project, this time with Russia. That country inconveniently sticks out across 180º longitude. When drawing a line from just below 180º to just above 180º, polymaps draws a very long line from one end of the map to the other. Using negative values or values above 180 doesn't make any difference. I think there should be a convention so that values over 180º are drawn to the east, rather than wrapping around and appearing in the far west.

Duplicate of #33. I think it's supposed to work with values above +180º and below -180º.


Can't specify no GRPC proxy when HTTPS/HTTP proxies exist on Windows

What version of gRPC and what language are you using?
1.39.1, C++, but the issue still exists on main
What operating system (Linux, Windows,...) and version?
Windows 10
What runtime / compiler are you using (e.g. python version or version of gcc)
Visual Studio 2019
What did you do?
I have HTTP_PROXY and HTTPS_PROXY environment variables set
I want no proxy for GRPC requests
In http_proxy.cc you check:
if (uri_str[0] == '\0') goto done;
But in Windows I can't specify a zero-length environment variable, and therefore can't disable the GRPC proxying while HTTP proxying is enabled
What did you expect to see?
No proxy used for GRPC requests. Some other GRPC_PROXY value that means don't use a proxy ?

The channel arg GRPC_ARG_HTTP_PROXY would be better for your purposes. It would take preference over the environment variables.