Many tasks + heavy background processes running = new epileptic is born

Hi guys!
This library is very cool, we're using it in our CMA project to set up and run Magento 2 projects in a single command.
But I want to report, that we have "flashing" problems in a console when our program is running.
This problem appears everywhere: in VSCode, PHPStorm, ITerm2, Konsole and so on.
https://user-images.githubusercontent.com/18352350/126329085-bf8e9d6d-0882-4356-953f-4820e3a91004.mp4
It appears mostly when something heavy is running in the background, like PHP compilation or Magento setup, but can also flash it when the program is starting.
What do you think, where should I dig in to fix this problem?
Thanks in advance!

Hello @ejnshtein,
Thank you for the kind words. This is unfortunately pretty normal because it tries to update the renderer every 100ms or whenever a task changes state or gives output etc. This flickering can occur where it does not get enough CPU time and a task updates too frequently.
The solution to this will be to swap out the underlying library of log-update for stdout-update, which I have tried and can achieve to keep the lines that will not encounter any changes static and update only the rest minimizes this issue. But still has many problems with the compatibility of the tests. It also has a really old v4.0.0-beta version of this library to test it out.
I really do not know when I will really finish and merge it, but it's in my plans and since I want to archive more structural changes with the upcoming major update, currently I do not have the time for it.
Listed in upcoming changes for a while now:
https://listr2.kilic.dev/migration/upcoming-changes
The related issue is tracked here:
https://github.com/cenk1cenk2/listr2/issues/296

That's good to hear!
We will try the beta version as well.
Thanks for your work!

Indeed, beta is not flickering, but it doesn't update properly the console. At least something is working :smile:

Okay, will be waiting for the new major version! Good luck!

I will be closing this issue in favor of #296 . Feel free to reopen it any time. But the problem is acknowledged.

import { execaCommand as command } from 'execa'
import { Listr } from 'listr2'

const tasks = new Listr([
  {
    title: 'This task will do a curl request.',
    task: async (_, task): Promise<void> => {
      const execute = command('rsync -a --info=progress2 /path/to/[big folder] /path/to/dest')

      execute.stdout.pipe(task.stdout())
      execute.stderr.pipe(task.stdout())

      await execute
    },
    rendererOptions: {
      outputBar: Infinity,
      persistentOutput: true
    }
  }
])

await tasks.run()

Just a simple test:
It flashes on macOS 14.0 (23A344)


Capture deployment.environment

Kibana uses this value to categorize applications.
Open telemetry spec here
https://github.com/open-telemetry/opentelemetry-specification/blob/b46bcab5fb709381f1fd52096a19541370c7d1b3/specification/resource/semantic_conventions/deployment_environment.md

I've added "default" as the default value for this. I'm going to punt this feature to a later version.

duplicated in #149


clean cached token on logoutError to avoid app returning to loggedIn …

…state after connection resumes

Coverage decreased (-0.02%) to 27.907% when pulling d93ae2fdfc655383464d6020a4fd9ad134914d6a on u9520107:bug/offline-logout into 1d568ded068951edb95f485cd2e2263c9838e68f on ringcentral:master.


dial tcp [::1]:9444: connect: connection refused

Hello,
This error keeps popping up when the UI refreshes and it looks the container is crashing. How do i enable debug for pyrra to trace this further ? I have read through the documentation and cant find much on how to debug.
I'm using the helm chart for deployment - https://artifacthub.io/packages/helm/rlex/pyrra
Here is my k8s container / pod status
$ k describe pod pyrra-7f6944bccc-8wnvj -nmonitoring
Name:             pyrra-7f6944bccc-8wnvj
Namespace:        monitoring
Priority:         0
Service Account:  pyrra
Node:             aks-default-49586363-vmss000003/<IP_ADDRESS>
Start Time:       Mon, 12 Dec 2022 13:19:39 +0530
Labels:           app.kubernetes.io/instance=pyrra
                  app.kubernetes.io/name=pyrra
                  pod-template-hash=7f6944bccc
Annotations:      kubectl.kubernetes.io/restartedAt: 2022-11-29T14:08:04Z
Status:           Running
IP:               xxx.xxx.3.60
IPs:
  IP:           xxx.xxx.3.60
Controlled By:  ReplicaSet/pyrra-7f6944bccc
Containers:
  pyrra-kubernetes:
    Container ID:  containerd://0c79e37a237c951bc0f5eaaab0e3ffd7855a9342290c49e82894887a80708831
    Image:         artifactory.test.com/docker-virtual-all/pyrra:latest
    Image ID:      artifactory.test.com/docker-virtual-all/pyrra@sha256:eda3578a09ec7533adcc24c1d9dcaa463d428d4c09bf0bed23ac9b20a04190ae
    Port:          <none>
    Host Port:     <none>
    Args:
      kubernetes
    State:          Running
      Started:      Mon, 12 Dec 2022 13:43:14 +0530
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 12 Dec 2022 13:29:38 +0530
      Finished:     Mon, 12 Dec 2022 13:43:13 +0530
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f2l7r (ro)
  pyrra:
    Container ID:  containerd://69697c55f4f8a2537cc3012a8eaf3e7dedd7d3e22a7ded8a31c49cfd69b376af
    Image:         artifactory.test.com/docker-virtual-all/pyrra:latest
    Image ID:      artifactory.test.com/docker-virtual-all/pyrra@sha256:eda3578a09ec7533adcc24c1d9dcaa463d428d4c09bf0bed23ac9b20a04190ae
    Port:          9099/TCP
    Host Port:     0/TCP
    Args:
      api
      --prometheus-url=http://kube-prometheus-stack-prometheus.monitoring:9090/prometheus
      --api-url=http://localhost:9444
      --prometheus-external-url=http://monitoring.test.com/prometheus
      --route-prefix=/pyrra
    State:          Running
      Started:      Mon, 12 Dec 2022 13:19:39 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f2l7r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-f2l7r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  38m                default-scheduler  Successfully assigned monitoring/pyrra-7f6944bccc-8wnvj to aks-default-49586363-vmss000003
  Normal   Pulled     38m                kubelet            Container image "artifactory.test.com/docker-virtual-all/pyrra:latest" already present on machine
  Normal   Created    38m                kubelet            Created container pyrra
  Normal   Started    38m                kubelet            Started container pyrra
  Warning  BackOff    29m                kubelet            Back-off restarting failed container
  Normal   Pulled     15m (x4 over 38m)  kubelet            Container image "artifactory.test.com/docker-virtual-all/pyrra:latest" already present on machine
  Normal   Created    15m (x4 over 38m)  kubelet            Created container pyrra-kubernetes
  Normal   Started    15m (x4 over 38m)  kubelet            Started container pyrra-kubernetes

pod logs

$ k logs pyrra-7f6944bccc-8wnvj --all-containers -nmonitoring | grep -v query
level=info ts=2022-12-12T08:13:15.041643356Z caller=main.go:118 msg="using Prometheus" url=http://localhost:9090
1.670832795593303e+09   INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
1.6708327955937471e+09  INFO    setup   starting manager
1.6708327955942245e+09  INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6708327955944364e+09  INFO    Starting EventSource    {"controller": "servicelevelobjective", "controllerGroup": "pyrra.dev", "controllerKind": "ServiceLevelObjective", "source": "kind source: *v1alpha1.ServiceLevelObjective"}
1.6708327955944836e+09  INFO    Starting Controller     {"controller": "servicelevelobjective", "controllerGroup": "pyrra.dev", "controllerKind": "ServiceLevelObjective"}
1.6708327956947458e+09  INFO    Starting workers        {"controller": "servicelevelobjective", "controllerGroup": "pyrra.dev", "controllerKind": "ServiceLevelObjective", "worker count": 1}
level=debug ts=2022-12-12T08:13:15.694929334Z caller=servicelevelobjective.go:55 controllers=ServiceLevelObjective reconciler=servicelevelobjective namespace=monitoring/edge-http-errors msg=reconciling
level=info ts=2022-12-12T08:13:15.839083758Z caller=servicelevelobjective.go:90 controllers=ServiceLevelObjective reconciler=servicelevelobjective namespace=monitoring/edge-http-errors msg="updating prometheus rule" namespace=monitoring name=edge-http-errors
level=debug ts=2022-12-12T08:13:15.8742071Z caller=servicelevelobjective.go:55 controllers=ServiceLevelObjective reconciler=servicelevelobjective namespace=monitoring/edge-latency-slo msg=reconciling
level=info ts=2022-12-12T08:13:15.875084014Z caller=servicelevelobjective.go:90 controllers=ServiceLevelObjective reconciler=servicelevelobjective namespace=monitoring/edge-latency-slo msg="updating prometheus rule" namespace=monitoring name=edge-latency-slo
level=info ts=2022-12-12T07:49:40.016393867Z caller=main.go:118 msg="using Prometheus" url=http://kube-prometheus-stack-prometheus.monitoring:9090/prometheus
level=info ts=2022-12-12T07:49:40.016453568Z caller=main.go:171 msg="UI redirect to Prometheus" url=http://monitoring.test.com/prometheus
level=info ts=2022-12-12T07:49:40.016463868Z caller=main.go:172 msg="using API at" url=http://localhost:9444
level=info ts=2022-12-12T07:49:40.016472368Z caller=main.go:173 msg="using route prefix" prefix=/pyrra

The error dial tcp [::1]:9444: connect: connection refused sounds like the actual problem is with the Pyrra kubernetes-backend because that listens on port 9444. I'd be curious what errors you can find when describing the pod.
However, given that the pyrra-api also has OOMKills, you probably want to set way higher resource limits on the pyrra-api pod. Currently, all queries are proxied and converted by the pyrra-api, so if there are more requests inflight the pyrra-api container will actually use quite some memory. I'd suggest to limit it at 256Mi if not 512Mi to be on the safe side.

Hey,
Thanks for the fix. Increasing the memory limit has fixed this.
Describing the pod doesn't show any errors apart what shared earlier i,e Laststate - Terminated/OOMkilled.
BTW, How to enable debug - the current logs are pretty sparse.
Thanks


Allow direct download of files (VCFs)

Should be able to use what @jburos has here: https://github.com/jburos/tcga-blca/blob/master/query_tcga/query_tcga.py#L246-L302

Thanks for making this an issue! Makes it easier to discuss. I made repo public so that links would work better.
In hopes of fostering discussion, wanted to note some design decisions I've been thinking about.
Enable easy switch between using gdc-client vs api

Implementation in tcga-blca has concept of a DATA_DIR -- root directory in which files get loaded. This currently mirrors directory structure imposed by gdc-client.

Ideally this structure should be the same irrespective of whether user downloaded via api or gdc-client.
Also, this directory structure may or may not be conducive to Cohorts integration
Presumably, gdc-client could change this directory structure.

Authorization

Process for linking to / referencing authorization token file
It would be nice if nothing broke if auth file not present

The desired user interface for downloading files
What should the command look like for downloading files? Should this be get_vcfs, analogous to get_cases, with download behind the scenes? Or should the download event be handled more explicitly?
We should probably also reconcile the current approaches to filtering in order to support this.
extensibility
should download of VCFs be different from that for other sample-data files (e.g. Raw Sequencing Data)?


[silgen] When emitting convertBridgeObjectToRef, forward ownership th…

…rough the instruction rather than unconditionally adding a cleanup.
Otherwise, we may put cleanups on borrowed values.
rdar://34222540

This is coming up a bunch more since we are calling builtins with @guaranteed parameters instead of @owned parameters. We may want to have builtins have their own convention. Not sure yet (depends how much breaks). This should be NFC without normal arguments being at +0.

@swift-ci test and merge


how to get wio_ekf

Hi, Thanks for your sharing!
However, when I check the test bag, I wonder how can I get the wio_ekf ? since I only got wheel odometry in my dataset.

Wheel odometry should work fine in place of WIO_EKF. That topic was produced from a separate sensor fusion package that is on open-sourced at the moment (it combines IMU and wheel odometry).
I think just wheel odometry would be ok - your roll and pitch isn't going to be very helpful, but if you have enough lidar features that should be ok.
Otherwise, the general approach is to use the IMU for roll and pitch, wheel odom for x, y, z, yaw - but fused through an ekf I believe.

Thanks for your quick reply! I get it!


tour: first lesson's random number always returns 1

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    fmt.Println("My favorite number is", rand.Intn(10))
}

this always prints:
My favorite number is 1

Is there a fixed seed? This might confuse new users, expecting the number to change every time pressing the run button.

I replied via email, but it's not shown. Here is the text:
math/rand is deterministic (when used without seeding) and the
left side also gives a note:
Note: the environment in which these programs are executed is
deterministic, so each time you run the example program rand.Intn
will return the same number. (To see a different number, seed
the number generator; see rand.Seed.)

math/rand is deterministic (when used without seeding) and the
left side also gives a note:
Note: the environment in which these programs are executed is
deterministic, so each time you run the example program rand.Intn
will return the same number. (To see a different number, seed
the number generator; see rand.Seed.)

Better to add comment in the lesson's code. This will enable users to understand instantly rather than getting surprised.

For what it's worth: I was working through the tour, got confused by this as well, and ended up here.

I don't understand why the first lesson on Golang's website involves importing a random number generator but not providing an example for how a random generator is actually implemented?  It seems confusing to not include an example of Seed.


[api-docs] Remove quotes on invalid recipients

A new commit was made into the api-docs repo. This issue was created automatically. Need to review if we made any change to the library for this commit.
https://github.com/discord/discord-api-docs/commit/d0bb0b1854bbc182ccea9c8301ada308a1f92a2b

This commit will not effect our lib's code. No changes required.


FLPATH-377: workflow log

What this PR does / why we need it:
display custom log in ui
Which issue(s) this PR fixes
Fixes #FLPATH-377
Change type

[ ] New feature
[ ] Bug fix
[ ] Unit tests
[ ] Integration tests
[ ] CI
[ ] Documentation
[ ] Auto-generated SDK code

Impacted services

[ ] Workflow Service
[ ] Notification Service

Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.

I do not really like the implementation TBH. As a user experience, I think that it's not great. For me, the implementation should be something.
this.log.error()
this.log.info()
And execute context should be inherent inside the class, and the log should log the workflow.ID as wid='context.get(workflowid)' In this way, a user can retrieve all the logs related to a workflow based on a simple query in any 3-party tool (Loki, ElasticSearch, etc..)
Dancing the context with multiple functions is not the best, and a function like addTaskLog is not great for user experience.
But the code lgtm.

agree.  from user experience, we should simplify the code to get context property.  I will fix it in next PR.

https://issues.redhat.com/browse/FLPATH-403


Unstable Fabric Loader versions are marked as reccomended.

The logic here is not correct: https://github.com/PrismLauncher/meta/blob/master/generateFabric.py#L72-L73
As seen here (at the time of writing): https://meta.fabricmc.net/v2/versions/loader loader 0.14.15 is not marked as stable, 0.14.14 should be the default version users install.
It seems my change made here: https://github.com/PrismLauncher/meta/commit/6f742903d06f885e76d909bc6e8859a1e13acf2a was lost during a refactor.

Yep. This was lost during a refactor. I will fix this!

Fixed in 9d6c195c47948042bbb23394229843d2bd8adb9a

Also triggered a manual update to our meta: https://github.com/PrismLauncher/meta-launcher/commit/f6da648794beea603be0bd3a496dba1c386774e7

Perfect, many thanks for resolving this so quickly 👍


hiding _id in embedded documents doesn't work

db.mycollection.find({}, {_id: 0, 'myembeddeddoc._id': 0});
It works okay in mongo shell.

@aheckmann ping. Is there any workaround for this?
query.exec(function(err, docs) {
delete docs[1]._id; // doesn't seem to work either :(
});

found a workaround:
query.exec(function(err, docs) {
delete docs[1]._doc._id;
});

nice! thanks @aheckmann. I'm going to try that.

btw latest version in npm has the fix

even better, thanks man

@c4milo - fixed in master (#590)

great! thank you!


issues with the boss

In the first fight, the boss does not wake up, and in the Godhome fight, the boss flies into the bottom left corner of the arena. please fix!!!

Could you send me your ModLog.txt and Player.Log files? They contain more information that will allow me to see what's going on. You can find them located at "C:\Users\USERNAME_HERE\AppData\LocalLow\Team Cherry\Hollow Knight" if you are on Windows

It must have been Pale Court, actually. Luckily you patched it.


Java indexer: eliminate possible AssertionError [phab:D878]

Created by schroederc at 2016-05-06 16:44:40:
The normal behavior of TypeTag#getPrimitivTypeKind() (and
JCPrimitiveTypeTree#getPrimitiveTypeKind() by proxy) throws an
exception if the tag is an ERROR type.  This bypasses the exception to
instead just treat errors as normal types so that we can continue
analyzing the rest of the compilation.
This change also adds a --verbose flag to the indexer to emit debug
messages.

shahms commented inline (context lost during migration) at 2016-05-06 16:50:15:
Why LinkedList rather than ArrayList?

shahms wrote at 2016-05-06 16:50:15:
LJTM

shahms accepted this revision at 2016-05-06 16:50:15

schroederc commented inline (context lost during migration) at 2016-05-06 16:52:29:
¯_(ツ)_/¯  Already had that imported and this is just for debugging.

schroederc closed via commit f75f0b68d269d286f1962581049d3c4bae353f7d at 2016-05-06 16:52:49


Fixing hbase/Dataflow shading.

A user who has bigtable-hbase-1.* in the classpath should be able to use bigtable-hbase-dataflow.  Currently, as see in #958, the shading we added so that BigtableIO and CloudBigtableIO can coexist doesn't quite work for bigtable-hbase.

LGTM


[Issue] 0.4 not working with private registry

Current Behavior
Context:

private registry with root CA in /etc/docker/certs.d/
Working correctly with curl
Working correctly with regctl 0.3

When using regctl 0.4 I have the following error on the command regctl repo ls registry.snap.lizlan:5000
WARN[0000] Sleeping for backoff                          Host="registry.snap.lizlan:5000" Seconds=1.9999968909999999
WARN[0002] Sleeping for backoff                          Host="registry.snap.lizlan:5000" Seconds=3.999993341
failed to list repositories for registry.snap.lizlan:5000: Get "https://registry.snap.lizlan:5000/v2/_catalog": x509: certificate signed by unknown authority

Expected Behavior
Working withour error as with regctl 0.3 in the same context.
Steps To Reproduce

Run private registry using self-signed Authority CA
Install your registry CA in the appropiriate location: `/etc/docker/certs.d/<registri_uri>/ca.crt
Check access to the registry (docker, curl etc...)
Run repo ls ...  with regctl 0.4

Version
{
  "VCSRef": "9546658ede6901191b9692a7f720c37150940ddd",
  "VCSTag": "v0.4.0"
}

Environment

Linux  x86_64  (buster, bullseye)
Tested with both docker image and binary.
Docker registry registry:2.7.1

What else
Work as expected with regctl v0.3

Thank you for catching this! I've got a fix available in the latest CI build, and I'm hoping to have 0.4.1 released over the next weekend that will also include this. You can pull the binaries from the latest GHA run:
https://github.com/regclient/regclient/actions/runs/1983942840
If you still see issues with those, let me know and we can reopen this.


Question: EnContactMatcher performance

Good morning,
While trying to use EnContactMatcher.ts with 10k contacts the constructor takes up to 1 min. By changing inner-matchers to non accelerated, it improves considerably (it goes down to 'only' 30 seconds).
I have a few questions:

Are those times reasonable/expected with 10k contacts?
Would it be ok to open a PR adding a new configuration flag for using or not accelerated matchers in EnContactMatcher? Another option would be passing in the EnContactMatcher constructor the fuzzy matcher class to use? Which one would you prefer?
With 10k contacts, there's a very small difference on the performance for find method call (0.1 seconds with accelerated vs 0.3 seconds without). Given the find time is almost negligible compared to the constructor time for accelerated vs non accelerated , we chose to use non accelerated fuzzy matchers. Does this make sense?

Thanks and congratulations for this amazing library :)

The current preprocessing of EnContactMatcher looks what we need to increase accuracy results.
Would it be possible to make EnContactMatcher serializable? Given the constructor takes some time, when being passed a large number of contacts, we'd like to serialize and store in Redis so it can be re-used (deserialized).
Thanks!


Shine.js on SVG element

How to make Shine.js work on SVG elements or Font-icons ?

Next

Any updated on applying this to an SVG?
Seems like the SVG shadow is very similar to the box and text ones, so surely simple enough to modify?
https://jsfiddle.net/jamblo/kojh2xLk/387/


resize breaks on negative input with unhandled exception

Adding the following line in the resize test breaks it:
image.clone().resize(-15, Jimp.AUTO).write("output/lenna-resize-auto.png");

Looking at the code in resize.js, it will set the target height and width to a positive value. In index.js, it doesn't do that and I guess that's where the break happens.
It would make sense to add a check for negative values in index.js as well. If there is no check, the following exception is thrown and I'm unable to catch it at all:
resize.js (13/14)
buffer.js:1040
    throw new TypeError('"value" argument is out of bounds');
    ^

TypeError: "value" argument is out of bounds
    at checkInt (buffer.js:1040:11)
    at Buffer.writeUInt32BE (buffer.js:1133:5)
    at Packer.packIHDR (/Users/joern/git/aggregator/lib/jimp/node_modules/pngjs/lib/packer.js:75:7)
    at PackerAsync.pack (/Users/joern/git/aggregator/lib/jimp/node_modules/pngjs/lib/packer-async.js:24:34)
    at .<anonymous> (/Users/joern/git/aggregator/lib/jimp/node_modules/pngjs/lib/png.js:58:18)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Makes sense. Will add it to the backlog - or feel free to fix it and make a pull request.


Use of jQuery

I'd like to discuss the possibility of dropping jQuery more thoroughly. #1175 is closed, which sort of indicates that discussion is closed. I think it should be reopened.
How much of this project uses jQuery? How much of that could be accomplished with modern JavaScript? If the transitions and animations are done in jQuery, how much of that could be rewritten in CSS3?
From "You Might Not Need jQuery":

If you're developing a library on the other hand, please take a moment to consider if you actually need jQuery as a dependency. Maybe you can include a few lines of utility code, and forgo the requirement.

Maybe two versions of this library could be released, one using jQuery, and another focused on newer browsers, using vanilla JavaScript. The aforementioned page includes several ways in which jQuery support can be replaced with a cross-browser vanilla JS solution.
So, in short, how feasible is a jQuery removal? Is this planned, or at least open to further discussion, even though #1175 is closed?

Reopened https://github.com/Semantic-Org/Semantic-UI/issues/1175


Kubernetes 1.19 or 1.20

It’s not an issue itself, but I’d like to know if would be possible to setup an EKS cluster with Kubernetes version 1.19 or 1.20.
Actually EKS-D installed on Ubuntu 20.04.

Hi @mcolladoio
The eks snap packages and releases the eks-d k8s distribution. At this point the eks-d project has done one release of the 1.18 upstream kubernetes, see [1].
If you want to get a similar experience to what the eks snap offers you can snap install microk8s --classic --channel=<1.19 or 1.20>/stable. Remember to snap remove eks first.
[1] https://github.com/aws/eks-distro/releases

Thanks for your reply @ktsakalozos ! Given that information I'll close the issue.


Cannot read property 'jsx' of undefined

Describe the Bug
I'm using it in a internal library where i've bundled reactflow as dev dependency and using it in a app which uses umd module
following version that i'm using :
"reactflow": "^11.4.2"
"react-dom": "^18.2.0"
"react": "^18.2.0"

it seems like it can't find jsx runtime
This is the snippet of minifed code when my internal library bundles and runs
import { jsx, jsxs, Fragment } from 'react/jsx-runtime';

const ReactFlowProvider = ({ children }) => {
    const storeRef = useRef(null);
    if (!storeRef.current) {
        storeRef.current = createRFStore();
    }
    return jsx(Provider$1, { value: storeRef.current, children: children }); // it throws an error in here stating that jsx is undefined
};
ReactFlowProvider.displayName = 'ReactFlowProvider';

it seems like react doesn't have umd version of jsx runtime

this is my tsconfig file look alike
//tsconfig.json
{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "baseUrl": "./src",
    "declaration": true,
    "declarationDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "jsx": "react",
    "module": "esnext",
    "moduleResolution": "node",
    "noFallthroughCasesInSwitch": true,
    "noUnusedLocals": true,
    "noUnusedParameters": false,
    "outDir": "./dist",
    "skipLibCheck": true,
    "sourceMap": true,
    // Enabling --strict enables
    // --noImplicitAny, --noImplicitThis, --alwaysStrict, --strictBindCallApply,
    // --strictNullChecks, --strictFunctionTypes and --strictPropertyInitialization
    "strict": true,
    "target": "ES5",
    "typeRoots": ["./types", "./node_modules/@types"],
    "resolveJsonModule": true
  },
  "include": ["packages/*/src"],
  "exclude": ["**/__mocks__", "**/*.test.ts*", "**/*.stories.*"],
  "types": ["node"]
}

and i'm using rollup to pack my library look alike
// rollup.config.js
import path from "path";
import autoprefixer from "autoprefixer";
import alias from "@rollup/plugin-alias";
import resolve from "rollup-plugin-node-resolve";
import external from "rollup-plugin-peer-deps-external";
import postcss from "rollup-plugin-postcss-modules";
import sourceMaps from "rollup-plugin-sourcemaps";
import typescript from "rollup-plugin-typescript2";
import commonjs from "rollup-plugin-commonjs";
import replace from "@rollup/plugin-replace";
import svg from "rollup-plugin-svg";
import json from "rollup-plugin-json";
import assets from "postcss-assets";
import pkg from "./package.json";

// Using graceful-fs to patch fs to avoid EMFILE errors that can
// occur on the build server.
const realFs = require("fs");
const gracefulFs = require("graceful-fs");
gracefulFs.gracefulify(realFs);

const plugins = [
  svg(),
  json(),
  postcss({
    extract: "dist/dummy-components.css",
    modules: {
      // Any file without the .module.scss naming convention is
      // considered "global" and will not have the name mangled.
      globalModulePaths: [/(?<!.module)\.s?css$/]
    },
    plugins: [autoprefixer(), assets()],
    sourceMap: true
  }),
  typescript({ clean: true }),
  replace({
    "process.env.NODE_ENV": JSON.stringify(process.env.NODE_ENV || "development")
  }),
  resolve(),
  commonjs({
    namedExports: {
      "../../node_modules/react-is/index.js": ["isValidElementType", "isContextConsumer"],
      "../../node_modules/zustand/index.js": ["createStore"]
    }
  }),
  sourceMaps()
];

export default [
  {
    input: pkg.source,
    output: [
      {
        file: pkg.main,
        format: "cjs",
        sourcemap: true
      },
      {
        file: pkg.module,
        format: "esm",
        sourcemap: true
      }
    ],
    // exclude dependencies AND peer-dependencies
    plugins: [external({ includeDependencies: true }), ...plugins]
  },
  {
    input: pkg.source,
    output: [
      {
        file: pkg.browser,
        name: "dummy.Components",
        format: "umd",
        sourcemap: true,
        globals: {
<EMAIL_ADDRESS>"tippy",
          formik: "Formik",
          "js-joda": "JSJoda",
          react: "React",
          "react-dom": "ReactDOM",
          "react-draggable": "ReactDraggable",
          "react/jsx-runtime": "jsxRuntime", // i tried giving here just like in reactflow library is doing but it doesn't works
          reactflow: "ReactFlow"
        }
      }
    ],
    plugins: [
      // include dependencies, exclude peer-dependencies
      external(),
      alias({
        entries: [
          {
            find: "react-draggable",
            replacement: path.resolve(__dirname, "../../node_modules/react-draggable/build/cjs/cjs.js")
          },
          {
            find: "react-color/lib/components/common",
            replacement: path.resolve(__dirname, "../../node_modules/react-color/lib/components/common/Checkboard.js")
          }
        ]
      }),
      ...plugins
    ],
    // this has it's own UMD build that should be included
    external: ["@dummy/render"]
  }
];

Do we have any workaround of such thing,
Your Example Website or App
No response
Steps to Reproduce the Bug or Issue
bundle react flow library into one single minified js w.r.t following version used (using umd module)
"reactflow": "^11.4.2"
"react-dom": "^18.2.0"
"react": "^18.2.0"

unforutnely react doesn't has umd module for jsx runtime which causes this error to happen

Expected behavior
It shouldn't throw such error
Screenshots or Videos
No response
Platform

OS: Windows
Browser: Chrome, Safari, Firefox
Version:
"reactflow": "^11.4.2"
"react-dom": "^18.2.0"
"react": "^18.2.0"

Additional context
No response

is there some other way were we could bundle not with jsx and use React.createElement ?
or any workarounds ?
cc: @moklick

I'm facing a similar issue. Unfortunately, the repo I'm contributing to doesn't have plans to upgrade to the new JSX Transform, since React.createElement is still supported in React 18, so when attempting to use Reactflow, every file in the Reactflow package returns: module not found: error: can't resolve 'react/jsx-runtime'
I've tried every solution I've been seen online to try and work around this, but no luck :(
Are there any alternative ways to bundle as suggested by @avishwakarmaplex ?

@moklick i'm packing reactflow into a package library which is been used inside an app, i can't change my library as "jsx": "jsx-react" since it would be a bigger move for us, since its a huge library, is there any other alternative ?
does<EMAIL_ADDRESS>is build without jsx-runtime ?

This should be fixed in 11.9.0

Thanks a lot @moklick
FYI I ended up stubbing the following code for jsx runtime as vendor and expose to window(using an older version of reactflow) which had fixed the issue, but i think now i could get rid of this code and update the reactflow to latest version cheers !!!!!!
(function () {
    const RESERVED_PROPS = {
        key: true,
        ref: true,
        __self: true,
        __source: true
    };

    function jsx(type, config, maybeKey) {
        let propName;

        // Reserved names are extracted
        const props = {};

        let key = null;
        let ref = null;

        // Currently, key can be spread in as a prop. This causes a potential
        // issue if key is also explicitly declared (ie. <div {...props} key="Hi" />
        // or <div key="Hi" {...props} /> ). We want to deprecate key spread,
        // but as an intermediary step, we will use jsxDEV for everything except
        // <div {...props} key="Hi" />, because we aren't currently able to tell if
        // key is explicitly declared to be undefined or not.
        if (maybeKey !== undefined) {
            key = "" + maybeKey;
        }

        if (config.key !== undefined) {
            key = "" + config.key;
        }

        if (config.ref !== undefined) {
            ref = config.ref;
        }

        // Remaining properties are added to a new props object
        for (propName in config) {
            if (Object.prototype.hasOwnProperty.call(config, propName) && !RESERVED_PROPS.hasOwnProperty(propName)) {
                props[propName] = config[propName];
            }
        }

        // Resolve default props
        if (type && type.defaultProps) {
            const defaultProps = type.defaultProps;
            for (propName in defaultProps) {
                if (props[propName] === undefined) {
                    props[propName] = defaultProps[propName];
                }
            }
        }

        return React.createElement(type, Object.assign({ key, ref }, props));
    }

    window.jsxRuntime = { jsx, jsxs: jsx, Fragment: React.Fragment };
})();

You should not need that anymore :)


Fix for issue #4

Thanks @midnightprioriem for this plugin that helped with my documentation!
I was running into the same issue as @chrieke, and the fix seemed trivial, so I made this fork and tested locally... all runs as expected.
With this PR, you can add the hash-links at the end of a filename and have them added back in after the relative link is resolved from, the filesystem crawl.
The regex now generates the following match groups for Link Text:
0: Whole link markup - [Link Text](filename.md#anchor-link)
1: Link text part without brackets - Link Text
2: Full URL without parentheses including hash anchor if present - filename.md#anchor-link
3: Filename without any hash anchor if present - filename.md
4: File extension - .md
5: Hash anchor with hash mark - #anchor-link
Because the match group numbers changed, I had to change references in some other lines.
Thanks again!

Looks and works great, thanks!


feat: alert

Closes: #96

@tdelabro I have a problem with set_sub_ver: [][]const u8, I have implemented the basic structure by ignoring it.
I have created a comment for the code I wrote to run set_sub_ver: [][]const u8, and the error it gives me is:
Segmentation fault at address 0xaaaaaaaaaaaaaaaa
Panicked during a panic. Aborting.
error: while executing test 'network.protocol.messages.alert.test.ok_fullflow_alert_message', the following command terminated with signal 6 (expected exited with code 0)

Can you check from it please

This message is deprecated:
https://bitcoin.org/en/alert/2016-11-01-alert-retirement#reasons-for-retirement


Comment anchor scrolls to the wrong position

Related to #1402

implemented useScrollTo hook
adjusted scroll for comments with key-factors and for key-factors block

But I am curious: why did you decide to make that a hook, and not just a simple helper function?

In order to make sure the code can be executed only on the client. I use useCallback hook, which ensures we access window object in a proper environment.


Fix Theming issues.

If you clicked add contact through the main menu so settings -> Add Contact. The theme would be broken and set to default. It worked normally through the sidenav -> Add Contact though.
Noticed that if you did main-navbar, ul would be undefined. Now fixed!
Also fixed the sign and verify pages as well in a later commit.

@ckhatri I can't seem to replicate the issue with the theme. I tried the steps above but still seems to work okay for me. Would you mind doing a gif please? :)

@alexbarnsley
here ya go: https://gfycat.com/ScarceTautAstarte

@ckhatri did you get round to taking a look at @j-a-m-l's suggestions? 😄

Closing due to lack of a response.


refactor : migrated code to mvvm architecture

Fixes #231
Changes:

The Codebase is now migrated to MVVM Architecture
MVVM got rid of the PreviewChangedListener
Observables help us manage data change within our app without using listeners
File Hierarchy is now according to the MVVM Architecture
We use LiveData wherever possible

Additional Context:
Here is the diagram I tried to follow while following this architecture

@yashk2000 @anhanh11001 @ShridharGoel @liveHarshit @yatri1609 @naman653 @cweitat Please Review it and let me know if I missed anything or any thoughts on improving this PR.

@liveHarshit great, I've mentioned them inside the message and also updated the pr name accordingly

@iamareebjamal please review.

Very unrelated changes making it difficult to review

@adityastic Can you please solve different issues in different PR. Also, you have not any issues about these changes.

@iamareebjamal @liveHarshit sure, Let me separate everything other than MVVM from this PR

@iamareebjamal I've updated the PR with the proper changes sir, also included a diagram which was used as a reference. @liveHarshit I talked about this in the comments of #219, Sorry for not opening a separate issue after the same. I've opened the specific issue and referenced it as well.
Please Review
Regards

@iamareebjamal IMO, Codacy PR review should integrate for this project?

@liveHarshit that would be great, will help us to improve our code quality

@iamareebjamal What is the codecy logic for showing undocumented public classes and functions?

@adityastic Do not use unsafe casting, you can use kotlin smart cast instead or avoid casting by changing in logic.

@adityastic Do not use unsafe casting, you can use kotlin smart cast instead or avoid casting by changing in the logic.

@iamareebjamal What is the reason for showing undocumented public classes and functions in codacy review?

@liveHarshit Done, fixed them. Have a look

@iamareebjamal please review sir

@iamareebjamal @liveHarshit Codacy/PR specified the suggestions errors:

TooManyFunctions - Interfaces and Activities need to have those functions, If I even try to split the functions to two interfaces it will still end up in the activity which also has this suggestions
LongParameterList - Because the no of parameters is more than 6, Even If I create a data class with the same objects it will show these errors in those data classes.

What should I do about the Codacy/PR Quality Review?

What should I do about the Codacy/PR Quality Review?

Do not cast. And for long parameters use a new line. Reset good. :+1:

@liveHarshit on it. Just a min

Even If I create a data class with the same objects it will show these errors in those data classes.

No , it won't

@liveHarshit Build was Successful, Please Review. @iamareebjamal you are amazing sir, it worked like a charm, thanks for the suggestion, Ive updated the PR please review

@iamareebjamal please review sir, peer review done


Update Spring Cloud Azure version to 4.4.1

As title.

close and work on https://github.com/Azure-Samples/azure-spring-boot-samples/pull/542

We should merge to spring-cloud-azure_v4.4.0 branch instead of main branch.

You meant the 4.4.1 branch, right?

You meant the 4.4.1 branch, right?

@saragluna , yes, create a new branch named spring-cloud-azure-v4.4.1 from spring-cloud-azure-v4.4.0. Here is the PR: https://github.com/Azure-Samples/azure-spring-boot-samples/pull/542


refactor(rust): change configuration get to delegate error mapping

Current Behavior
The current implementation is calling std::process::exit, which is something we want to stop using to handle errors properly and give the user a better-formatted output when a command fails.
Proposed Changes
This PR refactors configuration set command to run run_impl and handle all the errors propagated through it in a single place
Close #3557
Checks

[x] All commits in this Pull Request are signed and Verified by Github.
[x] All commits in this Pull Request follow the Ockam commit message convention.
[x] I accept the Ockam Community Code of Conduct.
[x] I have accepted the Ockam Contributor License Agreement by adding my Git/Github details in a row at the end of the CONTRIBUTORS.csv file in a separate pull request to the build-trust/ockam-contributors repository.

@mergifyio rebase


Bug: Validation error "Error: Error: undefined"

What happened?
It  is not a bug
calling the get-credentials command from previous step will preserve the context and authorization
az aks get-credentials .....
I had this working before but today I get error , the testazure.yaml file is working fine from local machine using kubctl apply command
But the github action is failing with this validation error I also don't know how to disable the validation on the action
error validating "/tmp/testazure.yaml": error validating data: failed to download openapi: Get "http://localhost:8080/openapi/v2?timeout=32s": dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false

Version

[X] I am using the latest version

Runner
Github Actions
Relevant log output
error validating "/tmp/testazure.yaml": error validating data: failed to download openapi: Get "http://localhost:8080/openapi/v2?timeout=32s": dial tcp [::1]:8080: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
Error: Error: undefined

Hello @pyramidsbuilder how did you resolve it? I'm facing the same issue ....

@Duri9292 you should check your inputs (secrets) at the set-context step in your action.k8s-url & k8s-secret


Observability OpenTelemetry update

Pull Request Template
What are you trying to address

General update to the Observability section of the engineering playbook related to OpenTelemetry concepts, patterns, guidances, best practices and recommendations.
This PR closes issue #735

Description of new changes

General update to the Observability section of the engineering playbook related to OpenTelemetry concepts, patterns, guidances, best practices and recommendations.
Added new quick sections to relevant existing Observability patterns or practices that are solved by using OpenTelemetry

For all pull requests

[X] Link to the issue you are solving (so it gets closed)
[ ] Label the pull request with the appropriate area(s)
[X] Assign potential reviewers (you may also want to contact them on Teams to ensure timely reviews)
[ ] Assign the project - Engineering Playbook Backlog
[] Assign the pull request to yourself

:shipit:


New version of the cling patches

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

Failed to even lint the recipe, probably because of a conda-smithy bug :cry:. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

cc @chrisburr  this works locally (except for the startwith issue on config values)
I wish the root and cling folks used a single set of patches, but unfortunately, there is no common set that works.

I'll take a look locally at the startswith issue. (Btw you'll need a rerender to pick up the changes to the build config.)

I wish the root and cling folks used a single set of patches, but unfortunately, there is no common set that works.

Me too...I'm trying to push for it when they update from LLVM 5 but we'll see (which is increasingly urgent due to issues GCC 9).

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

It looks like the 'clang_variant' output doesn't have any tests.

@chrisburr tests appear to be passing.
Leving this open for a bit to give you a chance to comment!

Are these variant clangdev packages used as static libraries or as tools? I'm wondering whether there's any benefit to having variants

They're used as static libraries

If they are used as static libraries, why create variants? ROOT can build this in their build system right?

If they are used as static libraries, why create variants? ROOT can build this in their build system right?

Cling also uses such a variant. It seems to me that it would complicate the build quite a bit to have to build clang completely as part of cling.

now the missing libarchive bug.


ListFolderResult / MetaData not flagging if deleted items are files or folders.

When returning ListFolderResult, or more specifically the MetaData inside ListFolderResult.Entries, if the item in question is marked as IsDeleted then both IsFolder and IsFile are both marked as FALSE, making it impossible to determine if the deleted item is a file or folder.
If the item is not marked as IsDeleted then IsFolder and IsFile are correctly set.
I assume this is not by design but a bug.
Can anyone replicate this and confirm / fix. Or if this is by design how would you determine between a deleted file and a deleted folder?

This is the expected behavior. The IsDeleted, IsFile, and IsFolder methods allow you to check which subclass a particular Metadata object is. That is, it will be only one of DeletedMetadata, FileMetadata, or FolderMetadata respectively.
The API doesn't offer the ability to tell if a DeletedMetadata was previously for a folder or a file, but I'll pass this along as a feature request.
Note that the path for any particular DeletedMetadata could have referred to both a file and folder at different points in the past though, so you can check the file revisions or the metadata your app previously received for more information, depending on what your app is looking for exactly.


Registration Form 30 - Too Strict of Test

Describe the Issue
The test for this challenge is a bit too strict with whitespace.
Affected Page
https://www.freecodecamp.org/learn/2022/responsive-web-design/learn-html-forms-by-building-a-registration-form/step-30
Your code
        <label>
          How did you hear about us?
          <select>
            <option></option>
            <option></option>
            <option></option>
            <option></option>
            <option></option>
          </select>
        </label>

Expected behavior
This code should pass
Screenshots
No response
System
All
Additional context
No response

assert.match(document.querySelector('fieldset:nth-child(3) > label:nth-child(3)')?.innerHTML?.replace(/[\t\n]+/g, ''), /^How did you hear about us\?/);

Should be able to tighten up the replace regex. Or use trim.


Add new QBG methods to C API

Hello,
Following up on #136, in order to have a type-safe API in ngt-rs for QBG indexes, the following functions would be required in the C API:

ObjectID qbg_append_object_as_uint8(QBGIndex index, uint8_t *obj, uint32_t obj_dim, QBGError error);
uint8_t* qbg_get_object_as_uint8(QBGIndex index, ObjectID id,  QBGError error);

On a related topic, I have questions about the QBGConstructionParameters:
What is the internal_data_type  ? Should it always be the same as data_type ?
Similarly, what is extended_dimension compared to dimension ? Are there some rules between them ?

Hello,
V2.0.13 has been released and includes the C APIs you requested.
internal_data_type is the vector data type in the graph for QBG. You should always set float to internal_data_type and data_type, because testing against other data types is not sufficient.
dimension must be set as the dimension of the given data. When the rotation is enabled, the given vector can be extended inside QBG to improve quantization errors. extended_dimension should be equal to or larger than dimension, and should be a multiple of 16.

Hello,
Thank you for the updates and the explanations.
I was able to use qbg_append_object_as_uint8 and qbg_get_object_as_uint8 on my side.
However I am not sure that I understand your answer regarding data_type/internal_data_type. If I look at the C API I can see that QBGConstructionParameters is initialized as follows:
void qbg_initialize_construction_parameters(QBGConstructionParameters *parameters)
{
  // ...
  parameters->internal_data_type = NGTQ::DataTypeFloat;
  parameters->data_type = NGTQ::DataTypeFloat;
  // ...
}

So I thought that we could also use NGTQ::DataTypeUint8 to create a QBG containing uint8_t objects. Is this correct ?
Or is the only possible value NGTQ::DataTypeFloat ?
Actually, if I try to set them both to NGTQ::DataTypeUint8 I get a SIGABRT with the following gdb backtrace:
Thread 9 "qbg::index::tes" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffdbffd6c0 (LWP 221840)]
0x00007ffff729f26c in ?? () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff729f26c in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff724fa08 in raise () from /usr/lib/libc.so.6
#2  0x00007ffff7238538 in abort () from /usr/lib/libc.so.6
#3  0x00005555557ae73a in NGTQ::QuantizerInstance<unsigned char>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) ()
#4  0x0000555555769b75 in NGTQ::QuantizerInstance<unsigned char>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, NGT::Property&, bool)
    ()
#5  0x000055555578567f in NGTQ::Index::getQuantizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, NGT::Property&, bool) ()
#6  0x0000555555785b38 in NGTQ::Index::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) ()
#7  0x0000555555785c1e in QBG::Index::Index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) ()
#8  0x0000555555785ff7 in qbg_open_index ()
#9  0x00005555555c8eef in ngt::qbg::index::QbgIndex<u8, ngt::qbg::index::ModeWrite>::create<u8, &std::path::Path> (path=..., create_params=...) at src/qbg/index.rs:50

Hello,
NGTQ::DataTypeUint8 is for the obsolete module NGTQ. I will consider reimplementing uint8 as a data type for QBG in the near future.

Hello,
I have released v2.0.15 which includes reimplementation of uint8.
data_type is the type of the stored objects. internal_data_type is the type of the objects on the graph used by QBG.  You can specify different data types for them.

Thanks a lot, it works fine now !
Looking at your commit it seems that QBG can also handle float16 ? Does QG handle float16 too ?

Since I found that v2.0.15 does not work well in some cases, I have released v2.0.16 to fix it.
QG and QBG can handle float16 as well.

Thank you for the precision.
Then I guess some more functions would be needed in the C API for float16, I will list them in #141


Change vite hotfile location

Hello
Is it possible to change the hotfile location for a module?
I see in helpers it points to storage() which points to the root laravel storage path. I would like a separate hot file in my module .
I did this and it worked for me but want to know if there is another way:
if (! function_exists('module_vite')) {
    /**
     * support for vite
     */
    function module_vite($module, $asset): Vite
    {
        return ViteFacade::useHotFile(module_path('Retail','storage/vite.hot'))->useBuildDirectory($module)->withEntryPoints([$asset]);
    }
}

I think that's a good solution, another way would be when your module used its own vite config something like this may work
import { defineConfig } from 'vite';

export default defineConfig({
  server: {
    hot: true,
    host: 'localhost',
    port: 3000,
    hotFile: 'storage/vite.hot', // Custom path for the hot file
  },
  build: {
    outDir: 'public/build',
  },
});


Install Failures on Termux

Bug Description
I was unable to install a pyo3 pip package on termux without manually providing a config file. For some reason the library parameters were not set. From https://github.com/PyO3/pyo3/issues/1800#issuecomment-1106749656
Steps to Reproduce

Install termux on android
In termux, install python and rust: pkg install python rust
In termux, install based58: pip3 install based58 or manually from https://github.com/kevinheavey/based58 (pkg install git)

Backtrace
$ PYO3_PRINT_CONFIG=1 maturin build
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
📦 Built source distribution to /data/data/com.termux/files/home/srctmp/based58/target/wheels/based58-0.1.0.tar.gz
   Compiling pyo3 v0.15.1
   Compiling pyo3-macros v0.15.1
error: failed to run custom build command for `pyo3 v0.15.1`

Caused by:
  process didn't exit successfully: `/data/data/com.termux/files/home/srctmp/based58/target/debug/build/pyo3-c92d6bef7e401530/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.7
  shared=true
  abi3=true
  build_flags=WITH_THREAD
  suppress_build_script_link_lines=false
warning: build failed, waiting for other jobs to finish...
error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --lib --`

Your operating system and version
TERMUX_APK_RELEASE=F_DROID TERMUX_APP_PID=16828 TERMUX_IS_DEBUGGABLE_BUILD=0 TERMUX_MAIN_PACKAGE_FORMAT=debian TERMUX_VERSION=0.118.0 Packages CPU architecture: aarch64 Subscribed repositories: # sources.list deb https://termux.mentality.rip/termux-main/ stable main # x11-repo (sources.list.d/x11.list) deb https://packages.termux.org/apt/termux-x11 x11 main # sources.list.d/pointless.list deb https://its-pointless.github.io/files/24 termux extras Updatable packages: termux-keyring/stable 3.3 all [upgradable from: 3.2] Android version: 8.1.0 Kernel build information: Linux localhost 3.18.71-perf-gc7467a0 #1 SMP PREEMPT Thu Jan 24 02:00:44 EST 2019 aarch64 Android Device manufacturer: BlackBerry Device model: BBB100-1
Your Python version (python --version)
Python 3.10.4
Your Rust version (rustc --version)
rustc 1.60.0
Your PyO3 version
pyo3 = { version = "0.15.1", features = ["extension-module", "abi3-py37"] }
How did you install python? Did you use a virtualenv?
apt
Additional Info
This might be a bug with based58 . I'm not sure.
I tried mutating the Cargo.toml line and encountered different sets of errors. Providing a manual config based on the output of manually running cargo let me install the package successfully, as described in the linked origin issue.
The way Termux works, the root system and all libraries are installed relative to a prefix local to the application data folder. This path is stored in $PREFIX and is almost always /data/data/com.termux/files/usr . In order to run conventional shell scripts with #!/usr/bin/foo lines, Termux keeps LD_PRELOAD filled with a dynamic library called termux-exec that mutates this behavior when the scripts are executed, so as to find their launching processes relative to the prefix. Everything in Termux is installed without root.

Thanks @xloem. Can you share the custom config file you needed to create as your workaround?

Sorry for leaving that out.
$ PYO3_PRINT_CONFIG=1 cargo build

implementation=CPython
version=3.7
shared=true
abi3=true
lib_name=python3.10
lib_dir=/data/data/com.termux/files/usr/lib
executable=/data/data/com.termux/files/usr/bin/python
pointer_width=64
build_flags=WITH_THREAD
suppress_build_script_link_lines=false

Oops fat finger error!
@xloem can you try the latest maturin beta (0.12.13b1)? I think with recent changes credit to @ravenexp and @messense this might actually just work.

(to clarify latest maturin beta, without your config file. There was a change that allows PyO3 to look for Python in more cases and it might have happened to fix this case.)

@xloem can you try the latest maturin beta (0.12.13b1)? I think with recent changes credit to @ravenexp and @messense this might actually just work.

That will only work if the upstream package upgrades from PyO3 v0.15.1 to v0.16.x

implementation=CPython
version=3.7
shared=true
abi3=true
lib_name=python3.10
lib_dir=/data/data/com.termux/files/usr/lib

Can we even support abi3 on Android given that unlike other Unixes, the libpython DSO must be linked explicitly by the extension modules, but nothing like python3.dll exists for Unix-like platforms?

That will only work if the upstream package upgrades from PyO3 v0.15.1 to v0.16.x

Ah darn yes, I missed that.

Can we even support abi3 on Android given that unlike other Unixes, the libpython DSO must be linked explicitly by the extension modules, but nothing like python3.dll exists for Unix-like platforms?

I think so? My understanding is that as long as the wheel links against libpython.so or libpython3.so or similar which doesn't have a version-specific name, then at dynamic load time we should be able to load the abi3 symbols from any library with a matching name. I guess we might need to introduce some logic in pyo3-build-config to ensure this?

Oh, I was wrong, there is libpython3.so, which is not a symlink to libpython3.y.so, as I though before:
Dynamic Section:
  NEEDED               libpython3.10.so.1.0
  NEEDED               libc.so.6
  SONAME               libpython3.so

and it's even mentioned in PEP 384.
So yes, the build script logic can be altered to link to libpython3.so instead.

upgrading from 1.15.1 to 1.16.4 moves the error from pyo3 to pyo3-ffi (which also shows 1.16.4)
upgrading maturin to 0.12.13-beta.1 with upgrading pyo3 does not appear to change the error
upgrading both eliminates the lib_name error. the compilation fails due to a different error. I'm not sure it's always occurring:
$ maturin build

    Finished dev [unoptimized + debuginfo] target(s) in 1m 37s
💥 maturin failed
  Caused by: Cannot repair wheel, because required library libm.so could not be located.

$ maturin pep517 build-wheel
💥 maturin failed
  Caused by: Cannot repair wheel, because required library libdl.so could not be located.

I guess Termux environment isn't manylinux compatible, so you should run maturin build --skip-auditwheel.

I'm sorry, I'm having trouble testing this, and I'm kinda frustrated and confused (phone keeps running out of space which disables internet, reinstalled rust repeated ...). The errors haven't quite seemed deterministic for me. I did get a non-wheel build to succeed repeatedly using the upgraded libraries. I haven't reproduced this today.
However, performing both upgrades appears to resolve the lib_name error.

No problem, thanks very much for testing & sharing all this information with us.
Hopefully what this concludes is that once the ecosystem has had a chance to update, this class of problems is resolved. For now the custom config file will work for you at least 👍

I guess Termux environment isn't manylinux compatible, so you should run maturin build --skip-auditwheel.

@messense is that true of all builds to Android? (Should maturin turn off wheel auditing if the android target is detected?)

is that true of all builds to Android?

Well I don't really know.

Should maturin turn off wheel auditing if the android target is detected?

It was already disabled when invoking with pip wheel which should work for the normal pip install process. I don't think people should run maturin build on deployment target unless they're debugging though.

Arright, everything works with --skip-auditwheel. I had to update both the rust buildfile to reference pyo 0.16.4 and the python buildfiles to reference maturing 0.12.13-beta.1 .


CPU client and no-chargenet reco

Includes new cpu_client for reco without the llh service. Also includes functionality to run reco without chargenet if desired.

looks like this can be safely merged (?)

Yes; I'll go ahead and merge it


Modify Integration Tests after Disabling Cross Tenant Token Introspection

This PR modifies the product-is integration tests after the cross tenant token introspection is disabled. With this config disabled, an access token of a tenant cannot be introspected using the supertenant introspection endpoint. The tenanted introspection endpoint should be used instead.
Related issue: https://github.com/wso2/product-is/issues/14270

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2783645883

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2784559871

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2783645883
Status: failure

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2784559871
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2786395595

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2786395595
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2786582624

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2786688490

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2786582624
Status: cancelled

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2786688490
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2790966834

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2790966834
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2791019010

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2791019010
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2791847734

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2791847734
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/2793610209

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/2793610209
Status: failure

The PR builder failed only because of the test failure: org.wso2.identity.integration.test.rest.api.user.approval.v1.UserMeApprovalTest.testListTasksWhenAvailable(org.wso2.identity.integration.test.rest.api.user.approval.v1.UserMeApprovalTest).
Since this is an intermittent test failure, merging the PR.


CanaryLinks: Threads keep their Canary links

How reproduce:

Open the Canary or PTB build of Discord.
Go to a server where you can make a thread.
Make a thread or find an existing one.
Right click the thread in the channel list, then click "Copy Link"
Paste the link somewhere and notice that the link is a canary link.

Thank you for the report, I'll fix it when the semester ends, im very busy atm!
@SnoozeThis wait for two weeks -> assign to @bepvte, remove label "snoozed"

invalid duration: not a number: two

(https://snoozeth.is/b0KUicDJqM4) I will wait until Thu, 05 May 2022 01:27:21 UTC and then assign to bepvte and remove label snoozed.

Resolved.


Doesn`t load the index page

I import all file to my host and change the config.php file with my db information, but no load the index page. An error comes out: "An Exception was thrown while handling: not found" when i search the url "localhost/project/warehouse-inventory-system/index.php" in browser.
Thankssss for all.

Most probably it was the problem with your configuration, not the script.

https://github.com/bitsandbots/inventory

the problem is in the file "load.php" in the folder "includes" just edit it and delete the space in line 1 of the code


🐛 BUG: vitest-pool-workers doesn't correctly resolve required modules if no extension is provided

Which Cloudflare product(s) does this pertain to?
Workers Vitest Integration
What version(s) of the tool(s) are you using?
0.4.25
What version of Node are you using?
20
What operating system and version are you using?
Mac
Describe the Bug
vitest-pool-workers can fail to apply the correct module resolution on require calls that don't include a file extension
For example me and @WalshyDev noticed this with the discord api types package, in which, the package.json includes both cjs and esm versions for the various exports but vitest-pool-workers doesn't seem to correctly take those into consideration causing errors when trying to import from the package.
Please provide a link to a minimal reproduction
https://github.com/dario-piotrowicz/vitest-pool-workers-ext-repro
Please provide any relevant error logs
No response

I think this is a duplicate of #6405 as well

I think this is the same issue as #6405


Adding additional features to messenger?

I'm currently using Alertify for simple error notifications and I'm considering switching to Messenger. However, I may actually need a library that does some things not in either in the future.
If I submitted pull requests adding features like these, which make Messenger a little more complex. Would they be accepted?

An optional image on the left, in place of the coloured circle
Support for icon sets such as Font-Awesome in that same space
Optional close on click behaviour (clicking dismisses the notification instead of a close icon)
Optional navigate/action on click behaviour (an action that's triggered when the notification is clicked, instead of presenting itself as a button or icon)

((I have a feeling I may be adding notifications such as "You have a new message from X" with the other user's avatar on the left of the notification to my application))

These all sound like wonderful additions to me, particularly if they’re all implemented optionally/progressive-enhancement-ally. PR away! And thx! :+1:

Incidentally, I believe you can do the last two using events.


Images (SVG) always maximally scaled

Hi there!
asciidoctor-pdf always scales small_image.svg to fit the width entirely into the pdf.
I see this as being ok (it is definitely better than the image-cut-off that one gets with asciidoctor-fopub asciidoctor/asciidoctor-fopub#43) ; but just note: it is not possible to decrease the width, should one like to do this.
Here is sample.adoc that you can test with html5, pdf (via asciidoctor-fopub), pdf (via asciidoctor-pdf)
[depends only of small_image.svg, that you can copy-paste from below]
= Test scaling of small image
:toc:

Here comes the image `small_image.svg`:

image::small_image.svg[]

== Scale the image (to make it easier to view)

image::small_image.svg[Small Image, width=1000]

[[small_image]]
[source,asciidoc]
image::small_image.svg[Small Image, width=1000]

=== Good for html5

The <<small_image,above scaled image>> is rendered nicely for html5

=== Bad for asciidoctor-fopub (docbook)

The <<small_image,above scaled image>> is clipped when rendered to pdf via asciidoctor-fopub

.Render to pdf via asciidoctor-fopub (docbook)
[source]
asciidoctor -b docbook -d book -a data-uri! sample.adoc
~/asciidoc/asciidoctor-fopub/fopub sample.xml

=== OK for asciidoctor-pdf (but always maximally scaled)

The `width`-scaling is ignored by asciidoctor-pdf:
there the <<small_image,image>> is always maximally scaled.

.Render to pdf via asciidoctor-pdf
[source]
asciidoctor-pdf sample.adoc

The scaling does not harmonize nicely between html5 and asciidoctor-fopub.

What fits nicely in html5, is unfortunately cut-off for pdf output

I believe there should be a possibility to scale to fit the width, but preventing cut-off of an image.
Source of image small_image.svg (for copy-pasting):
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!--  Currently unusable for asciidoctor-diagram. The diagrams require a constant dpi per document (a width in pt is provided in the svg's).

+1.

+1, running into this issue using asciidoctor-diagram to generate svgs.

I'll move this to 1.5.0 since it seems to be a showstopper for more and more users.

It's important to understand that there is a way to control the width of SVGs in PDF. It's just that the name of the attribute is different (and has slightly different semantics).
When converting to PDF, you should use the scaledwidth attribute instead of (or in addition to) the width attribute. The scaledwidth attribute takes precedence in the PDF backend.
There is one major different with scaledwidth though. The value is a percentage width of the bounds (available width) rather than an absolute value. The reason for this is both historical (compatibility with AsciiDoc Python) and because a PDF can have vastly different scaling and therefore it helps to think in these terms for consistency.
Here's how you would write your image macro so that the SVG is not maximally scaled.
image::small-image.svg[Small Image, 600, scaledwidth=50%]

That at least addresses the title of this issue, which is that this value cannot be controlled.
Now, I realize that we still don't have all the necessary controls, so I'd like to proceed by making sure we have a way to handle the essential use cases for scaling images in the PDF.

Looking at it closer, I now see that scaledwidth doesn't work if width is set when using the DocBook toolchain (a2x, fopub). This should actually be fixed by what we put into the DocBook output...but that's a separate issue.

I set the pdfattribute when building a pdf.

You can also check for backend-pdf, which is set when using Asciidoctor PDF.

I did some testing and it appears that Asciidoctor Diagram will pass through the scaledwidth attribute. Thus, to get a usable result for Asciidoctor PDF, please use:
[graphviz, graphviz-example, svg, scaledwidth=15%]
....
digraph g {
    a -> b
    b -> c
    c -> d
    d -> a
}
....

Asciidoctor PDF will ignore the width attribute if you set it (as it is intended for HTML). The DocBook toolchain (fopub and a2x) will not, so you need to leave it out when doing PDF through that toolchain.

...and when using SVGs, you are correct that the width attribute is ignore completely by Asciidoctor PDF.

I did some more thinking about this in the following thread.
http://discuss.asciidoctor.org/Unit-of-measure-for-image-dimensions-tp3040p3222.html
I think the solution is that we need width attributes per backend. It's kind of like scaledwidth, except that it's not changing the semantics at the same time.
The current width and height attributes in the image block are meant to be metadata only. They communicate to the processor the original rasterized dimensions of the image (or, in the case of SVG, the original viewport). These values are pixels.
We then need a way to control the scaling of the image. I propose the following attributes:

csswidth
cssheight (however, this breaks responsiveness of the image, so I recommend against using it)
pdfwidth
pdfheight

These attributes can take a value and a unit (e.g., pdfwidth=15cm, pdfwidth=50%). This way, we have a very clear way to control the width on different canvases.
What do you think about the css and pdf prefixes? Should we use screenwidth and printwidth instead? Any other ideas?

I want to clarify why the code behaves the way it does currently to help you think about a solution. I decided not to honor the width attribute for SVGs since the width is usually set for the HTML backend and is assumed to be a pixel value. The PDF canvas is in points and usually a very different size than the HTML canvas. Thus, you can't simply use that value in the PDF directly.
What we could do is if scaledwidth is not set (and pdfwidth isn't set either), we could take the width value and convert it from px to pt. Then, we could use that calculated value. wdyt? Good short term solution? I could try it, see how it goes. (Btw, I'd do the same thing for all image types).

I've proposed a fix. See https://github.com/asciidoctor/asciidoctor-pdf/pull/218
The fix does a couple things.

If the pdfwidth attribute is specified, it's assumed to be a measurement value, with or without units (e.g., 150pt, 5in, 200); default units is pt
Otherwise, if the scaledwidth attribute is specified, it's assumed to be a percentage width of the current bounds; note that pdfwidth can also be a percentage with same behavior
Otherwise, if no width attribute is specified, it uses the native pixel width of the image (after scaling pixels to points by multiplying value by 75%)
Otherwise, if the width attribute is specified, this value is used (after scaling pixels to points by multiplying value by 75%)

What I have not yet done is scale down the image to fit the page if no width is specified. Perhaps we need a maxwidth attribute (and/or maxpdfwidth).

To align with behavior we added to the listing block, perhaps instead of maxwidth, we should honor the autofit option for this purpose.
[%autofit]
image::small_image.svg[width=1000]

To clarify, autofit is assumed to mean scale down in this context, not scale up.

Another option is to scale the image down to fit on the page if only the width attribute is specified. If the pdfwidth attribute is specified, then that value is used as is, even if it makes the image larger than the page.

I decided to implement such that if neither the pdfwidth or scaledwidth attributes are specified, the maximum width of the image is the width of the current bounds / box. The pdfwidth and scaledwidth attributes can be used to override this behavior with an explicit value.

should the scaledwith attribute affect the HTML output as well? Both scaledwidth and pdfwidth (used on their own) appear to scale as expected in the PDF output, but I am not seeing anything in the HTML output to scale the images. Ideally, I want scaledwidth="80%" to scale the image to 80% of the text column width in all output forms.

should the scaledwith attribute affect the HTML output as well?

Not using scaledwidth. scaledwidth was really only introduced for the DocBook toolchain where this value had meaning.
You can set a percentage width for HTML using the width attribute.
image::diagram.png[width=80%]

However, technically this produces invalid HTML because as of HTML5, the width value can only be unitless (CSS pixel). We don't currently accommodate percentage scaling any other way in the HTML backend. Something to discuss.

I want to mention that I missed a use case with SVGs. If a width is set in the SVG and it's wider than the page, than the image is not being fit to the page. I didn't realize that Prawn SVG wouldn't handle this case. I've filed a follow-up issue: #242.

I've found that just using [width=x%] does the right thing for both HTML and PDF PNG diags. SVGs are still broken in terms of colour and scaling.

I'm confident that the fix for #242 will address the remaining issues.

I would still be interested in being having landscape pages in the PDF output for wide diagrams that have been made with that intention - which is pretty common. Since some diagrams might simply be wide because they were obtained in some strange way (e.g. very high-res screenshot), there's no way to infer whether a landscape page would be needed or not. Therefore it would have to be indicated explicitly, perhaps in the image macro, e.g. 'landscape'.
In Frame, I always did diags on landscape pages such that the diag takes the entire page, so no need to worry about text flows or complicated measuring - just inject a new landscape page and put that diagram and its caption on it. Actually, it's not quite as simple as that, because you may have blank space remaining on the current portrait page at the point you hit the image macro for the landscape diagram. There are two ways to deal with this:

always fill up the remainder of the portrait page in the normal way, then process any pending landscape page image directives
allow images to be declared 'floating' or not. Floating images is the normal default you want, so If there is nothing special in the macro line, do 1. above; but if there is a (say) 'nofloat' parameter, output the landscape page immediately with the diagram + caption, and then keep going.

For interactive HTML, if SVGs are used, large images can be obtained by the normal means in a browser. Concepts like 'nofloat' and 'landscape' can be ignored (although if you wanted to be super smart, 'landscape' might lead to a button being added to pop out the image in original size, or some pre-defined wider size).
The more I think about it, the more I think landscape images with float/nofloat need to be supported in PDFs, as this is extremely common in publishing and documentation. Shall I raise a new issue on that?

Shall I raise a new issue on that?

Please do. This is very useful analysis. I like each issue to be very focused on a single task. That gives the task the best chance to be addressed and helps keep the conversation on track.


seeds initial

hi，author.
Sorry to bother you so much. I would like to inquire about how to obtain your initial seed?  For example:  live555 seeds

Hi,
Please follow the tutorial in our README.md file. Basically, you capture the real traffics and turn them into seeds using wireshark.
Thuan


goto definition does not work to func or value in html tags

We temporarily closed this due to the lack of enough information. We could not identify whether it was a bug or a userland misconfiguration with the given info.
Please provide a minimal reproduction to reopen the issue.
Thanks.

Why reproduction is required


⚠ Fortnite has degraded performance

In b417de8, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1750 ms

Resolved: Fortnite performance has improved in 6554780.


Add support for existing BigQuery Schema (replace #40)

This takes the code proposed in PR #40 and attempts to address all outstanding items of concern as they were reviewed by @bxparks. #40 is very stale at this time and it does not seem that will get implemented at any point. I would like to request we close at #40 in favor of this PR but we should use that as a base of what changes I made since you last reviewed that PR.
The changes I have made as requested are:

Extend testdata structure to enable interpreting the existing schema as a string
commit id: 8fbe52d0210867ec319c4a47fb9c9cd0d1ef1026
Convert added pytest tests to normal unittest (replaced fixtures with nested for loops which is worse but satisfies not using a different test suite)
commit id: 1e286f14269066ce55628e143790a05f0a5a7b38
Delete pytest dep, and delete tox
commit id: 1e286f14269066ce55628e143790a05f0a5a7b38
Add --existing_schema command line flag for reading in existing schema from a file. (super easy)
commit id: 7afdf419a2c4a61c3b356d1b59ad5baaed8cc171
Write some tests that include an existing schema in testdata.txt
commit id: fff6f5bf61a69edf67a1d19106dc5f42ef71cab4
Remove ERRORS INFORMED from the data_reader. This didn't make sense to me and I did not see the value. All errors should be part of the error_logs that are returned.
commit id: fff6f5bf61a69edf67a1d19106dc5f42ef71cab4
Remove type_mismatch_callback. This was not properly tested and I didn't feel like maintaining/testing it.
commit id: c5a57de54515e01de649f2705000a358f9688355
Fix bug as it related to bigquery not providing a mode for some nested JSON fields by defaulting to NULLABLE
commit id: 96ca4ae148f6aa899665222d1c0e5a96968ecaae
Not requested but while updating the README I added the bash syntax flag to the generate-schema command examples

Summarizing original changes from @bozzzzo.

Break sanitize_names into it's own function and call this earlier in the schema generation process, not just during flattening.
Add ability to begin from an existing bigquery table schema by being able to reconstitute a schema_map from an existing bigquery schema.

@bxparks This is ready for review. Happy to have your feedback and incorporate anything else that is needed to get this merged. I know this is a big change but it's a particularly useful one when you are loading data into existing tables.

@abroglesc Thanks for picking this up!

When is this expected to be merged and in the stable release?
we are working with big query in an environment where the schema can change every day, and this solution is perfect for us.

Trying to get it done this week. Currently working through some other bugs related to how I am using the library then will focus back to getting this PR polished up (Since right now I am just vendoring this branch in my project to have it work)

One part that I don't actually like currently about this change is the aggression in converting to lower for the names, I actually think I am going to see if I can resolve name collisions without forcing to lower for the final schema. I don't like that we are now changing the casing on new schema elements that are inserted. (e.g. eventName becomes eventname when run through this now).
We previously didn't have any tests which included capital letters in the source data or schema. We should not be changing the keys of the incoming source data (unless there is a collision, then we should use the original/existing name)

@abroglesc: With regards to the handling of the case, I agree we should be case-preserving, not case-insensitive, so we shouldn't be blindly lower casing everything. I think the solution is in the schema_map dictionary. The field name is stored in 2 places, the schema_map.key and schema_map.info.name. I think if we store the lower cased version in the dict key, and preserve the case in info.name, we can preserve the case.

@ZebraCat: If you need this right away, you can clone this repo, create a branch, and do a git merge --squash this branch into your branch and use the code from there.
@abroglesc: FYI, I'm eventually going to squash merge this into develop, instead of creating a merge commit, to reduce the huge amount of clutter in the git history. I don't normally use squash merge on GitHub, so I don't what that does to the authorship info, which I would like to preserve. I would have preferred the commits from @bozzzzo to be rebased off the develop branch, then have your changes added on top of that. But when I tried that, there were so many merge conflicts, it wasn't worth the effort.

Ahh apologies, @bxparks I don't think github lends itself nicely to multiple people contributing to a PR unless they are given write permissions on the original repository so that has caused some issues here as it relates to how to continue with @bozzzzo's PR.
As for the using the key and the name, that sounds like a good plan.

Fixed the casing issue. Added a couple tests for that as well. Just need to address the rest of the comments. Overall the logic fixes are mostly implemented just small changes left.

Hi, Just want to check in to get a status on this PR. The sooner we get this in the better, because it takes a fair bit of effort to pull in all the context info into my head to allow me to review this code properly. I don't work on this code frequently, so if we wait too long, I start forgetting things.
BTW, there's a Makefile checked in at the root. You can run the unit tests and check flake8 errors by typing make at the top. You don't have to wait until you push tot GitHub.

Hi @bxparks,
I will try to have this all addressed today. I had a bit of time off recently where I disconnected and just getting back into the swing of things.

@bxparks re: using Makefile for checking Flake8 I started doing that but it wasn't scanning tests/ folder whereas the CI/CD in github was. This was added to the task in commit 9829bb0

@bxparks At this time I believe I have addressed all of your comments. Let me know if there is anything else we need to get this PR merged :)

I now truly believe this is ready for a final review after addressing the one final comment I missed.

Superceded by #61, closing.


Is there a file size limit for the JS SDK?

const fs = require("fs");
const { tableFromIPC, RecordBatchReader } = require("apache-arrow");

const filePath = "./arraydata.arrow";

const stream = fs.createReadStream(filePath);
const reader = RecordBatchReader.from(stream);

(async function () {
  const table = await tableFromIPC(reader);

  console.log("numRows", table.numRows);
  console.log("first row", table.get(0).toArray());
})();

The code above prints:
numRows 140000000
first row [ undefined, undefined ]

numRows is correct, but the values are coming out as undefined.
./arraydata.arrow is generated by the following python code.
import pyarrow as pa

nums1 = [42]
nums2 = [42.42]
mil = 1000000

for n in range(1, 140 * mil):
  nums1.append(n)
  nums2.append(1 / n)

arr1 = pa.array(nums1)
arr2 = pa.array(nums2)

schema = pa.schema([
  pa.field('nums1', arr1.type),
  pa.field('nums2', arr2.type),
])

with pa.OSFile('arraydata.arrow', 'wb') as sink:
  with pa.ipc.new_file(sink, schema=schema) as writer:
    batch = pa.record_batch([arr1, arr2], schema=schema)
    writer.write(batch)

If I lower the number of rows so that the file gets smaller then the values are coming out just fine. Is there a limit?

const fs = require("fs");
const { tableFromChunks, RecordBatchStreamReader } = require("apache-arrow");

const filePath = "./arraydata.arrow";
const chunkSize = 100000; // Set an appropriate chunk size based on available memory

const stream = fs.createReadStream(filePath);
const reader = new RecordBatchStreamReader(stream);

(async function () {
  let chunks = [];
  for await (const chunk of tableFromChunks(reader, chunkSize)) {
    chunks.push(chunk);
  }

  // Concatenate the chunks to form the complete table
  const table = chunks.reduce((acc, chunk) => acc.concat(chunk));

  console.log("numRows", table.numRows);
  console.log("first row", table.get(0).toArray());
})();


Failed to apply EXTERNAL delegate or  Failed to apply XNNPACK delegate.

Hi,
I have compiled a custom mobilenet model to TFLite and when I tried to benchmark the model I am getting the next errors.
 ./benchmark_model --graph=mobilenet.tflite --num_threads=6 --use_xnnpack=true
STARTING!
Log parameter values verbosely: [0]
Num threads: [6]
Graph: [mobilenet.tflite]
#threads used for CPU inference: [6]
Use xnnpack: [1]
Loaded model mobilenet.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
XNNPACK delegate created.
Failed to apply XNNPACK delegate.
Benchmarking failed.

or
LD_LIBRARY_PATH=/mnt/sd/aarch64_build ./benchmark_model \
--graph=mobilenet.tflite --external_delegate_path="/mnt/sd/aarch64_build/delegate/libarmnnDelegate.so" \
--external_delegate_options="backends:CpuAcc" \
--num_threads=6
STARTING!
Log parameter values verbosely: [0]
Num threads: [6]
Graph: [mobilenet.tflite]
#threads used for CPU inference: [6]
External delegate path: [/mnt/sd/aarch64_build/delegate/libarmnnDelegate.so]
External delegate options: [backends:CpuAcc]
Loaded model mobilenet.tflite
INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
EXTERNAL delegate created.
Failed to apply EXTERNAL delegate.
Benchmarking failed.

When using ./benchmark_model --graph=mobilenet.tflite --num_threads=6 --use_xnnpack=false everything works well and I can launch the benchmark.
I am seeing the benchmark_tflite_model.cc code and the problem seems to be in interpreter_->ModifyGraphWithDelegate.
How can I debug this problem or know more about it and why is happening?
Thank you and regards.

Hi @jlamperez,
Thank you for reaching out. Your commands look fine. Issues in ModifyGraphWithDelegate are quite vague but they can be related to the model as seen here: https://github.com/tensorflow/tensorflow/issues/56166. Have you possibly tried any other models to eliminate this?
I don't think there is a way to enable more useful output when using the benchmark_model, but you should be able to run it in GDB. Also, if you would you be able to share you model, I can try and reproduce this for you. Thank you!
Kind regards,
Matthew

Hi @matthewsloyanARM thank you for your answer,
I have seen that for the first case when --use_xnnpack=true  tflite is compiled with now when using thesetup-armnn.sh
  cmake -DTFLITE_ENABLE_XNNPACK=OFF \
        "$target_arch_cmd" \
        "$TFLITE_SRC"
  cmake --build . -j "$NUM_THREADS"

So xnnpack is not available in tflite..
For the second one, I have added some log statements and saw that my model has some dynamic_tensor_index_.
Is Where and SparseToDense operators supported in tflite delegate? I don't see them in Supported operations. Is this documentation updated?
Thanks!

Hi @jlamperez,
You are correct, thanks for pointing that out. It looks like the XNNPACK delegate will be automatically applied when running the benchmark_model, unless --use_xnnpack=false is specified, which is why it works when added to your command. Hopefully this makes sense?
This following example from the running benchmark_model --help shows this --use_xnnpack=false = explicitly apply the XNNPACK delegate. Note the XNNPACK delegate could be implicitly applied by the TF Lite runtime regardless the value of this parameter. To disable this implicit application, set the value to false explicitly.
Regarding your second query, the issue you are having is likely due to the dynamic tensors as the support is quite basic, it's hard to tell though without seeing your model. Would you know what operators have dynamic tensors in your model?
Lastly, where and SparseToDense are currently not supported in Arm NN. I have added these two operators to our backlog but there is no current estimate on when they will be added.
Kind regards,
Matthew

Hi @matthewsloyanARM,
I see for the benchmarking documentation that  use_xnnpack: bool (default=false)  so when launching   ./benchmark_model --graph=model.tflite --num_threads=6 it makes sense to me that xnnpack is not used.
But then if TfLiteArmnnDelegate EXTERNAL delegate fails, why is XNNPACK delegate for CPU created and works the benchmarking if I have compiled tflite cmake with  -DTFLITE_ENABLE_XNNPACK=OFF? Shouldn't that fail since I don't have XNNPACK?
I have two models and the dynamic tensors problems are located in this part of the models:
Model 1

Model 2

The first one I don't know which can be the problem because I see that Reshape and Shape are supported.
But for the second one, I see the problems in Where and SparseToDense operations.
Thank you for being so supportive! Very appreciate

Hi @jlamperez,
It seems like the benchmark_model will try to fallback to XNNPACK regardless if it is enabled or not with TFLITE_ENABLE_XNNPACK. I think the best thing to do is to always disable it when running the benchmark_model too with --use_xnnpack=false. As I noticed that it would still be created for me unless I explicitly set this flag, even though the default is false. Can you add this to your Arm NN TfLite Delegate command to eliminate this as an issue?
Thank you for supplying the images, it looks like this should be supported as there's nothing that jumps out at me. As you mentioned the Where and SparseToDense aren't supported, but these two operators should fallback to the TfLite Runtime.
Would it be possible for you to try running these models on our own model runner tool called ExecuteNetwork, it might give a better error message and will eliminate any issues with the benchmark_model. It might be built by default when using the build-tool and can be located in build/(debug/release)/armnn/tests/ExecuteNetwork. If it doesn't you should be able to build it by adding this your build-armnn.sh command --armnn-cmake-args='-DBUILD_TESTS=1'. You can then run execute network on using the delegate like this ./ExecuteNetwork --compute CpuAcc --model-path model.tflite. I would also advise to try --compute CpuRef to see if you get any difference.
Kind regards,
Matthew

Hi @jlamperez,
I am going to close this issue due to inactivity. However, feel free to reopen it anytime if the issue still persists or if you have any questions. Thanks again for getting in touch!
Kind regards,
Matthew

Hi @matthewsloyanARM
No problem.
I tried some other models with the Arm NN TfLite Delegate and they worked.
Do you know when Where and SparseToDense are going to be supported?
Thanks!

Hi Jorge,
thank you very much for your feedback.
Those 2 operators are not part of our roadmap at the moment.
We invite you to contribute if you may want to do so. The process for contributing to ArmNN is outlined in this Contributor Guide.
You should create a Gerrit review account at mlplatform.org and make your contribution there. You can use your GitHub credentials when creating your account.
Kind Regards

Hi @TeresaARM,
I have two models and in both, I have this error
Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors

I have converted the models with static-sized tensors but I'm still having the same problems I had in https://github.com/ARM-software/armnn/issues/716#issuecomment-1431695672. Arm NN is not falling back to tflite, which is weird because the model is working for me when I don't use ArmNN delegate.
How can improve the debugging and how should I contribute with the non-supported WHere and SparseToDense layers?

Hi @jlamperez,
thank you for your message, If you are willing to contribute you can create a Gerrit review account at mlplatform.org and make your contribution there. You can use your GitHub credentials when creating your account. The process for contributing to Arm NN is outlined in this Contributor Guide.
Where and SparseToDense layers would not be very straight forward.

SparseToDense is not supported by Compute Library, therefore you would not benefit of any performance uplift in Arm Cpus and Gpus, just by adding the layer to Arm NN.

Where is a special type of layer, we do not support this type of layers yet, but it is in our 1 year roadmap to do so.

However, by using the TFLite delegate, the not supported layers should fall back to TFLite. Is it possible for you to attach the model in a message in this thread?
Kindest Regards


Command: vue-sfc

This command can run a vue file in browser directly.
But need a better way to test its props and events 🤔
Now idea is create a file
Use render function maybe better
// vue-sfc.config.js
export default {
  attrs: {...},
  props: {...},
  on: {...}
}

interface config {
  attrs?: any,
  props?: any,
  on?: any,
  [key: string]: any
}

// render in vue
{
  render(h) {
    return h('your component', {...})
  }
}

Please see: https://vuejs.org/v2/guide/render-function.html#The-Data-Object-In-Depth

Maybe use vue-dev-server https://github.com/vuejs/vue-dev-server will be better.