🚀 Feature: introduce symmetry to schema openapi generate-* commands

🔖 Feature description
backstage-repo-tools schema openapi generate command operates on the entire repository and doesn't take any arguments. backstage-repo-tools schema openapi generate-client expects a path to openapi schema and the root of the workspace where the code will be generated. This setup works fine if the development of the backend plugin starts before the frontend client. It's awkward if that work happens in parallel. It's not a significant issue, but making it feel more consistent would be nice.
🎤 Context
I had the pleasure of showing the new OpenAPI generator functionality to developers in our workshop, and it was very well received. Updating the schema and re-generating the client felt very natural. I wanted to share some of the bumpy parts of the process so we could iron them out.
✌ Possible Implementation
One possibility is to have a package generate command added to @backstage/cli that would invoke generate on every workspace that defines the script. In the workspace, we could specify backstage-repo-tools schema openapi generate client --input= --output= or backstage-repo-tools schema openapi generate server --input= ---output=.
👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
Yes I am willing to submit a PR!

Thanks for this! Couple of questions on the bumpiness and then I'll respond to the concerns.

This setup works fine if the development of the backend plugin starts before the frontend client. It's awkward if that work happens in parallel. It's not a significant issue, but making it feel more consistent would be nice.

Would this be helped if the generate-client command generated a full client package instead of just a generated folder? How much of DefaultApi.client.ts were you using?

This setup works fine if the development of the backend plugin starts before the frontend client.

Did you start with an OpenAPI schema? Or were you working iteratively across the full frontend/backend? I was working with Andre last week to try and migrate the Azure DevOps plugin to the OpenAPI tooling and found some issues on the iteration.

The generate-client command ended up being a little different than the existing generate command because we had to support the weirdness of the packages/catalog-client setup. I could definitely see some sane defaults being added though, like specifying a package instead of a path to a schema file and having the output directory default to plugins/{pluginId}-client with an option to override.
re: an overall generate command, I spoke with @Rugvip about something like this at KubeCon and I think it would be great. An interim step between now and then could be migrating the existing generate command to generate server and generate-client to generate client. A similar structure could be nice for the verify command as well for CI/CD. Alternatively, we could put these under a named argument on the generate command to make it a little less hectic.I like the idea of a package vs repo distinction as it's not super clear from the command itself which one does which. I could see generate being just package and verify being both.
I'm not completely sold that this should sit in @backstage/cli, as it's a bit more advanced and pretty use case specific than a lot of the commands there. It also adds some more versioning complexity that sounds painful.

Would this be helped if the generate-client command generated a full client package instead of just a generated folder?

Not necessarily. I think that use case could be covered under backstage-cli new but the fact that it only generates needed files allows us to re-run the same generator multiple times when openapi.yaml updates.

How much of DefaultApi.client.ts were you using?

All of it.

Did you start with an OpenAPI schema?

In this case, we were creating a full-stack plugin from scratch. We started with the UI, created an OpenAPI schema for it, implemented the UI, generated types from the schema, and created the ApiRef, hooked it up to the component.

Or were you working iteratively across the full frontend/backend?

We did a bit of iterating. We updated the schema halfway through, then re-ran the generator and updated the typescript to get it to compile.

I was working with Andre last week to try and migrate the Azure DevOps plugin to the OpenAPI tooling and found some issues on the iteration.

What were the issues?

I'm not completely sold that this should sit in @backstage/cli, as it's a bit more advanced and pretty use case specific than a lot of the commands there. It also adds some more versioning complexity that sounds painful.

Yeah, we don't need to worry about this right now. It might make sense to add generate to @backstage/cli in combination with caching - maybe Turbo Repo?

... that use case could be covered under backstage-cli new
Oh yeah, true.

How much of DefaultApi.client.ts were you using?
All of it.

Ah sorry, my question here was were you using DefaultApi.client.ts as the raw exported client? Or did you add a wrapper like in CatalogClient.ts?

In this case, we were creating a full-stack plugin from scratch. We started with the UI, created an OpenAPI schema for it, implemented the UI, generated types from the schema, and created the ApiRef, hooked it up to the component.

Cool! 🚀

What were the issues?

Mostly on the "generating a spec from test data" side. The whole flow is pretty bumpy and not something I tested out with the catalog when I was starting out. There are a couple of easy improvements and some longer term issues with how we're executing the subcommands.

It might make sense to add generate to @backstage/cli in combination with caching - maybe Turbo Repo?

Definitely a question for the core maintainers. If this tooling starts getting more traction, the caching is going to be really important.

were you using DefaultApi.client.ts as the raw exported client? Or did you add a wrapper like in CatalogClient.ts?

We were using the export directly. We didn't wrap it. The API Factory instantiated the class explored from the generated code.

Mostly on the "generating a spec from test data" side.

Oh, I see. Hmm... I didn't try this at all. Is this using Optic?

We were using the export directly. We didn't wrap it.

Oooo, cool, thoughts on the API? One object with body, query, path as typed subobjects.

Oh, I see. Hmm... I didn't try this at all. Is this using Optic?

Yup, it's more for migrating/bootstrapping existing plugins to the tooling by generating a basic spec from test data sent locally using supertest. Really cool stuff.

thoughts on the API?

It's a bit weird but not too weird. Where does that format come from? I assumed it was OpenAPI Tools. I assumed that I could just pass path & query together, but then types told me otherwise. I would expect something like getProducts(params, options).

Combining path and query seems odd to me, how would you handle name conflicts and whatnot.

I assumed that conflicts are not possible. Does OpenAPI allow params with same names in path and query?

I assumed that conflicts are not possible. Does OpenAPI allow params with same names in path and query?

Unfortunately yes, it's probably the weakest part of OpenAPI. Parameters are just an array of objects with an in: ... block. At the spec level, there's nothing stopping you from defining the same parameter twice with different types, which is not great. Tooling might prevent that, but it's a hard requirement to pin down especially across the ecosystem.


Small bank long run test is failing with Client encountered unexpected error

Small bank long run test is failing with Client encountered unexpected error. I am running the small bank test for 1hr with input tps 1000 and it is running for few minutes and exits with below error.
2020.03.02-17:45:47.700^[[31m error^[[39m [caliper] [defaultTest]       ------ Failed 'smallOperations' testing with the following error ------
Error: Client already exited
at ChildProcess. (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/client/client-orchestrator.js:279:41)
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
2020.03.02-17:45:47.704^[[31m error^[[39m [caliper] [defaultTest]       ------ Failed 'query' testing with the following error ------
Error: Client encountered unexpected error
at ChildProcess. (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/client/client-orchestrator.js:274:41)
at emitOne (events.js:116:13)
at ChildProcess.emit (events.js:211:7)
at ChildProcess.target.send (internal/child_process.js:582:12)
at ClientOrchestrator._startTest (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/client/client-orchestrator.js:151:24)
at ClientOrchestrator.startTest (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/client/client-orchestrator.js:72:20)
at DefaultTest.runTestRounds (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/test-runners/default-test.js:104:77)
at Object.module.exports.run (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/caliper-flow.js:113:47)
at
at process._tickCallback (internal/process/next_tick.js:189:7)
2020.03.02-17:45:47.704^[[32m info ^[[39m [caliper] [caliper-flow]      ---------- Finished Test ----------
2020.03.02-17:45:47.706^[[31m error^[[39m [caliper] [caliper-flow]      Error: Error: Table must define at least one row.
at exports.default.rows (/home/intel/latest_caliper/caliper-benchmarks/node_modules/table/dist/validateTableData.js:25:11)
at Object.exports.default [as table] (/home/intel/latest_caliper/caliper-benchmarks/node_modules/table/dist/table.js:116:35)
at Report.printTable (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/report/report.js:133:23)
at Report.printResultsByRound (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/report/report.js:255:14)
at Object.module.exports.run (/home/intel/latest_caliper/caliper-benchmarks/node_modules/@hyperledger/caliper-core/lib/caliper-flow.js:119:20)
at
at process._tickCallback (internal/process/next_tick.js:189:7)
2020.03.02-17:45:47.707^[[90m debug^[[39m [caliper] [null-observer]     No action taken by NullObserver on stopWatch
Context

Smallbank long run test config as below
"test:
name: smallbank
description: This is smallbank benchmark for caliper
clients:
type: local
number: 5
rounds:

label: smallOperations
txDuration:

3600
rateControl:
type: fixed-rate
opts:
tps: 1000
arguments:
accounts: 60000
txnPerBatch: 1
callback: benchmarks/scenario/smallbank/smallbankOperations.js

label: query
txNumber:

5000
rateControl:
type: fixed-rate
opts:
tps: 500
callback: benchmarks/scenario/smallbank/query.js
monitor:
type:

docker
docker:
name:

all
interval: 1
"

Expected Behavior

It should pass without any error as I am expecting in simple run for 1hr with same tps(1000)
Actual Behavior

Test exits with an client error and report won't generated.
Possible Fix

Not sure about the fix.
Steps to Reproduce

Create config file as mentioned above say file name as config_long.yaml
npm init -y
npm install --only=prod @hyperledger/caliper-cli
npx caliper bind --caliper-bind-sut fabric --caliper-bind-sdk 1.4.0
npx caliper benchmark run --caliper-workspace . --caliper-benchconfig benchmarks/scenario/simple/config_long.yaml --caliper-networkconfig networks/fabric/fabric-v1.4.0/2org1peergoleveldb/fabric-go-tls.yaml

Existing issues

[ ] Stack Overflow issues
[ ] GitHub Issues
[ ] Rocket Chat history

Context

Your Environment

Version used: caliper version 0.2.0 caliper-benchmarks version 0.2.0 fabric sdk version 1.4.0
Environment name and version (e.g. Chrome 39, node.js 5.4):
Nodejs version: v8.17.0
Operating System and version (desktop or mobile):
Ubuntu 16.04 LTS
Link to your project:

Here is the content config_long.yaml
test:
name: smallbank
description: This is smallbank benchmark for caliper
clients:
type: local
number: 5
rounds:

label: smallOperations
txDuration:

3600
rateControl:
type: fixed-rate
opts:
tps: 1000
arguments:
accounts: 60000
txnPerBatch: 1
callback: benchmarks/scenario/smallbank/smallbankOperations.js

label: query
txNumber:

5000
rateControl:
type: fixed-rate
opts:
tps: 500
callback: benchmarks/scenario/smallbank/query.js
monitor:
type:

docker
docker:
name:

all
interval: 1

@Ram-srini Could you please reproduce the issue with the current unstable Caliper release? Use the npm install --only=prod @hyperledger/caliper-cli@unstable command during install.
The new releases provide more detailed error reporting on the worker process side since the error seems to originate from there.
See the docs about the new CLI and benchconfig API of Caliper 0.3.0, should be trivial modifications.

https://hyperledger.github.io/caliper/vLatest/installing-caliper/
https://hyperledger.github.io/caliper/vLatest/bench-config/


Generator spec example

Is your feature request related to a problem? Please describe.
Looking at the documentation for generator specs I am not sure how I should be testing generators. How do I setup the test and make assertions about the success or failure of the generator under test.
https://relishapp.com/rspec/rspec-rails/v/4-0/docs/generator-specs/generator-spec
Also noticed that there is no example provided like there are for other spec types https://relishapp.com/rspec/rspec-rails/v/4-0/docs/generator-specs/generator-spec
Describe the solution you'd like
An example of how to test generators showing asserting the contents of particular files.
Describe alternatives you've considered
Additional context
I ran into this problem from  a rails plugin, not sure if that is any different, hopefully not.
The generator is loaded through a railtie, might be good to include an example of how to trigger the generator to be loaded.

You can check RSpec's own generator specs for examples https://github.com/rspec/rspec-rails/tree/main/spec/generators/rspec
It would be helpful to add a couple of commented out examples to https://github.com/rspec/rspec-rails/blob/main/lib/generators/rspec/generator/templates/generator_spec.rb along with documentation extension. A PR is welcome.

@pirj I've tried to my best ability to look for examples in the rspec generator specs but I simply can't make it work.
I copied the spec/support/generators.rb to my own machine and tried running it but it chokes on:
An error occurred while loading ./spec/generator/stimulus_generator_spec.rb.
Failure/Error: destination File.expand_path('../../tmp', __dir__)

NoMethodError:
  undefined method `destination' for RSpec::ExampleGroups::Stimulus:Class
  Did you mean?  description
# ./spec/support/generators.rb:9:in `set_default_destination'
# ./spec/support/generators.rb:13:in `setup_default_destination'
# ./spec/generator/stimulus_generator_spec.rb:9:in `block in <main>'
# ./spec/generator/stimulus_generator_spec.rb:8:in `<main>'

I felt like it shouldn't be this hard so wanted to chime in with my experience. Without a working example, I can't really provide a PR to guide others either.

@mhenrixon I believe you need ammeter to test your generators if you intend to test generators the same way rspec-rails does. This is purely optional, but rspec-rails on its own doesn't provide any convenience helpers to test generators.


File format to extension function

Hey @tcmitchell  @jakebeal , this tries to fix #244 , by implementing a function bound on the Document class to provide standard extensions for given file format.

This looks great. I made a couple of suggestions to improve the code. I'd also like to see a unit tests that checks at least 2 of the known file types and one unknown file type, verifying the raised exception.
There are unit tests that check for exceptions. Let me know if you can't find one. I would be happy to help you craft some or all of this test method. It should go in test_document.py.
Thanks!

This looks great. I made a couple of suggestions to improve the code. I'd also like to see a unit tests that checks at least 2 of the known file types and one unknown file type, verifying the raised exception.
There are unit tests that check for exceptions. Let me know if you can't find one. I would be happy to help you craft some or all of this test method. It should go in test_document.py.

Sure, I have made one named test_file_extension in test_document.py.


fastlane: 2.91.0 -> 2.105.2

The current package emits some annoying warnings and stacktraces.  I'm not sure why, but they seem to go away after the update.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

~[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)~
~[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"~
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)<PHONE_NUMBER> -><PHONE_NUMBER>)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build fastlane

Success on aarch64-linux (full log)
Attempted: fastlane
Partial log (click to expand)
patching script interpreter paths in /nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2
/nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/sigh/lib/assets/resign.sh: interpreter directive changed from "/bin/bash" to "/nix/store/017xymp5i03qzigafnfx7sgf6602gmzq-bash-4.4-p23/bin/bash"
/nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/gym/lib/assets/wrap_xcodebuild/xcbuild-safe.sh: interpreter directive changed from "/bin/bash" to "/nix/store/017xymp5i03qzigafnfx7sgf6602gmzq-bash-4.4-p23/bin/bash"
/nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/fastlane/lib/assets/completions/completion.bash: interpreter directive changed from "/bin/bash" to "/nix/store/017xymp5i03qzigafnfx7sgf6602gmzq-bash-4.4-p23/bin/bash"
/nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/fastlane: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/7zjkzc17n2r3jhdqjrqss10bbgwk67bf-ruby-2.5.1/bin/ruby"
/nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/bin-proxy: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/7zjkzc17n2r3jhdqjrqss10bbgwk67bf-ruby-2.5.1/bin/ruby"
checking for references to /build in /nix/store/6n65na5df0pwknhnh9lq45sj76frmpfr-ruby2.5.1-fastlane-2.105.2...
building '/nix/store/ica3vq4q7vc2h25nq7z19d98ddy0pp69-fastlane-2.105.2.drv'...
created 198 symlinks in user environment
/nix/store/x4ap4qmfxvxdl2x2m01xlk5rkwp4lbm9-fastlane-2.105.2

Success on x86_64-linux (full log)
Attempted: fastlane
Partial log (click to expand)
patching script interpreter paths in /nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2
/nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/sigh/lib/assets/resign.sh: interpreter directive changed from "/bin/bash" to "/nix/store/nii7pk6pv4x4as7vsxbvwyzjn67vax6r-bash-4.4-p23/bin/bash"
/nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/bin-proxy: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/wyqsn3wf4r7kf3z4s4fri831r2bgsxyp-ruby-2.5.1/bin/ruby"
/nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/fastlane: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/wyqsn3wf4r7kf3z4s4fri831r2bgsxyp-ruby-2.5.1/bin/ruby"
/nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/gym/lib/assets/wrap_xcodebuild/xcbuild-safe.sh: interpreter directive changed from "/bin/bash" to "/nix/store/nii7pk6pv4x4as7vsxbvwyzjn67vax6r-bash-4.4-p23/bin/bash"
/nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/fastlane/lib/assets/completions/completion.bash: interpreter directive changed from "/bin/bash" to "/nix/store/nii7pk6pv4x4as7vsxbvwyzjn67vax6r-bash-4.4-p23/bin/bash"
checking for references to /build in /nix/store/3isnmjhhhanq25iid0v1c26w33bxvyxm-ruby2.5.1-fastlane-2.105.2...
building '/nix/store/b7zvyfx9m9q6xwcr9xvifmzvp2c72v0j-fastlane-2.105.2.drv'...
created 198 symlinks in user environment
/nix/store/485y3ihwa1rb844w3ijd197srmqwh0ix-fastlane-2.105.2

Success on x86_64-darwin (full log)
Attempted: fastlane
Partial log (click to expand)
post-installation fixup
patching script interpreter paths in /nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2
/nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/bin-proxy: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/mvzj2hwmzd67h8yxjmqgjjkn2zp1vz81-ruby-2.5.1/bin/ruby"
/nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/bin/fastlane: interpreter directive changed from "/usr/bin/env ruby" to "/nix/store/mvzj2hwmzd67h8yxjmqgjjkn2zp1vz81-ruby-2.5.1/bin/ruby"
/nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/fastlane/lib/assets/completions/completion.bash: interpreter directive changed from "/bin/bash" to "/nix/store/f2lp21azk4w88f17zgf4mji64agb18ga-bash-4.4-p23/bin/bash"
/nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/gym/lib/assets/wrap_xcodebuild/xcbuild-safe.sh: interpreter directive changed from "/bin/bash" to "/nix/store/f2lp21azk4w88f17zgf4mji64agb18ga-bash-4.4-p23/bin/bash"
/nix/store/w56vw2k1bpxvpgdvj6p2afgi57zs9s3k-ruby2.5.1-fastlane-2.105.2/lib/ruby/gems/2.5.0/gems/fastlane-2.105.2/sigh/lib/assets/resign.sh: interpreter directive changed from "/bin/bash" to "/nix/store/f2lp21azk4w88f17zgf4mji64agb18ga-bash-4.4-p23/bin/bash"
building '/nix/store/5whs8r92ylprdsaxird4w4p9g9jnmg8q-fastlane-2.105.2.drv'...
created 198 symlinks in user environment
/nix/store/vyc675z6rv7jm1lqp6qn7z2iclqgz4bb-fastlane-2.105.2


on(event, selector, callback) should support multi callbacks

Now only the last callback is executed.

done by impression.js


how to remove events

see in the following screenshot. I added an event which sets foo = true and now I want to remove it. How would I do that?

Working on this, should land sometime next week with branching (if/else) inside components update. For now, you can drag foo onto true, so the event would be foo=foo and do nothing.

Fixed, there is now a button to remove mutations from events. Design is still terrible, but we will get around to that in a month.


How to invoke HoverFast on command line

How do I invoke HoverFast from the command line?  The documentation says to use the command, "HoverFast" followed by options.  Like:
HoverFast --version
However, HoverFast isn't a command on my $PATH.
For context, I didn't have Docker available.  So installed like:
git clone https://github.com/choosehappy/HoverFast.git
cd HoverFast
pip install .
Thanks so much!  It's looks like a really amazing tool.

Hey there!
Happy to see you're using HoverFast. Could you provide me with some information on the Operating System you are using and how you installed the tool? Was is done through conda or did you install it directly on the machine?
With this info I should be able to guide you a bit better and tell you exactly how to run it successfully!
Cheers,
Petros

Installing in my /home directory of an HPC, which is Linux Red Hat 8. It doesn't have conda (or miniconda).  And I don't have Docker permissions.
I created a python virtual environment with python 3.11.  Then I ran:
git clone https://github.com/choosehappy/HoverFast.git
cd HoverFast
pip install .
If Docker or Conda is required, I'm sure I could get my sys admin to install with those.
Thanks!

Thanks for the details! It sounds like the HoverFast command might not be recognized because the entry point script wasn’t added to your $PATH when you installed it.
Here’s how you can try to resolve this:

Ensure the installation was successful:
After running pip install ., you should see that the installation completed without errors. You can verify this by listing the installed packages in your virtual environment with pip list and checking for HoverFast.

Check the Scripts Directory:
The HoverFast command is likely installed in the Scripts or bin directory of your virtual environment. You can find it by navigating to the bin directory in your virtual environment:
cd /path/to/your/virtualenv/bin
ls

Add the Script to Your PATH:
If the HoverFast command is present in the bin directory but not recognized globally, you can add the directory to your $PATH temporarily:
export PATH=/path/to/your/virtualenv/bin:$PATH

After this, you should be able to run HoverFast directly from the command line.

Now, you should be able to invoke HoverFast using the command you mentioned, like HoverFast --version.
Keep in mind, for HPC environments, singularity (apptainer) might be more suitable since it does not require the same level of priviliges, so that might also be worth a try!
Let me know if this helps or if you run into any other issues!
Cheers,
Petros

That worked great
Just had to add the virtual env's /bin to my $PATH, like you said.
Thanks again


Unexpected value returned for aggregate collection in distributed query

The fix for issue #2914 (specifically be26632e0db6a7eef70eab9800150f2b51c5930c) seems to have caused a regression when using aggregate collections in distributed queries. For example, this test:
diff --git a/distributed/src/test/java/com/orientechnologies/orient/server/distributed/DistributedAggregateCollectionTest.java b/distributed/src/test/java/com/orientechnologies/orient/server/distributed/DistributedAggregateCollectionTest.java
new file mode 100644
index 0000000..a0b65ee
--- /dev/null
+++ b/distributed/src/test/java/com/orientechnologies/orient/server/distributed/DistributedAggregateCollectionTest.java
@@ -0,0 +1,49 @@
+package com.orientechnologies.orient.server.distributed;
+
+import java.util.Collections;
+
+import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
+import com.orientechnologies.orient.core.record.impl.ODocument;
+import com.orientechnologies.orient.core.sql.OCommandSQL;
+import com.tinkerpop.blueprints.impls.orient.OrientBaseGraph;
+import junit.framework.Assert;
+import org.junit.Test;
+
+public class DistributedAggregateCollectionTest extends AbstractServerClusterTest {
+  private final static int SERVERS = 1;
+
+  @Override
+  public String getDatabaseName() {
+    return "DistributedAggregateCollectionTest";
+  }
+
+  @Test
+  public void test() throws Exception {
+    init(SERVERS);
+    prepare(false);
+    execute();
+  }
+
+  @Override
+  protected void onAfterDatabaseCreation(OrientBaseGraph db) {
+    db.command(new OCommandSQL("CREATE CLASS Item extends V")).execute();
+    db.command(new OCommandSQL("CREATE PROPERTY Item.name STRING")).execute();
+  }
+
+  @Override
+  protected void executeTest() throws Exception {
+
+    ODatabaseDocumentTx db = new ODatabaseDocumentTx("plocal:target/server0/databases/" + getDatabaseName());
+    db.open("admin", "admin");
+
+    try {
+      db.command(new OCommandSQL("INSERT into Item (name) values ('foo')")).execute();
+
+      final Iterable<ODocument> result = db.command(new OCommandSQL("select set(name) as names from Item")).execute();
+      Assert.assertEquals(Collections.singleton("foo"), result.iterator().next().field("names"));
+
+    } finally {
+      db.close();
+    }
+  }
+}

will fail with the following message using the latest code from the develop branch:
junit.framework.AssertionFailedError: expected:<[foo]> but was:<[{node=europe0.DistributedAggregateCollectionTest, context=[foo]}]>

Whereas if I revert be26632e0db6a7eef70eab9800150f2b51c5930c then the test passes. AFAICT the node+context value is an internal form that's used to help merge results across nodes, but shouldn't be exposed to clients.

You're right. We found the same with other tests: our Jenkins was complaining form days. Fixing it now.

Is DistributedAggregateCollectionTest a your test case?

Ok, merging your test case...

Fixed in 5e8d73511596551b7c3e55262552be466fa2a907

Thanks for the test case

No worries, thanks for the fast response


fix exception thrown in AuthenticationCombined when autocreating AD user

I thought I could submit these as separate pull requests but I guess not.  I'm guessing the new feature (general-purpose ListView filtering) will need to be adjusted - it's basically thrown into core with no options or finesse.

merged in <IP_ADDRESS>


chore: add HasValidateBasic compiler assertions to msgs

Description

Add the HasValidateBasic interface as a compiler assertion to each of the ibc-go tx msgs.
closes: #4665
Commit Message / Changelog Entry
type: commit message

see the guidelines for commit messages. (view raw markdown for examples)

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[ ] Targeted PR against correct branch (see CONTRIBUTING.md).
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards and Go style guide.
[ ] Wrote unit and integration tests.
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/).
[ ] Added relevant godoc comments.
[ ] Provide a commit message to be used for the changelog entry in the PR description for review.
[ ] Re-reviewed Files changed in the Github PR explorer.
[ ] Review Codecov Report in the comment section below once CI passes.

Hi @trinitys7, looks like the linter needs to be fixed and the branch will need to be kept up to date with main before merge. We will be happy to do this for you if you could allow us to push to your branch (usually there's something you can click on the right hand side of this pr)

@colin-axner, I made this PR from my organization fork, which doesn't allow maintainers to make edits. I will close this PR and create another one from my personal fork instead.


⚠ Bofrost.nl has degraded performance

In f58cd5e, Bofrost.nl (https://www.bofrost.nl) experienced degraded performance:

HTTP code: 200
Response time: 926 ms

Resolved: Bofrost.nl performance has improved in 9714489.


Update docfx.json

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

office-365-management-api/docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

office-365-management-api/docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Laravel 9.x support

Laravel 9.x support and php constraint >= 8.0 like in romans library

@sylfel ty
I will update dependencies and try to integrate your PR.

@sylfel can you check if my changes are the same that you sent before?
Also, you will need to update your dependency to use version ^3.0.

Hi @wandersonwhcr  !
No my changes are different, and your change don't work with laravel 9
illuminate-romans require illuminate/support ^8.35 or Laravel 9.x provide illuminate/support 9.x
All your last change are good, and for perfectly

@sylfel merged!
ty :partying_face:

https://github.com/wandersonwhcr/illuminate-romans/releases/tag/v3.0.0
Please, make sure you are using wandersonwhcr/illuminate-romans: ^3.0 into your package.json.
Happy Coding!

ty @wandersonwhcr  !
It work perfectly


Apply a template on a file at ingest (iput) time

Current behavior: a template can only be applied to the files that are already present in the grid.
Desired behavior: select one or more templates to be applied on the files that are being uploaded.

Going to backlog


stack.add_link() should ignore links where both x and y are '@'

Currently this causes stack.add_link() to fail.

Resolved by #146


OSError: h94/IP-Adapter does not appear to have a file named image_encoder/config.json. Checkout 'https://huggingface.co/h94/IP-Adapter/main' for available files.

Traceback (most recent call last):
File "inference_IMAGdressing_controlnetpose.py", line 164, in
pipe, generator = prepare(args)
File "inference_IMAGdressing_controlnetpose.py", line 52, in prepare
image_encoder = CLIPVisionModelWithProjection.from_pretrained("h94/IP-Adapter", subfolder="image_encoder").to(
File "/home/ss.tian/miniconda3/envs/IMAGDressing/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3122, in from_pretrained
config, model_kwargs = cls.config_class.from_pretrained(

python inference_IMAGdressing_controlnetpose.py --cloth_path /home/ss.tian/IMAGDressing/a
ssets/images/garment/2.jpg --pose_path assets/images/pose/1.jpg

python inference_IMAGdressing_controlnetpose.py --cloth_path /home/ss.tian/IMAGDressing/a ssets/images/garment/2.jpg --pose_path assets/images/pose/1.jpg

This is a bug we left behind and has been fixed in the new version.


support image caching in kind builds

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/override hegel-tooling-presubmit

/override hegel-tooling-presubmit


Fix IE9 compatibility

Abstract
The existing IE9 compatibility test is done at a suboptimal time.
Real world problem
The easiest reproduction can be done via the jQuery ready function, which is extremely commonly used.
<!DOCTYPE html>

  <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
  <link rel="stylesheet" href="https://code.getmdl.io/1.1.3/material.indigo-pink.min.css">

  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width">
  <title>JS Bin</title>

Note: I tried hosting this with several online code sharing sites, but none of them seemed to work properly in IE9. Manual testing isn't too difficult here luckily.
The current compatibility test
The test, and dummy function redefinitions, are being done in the load event handler. This event waits for images etc and fires later than DOMContentLoaded, or indeed the jQuery ready event. It is common to perform UI work in those two event handlers and MDL should support that use case.
As some bonus flair, the current test checks for the existance of addEventListener inside a function that will only ever execute if addEventListener exists and is capable of calling functions.
window.addEventListener('load', function() {
  'use strict';

  /**
   * Performs a "Cutting the mustard" test. If the browser supports the features
   * tested, adds a mdl-js class to the  element. It then upgrades all MDL
   * components requiring JavaScript.
   */
  if ('classList' in document.createElement('div') &&
      'querySelector' in document &&
      'addEventListener' in window && Array.prototype.forEach) {
    document.documentElement.classList.add('mdl-js');
    componentHandler.upgradeAllRegistered();
  } else {
    /**
     * Dummy function to avoid JS errors.
     */
    componentHandler.upgradeElement = function() {};
    /**
     * Dummy function to avoid JS errors.
     */
    componentHandler.register = function() {};
  }
});

Proposed solution
By immediately performing the compatibility test we can guarantee that after the MDL script has been loaded it won't provide IE9 with functions that will throw. As a bonus, doing it this way will also perform the addEventListener check before using it.
/**
 * Performs a "Cutting the mustard" test. If the browser supports the features
 * tested, adds a mdl-js class to the  element. It then upgrades all MDL
 * components requiring JavaScript.
 */
if ('classList' in document.createElement('div') &&
    'querySelector' in document &&
    'addEventListener' in window && Array.prototype.forEach) {
  window.addEventListener('load', function() {
    'use strict';
    document.documentElement.classList.add('mdl-js');
    componentHandler.upgradeAllRegistered();
  });
} else {
  /**
   * Dummy function to avoid JS errors.
   *
   * @suppress {checkTypes}
   */
  componentHandler.upgradeElement = function() {};
  /**
   * Dummy function to avoid JS errors.
   *
   * @suppress {checkTypes}
   */
  componentHandler.register = function() {};
}

Sorry, but this can't be merged. It breaks backwards compatibility by possibly making the upgrade fire sooner than expected. If a developer inserts the script in their  then this will attempt to upgrade the DOM right away, before it even exists. Then they'd need to know how to upgrade the DOM later and modify their app script to carry out such a task.
We simply can't risk breaking sites expecting this to continue to function as-is.
Thank you for attempting a patch, but unless the issue can be resolved in a non-breaking way there is nothing that can be done for MDL 1.x.

I'm pretty sure this modification only affects IE9 and doesn't change the timing of any MDL code other than the compatibility check.

I think the only concern here is, if someone did include the polyfills for these things after the MDL script since the test did execute later originally... it would now fail. So, that does cause a BC concern.
Unfortunately, we made the ComponentHandler a big pillar of MDL 1.x's JS. Therefore, any modifications to how it operates are extremely crucial. At this point, since we are already working on 2.x experimentation which will do away with this or make it optional and lessen its need, it is best we probably not mess with it. If we do and someone complains about it breaking, we then need to undo it which could leave people relying on the new method in their builds in a broken state as well.

Okay, what about something like this:
/**
 * Performs a "Cutting the mustard" test. If the browser supports the features
 * tested, adds a mdl-js class to the  element. It then upgrades all MDL
 * components requiring JavaScript.
 */
function cutsTheMustard() {
  return 'classList' in document.createElement('div') &&
    'querySelector' in document &&
    'addEventListener' in window && Array.prototype.forEach;
}

var initiallyCutTheMustard = true;

if (!cutsTheMustard()) {
  initiallyCutTheMustard = false;

  componentHandler.upgradeElement = function() {
    if (!cutsTheMustard()) return;
    // TODO: Execute the real function
  };
  componentHandler.register = function() {
     if (!cutsTheMustard()) return;
    // TODO: Execute the real function
  };
}

window.addEventListener('load', function() {
  'use strict';

  if (cutsTheMustard()) {
    if (!initiallyCutTheMustard) {
      // TODO: Restore original functions
    }

    document.documentElement.classList.add('mdl-js');
    componentHandler.upgradeAllRegistered();
  } else {
    componentHandler.upgradeElement = function() {};
    componentHandler.register = function() {};
  }
});

Again, the code for all supported browsers besides IE9 is the same as right now. For IE9, at script execution time if the test fails, we use more dynamic dummy functions that will start working at any moment some polyfill is loaded. Then later during  the load event we can remove that extra check, supporting both still-not-working-IE9 and polyfilled-IE9 cases.

I'm going to wait on making a decision here until @sgomes is back and can take a look at it.
Personally, this is a critical piece of the infrastructure and any changes here shouldn't be taken lightly. For example, developers could already be catching/listening for the error here and then setting up their app for IE9 in that handler. So, if this change removes that error from firing, their apps then break in IE9.

Developers could also be using the size of mdlComponentHandler.js in bytes as a constant in their animation calculations.
Frankly this policy that bugs need to be preserved in the interest of backwards compatibilty for people building code depending on undocumented bugs existing is rather worrying. Sure, I could easily deal with this specific IE9 issue with a fork. I'm more worried about all the other bugs that I haven't encountered/analyzed myself yet, and won't get fixed because someone might have built their app on the foundation that the bug will always exist and fail in the exact same way.

People using the size of the file is far less of a concern than looking for an error to setup some logic for a browser target.
We uphold SemVer, which means if something breaks it is on us to then undo whatever broke something or tell someone to have to drop everything and go patch their existing production code. I'd rather err on the caution side with our core piece of JavaScript.

Just talked with Sergio about this. We are both reluctant to make any changes here since it is plausible developers could be using the errors thrown from here to setup something for older browsers. Whether it be a compatibility layer, some message that the user is on an outdated browser, whatever. This is a natural artifact of how the component handler was implemented initially and released. Therefore, it is natural that developers may have looked at the error and used it to their advantage. Considering this, we won't be allowing any modifications to this internally since it has the possibility of breaking existing sites.
Thank you.


running two cameras inspired by multicam example

Required Info

Camera Model
D465

Firmware Version
SDK 2.0

Operating System & Version
Win 10

Platform
PC

SDK Version
2.0

Language
C++

EDIT:
Specs fix: my Camera Model is D465 and not D455 initially written. Sorry for the confusion.
All the answers were made till that point for D455.
After I managed to connect 2 realsense cameras and save frames for them in a buffer, I've noticed that my fps for both in the same amount of time dropped significantly.
For example, for 10 second of asynchronous writing of frames using callback, for each one of the cameras separately I get around 300 frames, for 30 fps it makes sense.
But when I run them simultaneously as proposed in the multi-cam example in the examples solution I get less, some times it a half for both, some times it is 250 and 200 respectively, some times even I get about 300 on the first and only few frames (like 40 or 10 or 2) on the second.
This is the code with which I run it.
I'll appreciate if the developing team could check and detect a wrong or insufficient use of the realsense SDK api or maybe a bug in the implementation or the possibility that my computer specifics cannot afford running 2 cameras with 30 fps each?
My proccessor is Intel(R) Core(TM) i3-4000M CPU @ 2.40GHz   2.39 GHz, 2 cores, 4 threads (maybe it is too weak, how can I do the calculation? I'm not a big expert).
Thank you in advance.
This is the working code:
using namespace this_thread; // sleep_for, sleep_until
using namespace chrono; // nanoseconds, system_clock, seconds

struct Frame {
    Frame(const void *pVoid, double d, int i, int w, int h, short bpp,
          int sib) : ts(d), data_size(i),
                     width(w), height(h), bytes_per_pixel(bpp),
                     stride_in_bytes(sib){
        frame_data = new uint8_t[w*h*bytes_per_pixel];
        assert(frame_data != nullptr);
        memcpy(frame_data, pVoid, w*h*bytes_per_pixel);
    }

    //~Frame(){
    //    delete this->frame_data;
    //}

    void* frame_data = NULL;
    double ts = -1;
    int data_size = -1;
    int width = -1;
    int height = -1;
    short bytes_per_pixel = -1;
    int stride_in_bytes = -1;

};

rs2::context ctx;

std::vector<rs2::pipeline> pipelines;

std::vector<std::string> serials;

auto devs = ctx.query_devices();
for (auto&& dev : devs)
    serials.push_back(dev.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER));

std::map<int, std::vector<Frame>> frames;
std::mutex mutex;
auto callback = [&](const rs2::frame& frame)
{
    //std::lock_guard<std::mutex> lock(mutex);
    if (rs2::frameset fs = frame.as<rs2::frameset>()) {
        //rs2::disparity_transform disparity2depth(false);
        //fs = fs.apply_filter(disparity2depth);
        // With callbacks, all synchronized stream will arrive in a single frameset
        for (const rs2::frame f : fs) {
            auto vf = f.as<rs2::video_frame>();
            Frame my_f = Frame(vf.get_data(), vf.get_timestamp(),
                               vf.get_data_size(), vf.get_width(),
                               vf.get_height(), vf.get_bytes_per_pixel(),
                               vf.get_stride_in_bytes());
            frames[f.get_profile().unique_id()].push_back(my_f);
        }
    } //else {
        // Stream that bypass synchronization (such as IMU) will produce single frames
        //frames_bypass.push_back(frame);
    //}
};

std::map<int, std::vector<Frame>> frames2;
std::mutex mutex2;
auto callback2 = [&](const rs2::frame& frame)
{
    //std::lock_guard<std::mutex> lock(mutex);
    if (rs2::frameset fs = frame.as<rs2::frameset>()) {
        //rs2::disparity_transform disparity2depth(false);
        //fs = fs.apply_filter(disparity2depth);
        // With callbacks, all synchronized stream will arrive in a single frameset
        for (const rs2::frame f : fs) {
            auto vf = f.as<rs2::video_frame>();
            Frame my_f = Frame(vf.get_data(), vf.get_timestamp(),
                               vf.get_data_size(), vf.get_width(),
                               vf.get_height(), vf.get_bytes_per_pixel(),
                               vf.get_stride_in_bytes());
            frames2[f.get_profile().unique_id()].push_back(my_f);
        }
    } //else {
    // Stream that bypass synchronization (such as IMU) will produce single frames
    //frames_bypass.push_back(frame);
    //}
};

rs2::pipeline pipe(ctx);
rs2::config cfg;
cfg.enable_device(serials[0]);
rs2::pipeline_profile profiles = pipe.start(cfg, callback);
pipelines.emplace_back(pipe);

rs2::pipeline pipe2(ctx);
rs2::pipeline_profile profiles2;
if(serials.size() > 1) {
    rs2::config cfg2;
    cfg2.enable_device(serials[1]);
    profiles2 = pipe2.start(cfg2, callback2);
    pipelines.emplace_back(pipe2);
}

// Collect the enabled streams names
std::map<int, std::string> stream_names;
std::map<std::string, int> stream_numbers;
for (auto p : profiles.get_streams()) {
    stream_names[p.unique_id()] = p.stream_name();
    stream_numbers[p.stream_name()] =  p.unique_id();
}

std::map<int, std::string> stream_names2;
std::map<std::string, int> stream_numbers2;
if(serials.size() > 1){
    for (auto p: profiles2.get_streams()) {
        stream_names2[p.unique_id()] = p.stream_name();
        stream_numbers2[p.stream_name()] = p.unique_id();
    }
}

sleep_for(nanoseconds(10000000000));

pipe.stop();
if(serials.size() > 1)
    pipe2.stop();

//std::vector<Frame> depth_frames = frames[0];
//Frame depth_frame0 = depth_frames[0];
std::vector<Frame> rgb_frames = frames[stream_numbers["Color"]];
std::vector<Frame> depth_frames = frames[stream_numbers["Depth"]];

std::vector<Frame> rgb_frames2 = frames2[stream_numbers2["Color"]];
std::vector<Frame> depth_frames2 = frames2[stream_numbers2["Depth"]];

Hi @ilyak93  A 2 core (dual-core) i3 would probably be a low-end configuration for using two cameras simultaneously on the same computer.  For example, a computing device known to be able to handle two 400 Series cameras being active at the same time is the Up Squared board, which is available in 4-core (quad core) specifications.
https://up-board.org/upsquared/specifications/
If you were collecting 300 frames but only actually needed one frame, a good solution may be a multiple-camera C++ script at https://github.com/IntelRealSense/librealsense/issues/2219#issuecomment-412899468 that was a modification of rs-multicam that captures an image simultaneously from every attached camera.
If you decide to use that script, it checks for whether the name of the attached camera is D415 and so you would need to change the code in that check to D455 or remove the checking mechanism.

Alternatively, another approach that is suited to short recording sessions of between 10 seconds (on a low-spec computer with modest memory) and 30 seconds (on a computer with larger memory capacity) would be to store the frames in the computer's memory witk the Keep() instruction instead of writing them to file.  With Keep(), you can store the entire streaming session's frames in memory and then perform batch-processing on them after closing the pipeline to perform actions such as post-process and align and then save all the frames to file in a single save-action.
As you are using both depth and color, you may also have less 'dropped' frames if you set the frame queue size to a value of '2' instead of the default '1' to introduce latency into the pipeline.
Use of Keep() and frame queue size is discussed for C++ in https://github.com/IntelRealSense/librealsense/issues/10042

@MartyG-RealSense, thank you for your reply.
About the first part, it is preferable for me to have all the frames, because 10 second and 300 frames is just an experiment that everything works properly, and generally I need to record as much frames as it can be recorded.
For the Keep() suggestion, notice in my code, that I don't write the files to the hard disk, while I collect them, but I collect them in a container while the data of the frames (and some metadata) is allocated on the heap, i.e RAM, so it seems that I'm doing manually what is probably is done by Keep(), am I ?
All I do in my callbacks is to allocate the memory in the heap, mem copy the data block into my allocated block and save some information.
Do you think using Keep() can make it more efficient and maybe make it work ?
Maybe the i3 is the bottleneck and even alternatively using Keep() wouldn't work ?
Could Keep() be more efficient then just allocating memory and copying the data to the allocated memory ?
There is an easy way for me to check whether the problem is in the fps, and it is to set it lower.
I probably can do it with:
cfg.enable_stream(RS2_STREAM_ANY, 1280, 720, RS2_FORMAT_ANY, 30);
and put framerate lower then 30, i.e 15 for example.
but it raises an exception in the following peace of code:
  rs2::pipeline pipe(ctx);
  rs2::config cfg;
  cfg.enable_device(serials[0]);
  cfg.enable_stream(RS2_STREAM_ANY, 1280, 720, RS2_FORMAT_ANY, 30);
  rs2::pipeline_profile profiles = pipe.start(cfg, callback);
  pipelines.emplace_back(pipe);

  rs2::pipeline pipe2(ctx);
  rs2::pipeline_profile profiles2;
  if(serials.size() > 1) {
      rs2::config cfg2;
      cfg2.enable_stream(RS2_STREAM_ANY, 1280, 720, RS2_FORMAT_ANY, 30);
      cfg2.enable_device(serials[1]);
      profiles2 = pipe2.start(cfg2, callback2);
      pipelines.emplace_back(pipe2);
  }

While without it. it works. What is the correct and simplest way to set the frame rate to less then 30 for each serial (device) in that way ?
Thank you in advance.

RS2_FORMAT_ANY is hardly ever used.  Instead, a specific stream type is defined.  Please try changing _ANY to a depth stream.
cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, fps);
cfg2.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, fps);
It should be noted that for the depth stream, the optimal resolution for depth accuracy on D455 is 848x480 rather than 1280x720
You could set the frame rate to a lower resolution simply by changing the '30' at the end of the bracket of the cfg instructions to a lower value.  Aside from 30 FPS, '15' and '5' are also supported on D455 (the minimum FPS on D455 is 5 FPS, whereas on D435 / D415 it is 6).

@MartyG-RealSense, thank you for your reply.
I probably should correct in my initial post, actually my RS-Cam is D465 ("Intel RealSense D465").
If you could edit your answer about the recommended resolutions and e.t.c, I'll be thankful, any efficient information is good.
About the streams, so if I need both color and depth streams, can I enable both of them that way for the same config for each serial in the following way (?):
cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 15);
cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 15);

I do not have a D465 to test with - my recollection is that it is a limited-availability RealSense hardware configuration dating back to 2019 that is not sold at retail - and so cannot test its supported resolutions myself, unfortunately.
My understanding based on a Python script in https://github.com/IntelRealSense/librealsense/issues/2882 is that you should be able to define pairs of cfg statements for the two separate pipelines by using the same stream definition lines but using cfg and cfg2 to distinguish between them as you did in your scripting above.

@MartyG-RealSense, Just to make sure I've got your understanding right, you mean that this peace of code:
rs2::pipeline pipe(ctx);
rs2::config cfg;
cfg.enable_device(serials[0]);
//cfg.enable_stream(RS2_STREAM_ANY, 1280, 720, RS2_FORMAT_ANY, 30);
cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 15);
cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 15);
rs2::pipeline_profile profiles = pipe.start(cfg, callback);
pipelines.emplace_back(pipe);

rs2::pipeline pipe2(ctx);
rs2::pipeline_profile profiles2;
if(serials.size() > 1) {
    rs2::config cfg2;
    //cfg2.enable_stream(RS2_STREAM_ANY, 1280, 720, RS2_FORMAT_ANY, 30);
    cfg2.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 15);
    cfg2.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 15);
    cfg2.enable_device(serials[1]);
    profiles2 = pipe2.start(cfg2, callback2);
    pipelines.emplace_back(pipe2);
}

should produce for me both color and depth for both serials (cams), is that correct ?
i've put "RS2_STREAM_ANY" because I thought that is what is put there in the default if I don't define a config, as I did in the first place, and then I've got color, depth and two more motion frames (which I actually don't need): Gyro and Accel.

Yes, I am referring to that code.
Accessing the second configuration if the list of serial numbers is greater than 1 with if(serials.size() > 1 seems to be an appropriate approach.  A C++ script in the link below that selects between a set of 4 cameras also uses this approach.
https://support.intelrealsense.com/hc/en-us/community/posts/360053150433/comments/360014118954
The multicam code that you have used for the two pipeline configurations looks okay to me overall too.
If cfg is not defined then the default stream profile of the particular RealSense camera model that is being used is applied.

Hi @ilyak93  Do you require further assistance with this case, please?  Thanks!

@MartyG-RealSense, we're good for now, thank you.

You're very welcome, @ilyak93 - thanks very much for the update!


Clubs Register Component

I want to have a page where the CLUBS can register themselves.
Refer to this design : https://github.com/IAmTamal/Milan/pull/18
Read Setup Guidelines for proper folder structure

Design changes

You need to use a different image than the one that is used for the TEMPORARY Login page.
You need to have Club Name , Club ID ,Password,  Club Location , Club Description , Club Logo
Make sure the page is fully responsive and has IMPRESSIVE UI
Use this File

Import the file with /clubs/register route in App.jsx

I would like to work on this issue!

I would like to work on this issue!


Strikethrough text not converted to Markdown

When converting a webpage to Markdown, the strikethrough text information is lost:
HTML:
Price: 80€ 70€

Current Markdown output:
Price: 80€ 70€

Desired Markdown output:
Price: ~~80€~~ 70€

This preserves the strikethrough formatting, which is important for pricing information and content editing. It also ensures correct interpretation by LLMs.

Hi @mquandalle.
Currently, Reader only works on the HTML tags level.
It does not look into the rendered CSS properties of each element.
To get a strikethrough in markdown, there needs to be a pair of corresponding  tags in the HTML.
This is also the defined behavior of the strikethrough syntax, which is part of GFM.


Example embedding zoid in a modern React.js app

Hi, I just spent some hours attempting to understand how to use zoid in the context of a modern react.js app, and thought I would share .... I am using react-rails so I don't even have access to the ReactDOM.render code .. instead I use react-router-dom to handle component rendering.
My test parent app is also React.js so I thought this example could help others:
import React from 'react';
import ReactDOM from 'react-dom';
import zoid from 'zoid';
import './App.css';

const MyZoidComponent = zoid.create({
  tag: 'my-tag',
  url: 'my-url'
});

const MyZoidReactComponent = MyZoidComponent.driver('react', {
  React,
  ReactDOM
});

function App() {
  return (
    <div className="App">
... other componentes
        <MyZoidComponent customText="XYZ" />
... other components
    </div>
  );
}

export default App;

That's it --  this renders an iFrame surrounded by the context div that zoid needs to do its magic.
Inside the iFrame (the child component) I render this routed component:
import React from 'react';
import { Route, Switch, BrowserRouter } from 'react-router-dom';
import zoid from 'zoid';
import MyComponent from './MyComponent';

const MyZoidComponent = zoid.create({
  tag: 'my-tag',
  url: 'my-url'
});

function App(props) {
  return (
    <BrowserRouter>
      <Switch>
        <Route
          path="/mypath"
          render={router => (
            <MyComponent {...props} {...window.xprops} {...router} />
          )}
        />
... other routes
      </Switch>
    </BrowserRouter>
  );
}

export default App;

MyComponent.js makes no reference to zoid ... it just receives the customText prop in props along with the routing props and anything else I want to give it locally.  It literally has no idea it's running in zoid vs any other React component, which is the way we want it...

It seems ok for a react app to provide multiple zoid components.
import React from 'react';
import { Route, Switch, BrowserRouter } from 'react-router-dom';
import zoid from 'zoid';
import MyComponent1 from './MyComponent1';
import MyComponent2 from './MyComponent2';

zoid.create({
  tag: 'my-tag1',
  url: 'my-url/mypath1'
});

zoid.create({
  tag: 'my-tag2',
  url: 'my-url/mypath2'
});

function App(props) {
  return (
    <BrowserRouter>
      <Switch>
        <Route
          path="/mypath1"
          render={router => (
            <MyComponent1 {...props} {...window.xprops} {...router} />
          )}
        />
        <Route
          path="/mypath2"
          render={router => (
            <MyComponent2 {...props} {...window.xprops} {...router} />
          )}
        />
... other routes
      </Switch>
    </BrowserRouter>
  );
}

export default App;

Further zoid.create call could be moved into the Component's code, keeping the App router above clean.
Great for application extensions where an extension provides more than one zoid component.

@dchersey  Your example react component file is great thank you for that.
What does the file with the <iframe> look like? I'm having a problem nailing down any syntax that's needed to show the desired component.

What url/script is included in the head?
Is there an actual <iframe> element, and if so what attributes does it need?

Thanks again.


ValueError: Invalid section name: 'description.'

With the package https://src.fedoraproject.org/rpms/qca-qt4/blob/rawhide/f/qca-qt4.spec
I get:
Traceback (most recent call last):
  File "/home/msuchy/projects/license-validate/./print-spec-changelog.py", line 9, in <module>
    with specfile.sections() as sections:
  File "/usr/lib64/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/specfile/specfile.py", line 135, in sections
    sections = Sections.parse(lines)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/specfile/sections.py", line 199, in parse
    data.append(Section(lines[start][1:], lines[start + 1 : end]))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/specfile/sections.py", line 64, in __init__
    raise ValueError(f"Invalid section name: '{name}'")
ValueError: Invalid section name: 'description.'

And I am not really sure if this is allowed, but apparently rpmbuild handles this. :shrug:

This is clearly a typo in the spec file, but you are right, rpmspec -P parses it without issues, so I suppose we should be able to handle it as well.

This is only one case in whole Fedora. If this would be complicated, I would not mind just proposing a PR for the qca-qt package. It is dirty hack anyway.

Considering #120 as well doesn't rpm possibly check only for the prefix of section names?

Hm, I don't really see such behaviour in the rpm source code. This chunk seems to be responsible for matching section names: https://github.com/rpm-software-management/rpm/blob/master/build/parseSpec.c#L87 and according to that, the section name in spec would have to be follow by a space or newline. But I may be missing something

Hm, I don't really see such behaviour in the rpm source code. This chunk seems to be responsible for matching section names: https://github.com/rpm-software-management/rpm/blob/master/build/parseSpec.c#L87 and according to that, the section name in spec would have to be follow by a space or newline. But I may be missing something

It's more likely it is being considered to be a macro expansion/substitution followed by dot, all part of %description section, thus valid.

I think we should fix this, because it actually makes sense and we should be consistent with RPM.


Added capability to copy just the current selection

Hello, I wanted to have the ability to copy just the selected text as HTML, and I gave it a try. A couple of functions now have the onlySelected parameter, which decides if the whole document should be copied, or just the current selection.
I also added an extra command, Copy selection as HTML.
Hope this helps with https://github.com/mvdkwast/obsidian-copy-as-html/issues/18.

Brilliant, thanks a lot !
I will take a look at it tonight !

Well, that works fine, thanks again !
To make things easier, I think we could add a command that copies the entire document if the selection is empty, or the selection otherwise. This should probably be bound to the default hotkey.


OAuth Support

OAuth support should be added back in. I disabled it because I don't understand OAuth and I generally don't want to support a security library I don't understand. The signpost OAuth library is still an optional dependency, but if there is a better library for OAuth let's use that.

https://github.com/pac4j is getting a lot of traction these days (used by Ratpack as well) - might be worth looking at related to this.

I didn't know about pac4j. Let's discuss next week about using Pac4j. It also supports Basic/Digest HTTP authentication, so we can scrap any of that stuff from our code base as well.
I'm wondering if we will need to really beef up the auth parts of the DSL to allow people to use this easily and correctly.

@cjstehno You asked if I could comment on this...
I just took a quick look over pac4j.  I didn't quite grok the usage for the client and couldn't find any good examples.  As an OAuth2 client, once you have the Bearer token, you just send it as an HTTP header on every subsequent request; it acts as your authentication.  I didn't see how you get that Bearer token from the client; perhaps you have to make every request through that client object? I didn't really see the method for that either.  But as I said, I didn't have much time to look.
Getting the token is straightforward but can be a pain (log in, go to /authorize to get a authCode, exchange the authCode at /token for real Bearer token, if it expires use a refresh token to get a new one), so having a lib for that would be useful.
Sorry I don't have more useful comments.

@kenwdelong No problem... OAuth often seems like less of a standard and more like some sort of magic ritual. :-) Thanks for taking a look.


Improve and fix MenuAnchor/MenuItemButton UX for desktop

I've been using MenuAnchor + MenuItemButton in my ADB browser app written in Flutter. In doing so, I found some issues which seem relevant for proper desktop support.
Setup:

Material 3
Windows 10 22H2
Flutter 3.16.2

Issues:

[ ] https://github.com/flutter/flutter/pull/139396 MenuItemButton doesn't have an autofocus attribute, thereby making it harder to navigate with the keyboard.
[ ] MenuAnchor does not close the menu when clicking outside with gestures other than tap e.g right clicking. This causes issues such as the image shown below.
[ ] MenuAnchor throws an exception when navigating horizontally using the arrow keys. The sample on Flutter API docs serves as a MVP for the bug.

MenuAnchor not closing on secondary tap:

Thanks Flutter Team!

I started looking and saw TapRegion was the ancestor widget which handles outside clicks for closing the menu.
This line is responsible for the secondary tap issue mentioned previously.
https://github.com/flutter/flutter/blob/d7d1572f37f365f7c5a41ac0be875cbec838c456/packages/flutter/lib/src/widgets/tap_region.dart#L237-L239

Hi @Fernthedev
Using the 1st example at MenuAnchor, I can navigate between menus normally with arrow keys. I'm not sure which case autofocus can be used, could you be more specific?
https://github.com/flutter/flutter/assets/104349824/6f3a3cf6-a4a5-4839-abc5-769391cdbbd0
Also, will the linked PR solve the 2nd and 3rd problems above? Or are they other issues?

Hi, thanks for getting back to me. The other issues will not be resolved with the PR. I might look into those if Flutter gives me a direction for resolving them. Specifically, whether if TapRegion should have an option to listen to other mouse buttons, or some other solution.
As for the 1st issue, yes you can navigate the menu with the keyboard. However the autofocus attribute is useful in the case where the focus can be requested for a specific element on menu open. For my use case, this is better as it doesn't rely on Flutter's keyboard navigation to figure out how to focus the menu in a listview (Maybe that's just me or a bug in my code, but I don't think the PR can be disregarded yet anyways). I'll try to get some video footage later today.
Thanks!

However the autofocus attribute is useful in the case where the focus can be requested for a specific element on menu open.

Thanks for the update. Labeling this point as a feature request.
For two remaining issues, please file separate issues for them for better tracking.

Done, should I adjust this issue to focus on the autofocus feature then?
Here are the new subsequent issues:
https://github.com/flutter/flutter/issues/139555
https://github.com/flutter/flutter/issues/139556

Here's some footage example showcasing the difficulty of navigating to the menu in my desktop app. It also accidentally shows a separate MenuAnchor keyboard bug, which seems harder to resolve since each MenuAnchor is individually controlled.
https://streamable.com/t807zn


Modus Create - ReactJS Challenge

Proposed Changes

Detail view for item on click
Dynamic routing based on Item id
Detail view design with Item title and subtitle that shows percentage with red minus sign showing outflow, green plus sign for inflow
Added a pie chart showing how much of the entire budget this item is contributing with should be on that page
Add a back button to return back to the previous route
Detail view is mobile and desktop responsive

Screenshot

Budget item masterview

Budget item detail view

Checklist

[x] Feature developed
[x] Created/updated unit tests
[ ] Comments in code
[ ] Documentation written

Coverage decreased (-2.2%) to 75.681% when pulling 82e7dbfdc2940d9ecdb561d0e3bde47e88554b3a on juicycleff:master into ddf7c8e1ee591036dbf2f6285e5ae412423f4f6e on ModusCreateOrg:master.


hide all controls for Vimeo?

Is it possible to hide all controls when using the player for Vimeos? I don't mean autohide - we want everything hidden all the time.
We can't use Vimeo's background settings as it disables the listeners so we can't run the desired functions after.
D.

This should work:
new Plyr(element, {controls: []}

You may get an error (maybe only if you have enable captions), but it should be fixed soon (#1061)

This should be ok now in v3.3.21 👍


Bug Report: game crash

The game crashes when I try to load a week that isn't week 6, what do I have to do?

Ok I fixed it updating the game with the new commits, but now I have another problem


Openblas 2.19 and above is not working on Ubuntu 16.04 for Power 8

Hi,
I was running HPL-2.2 with openblas 2.19,2.20 and could see that the benchmark never exits even after running it overnight for a very small problem size such as 200. I cross verified it once on 2.18 and could see that it completes in less than a second. Please find the command that I'm using :
mpirun -np 2 -bind-to-core --allow-run-as-root --mca btl sm,self,tcp xhpl
My guest configuration is as below :
Number of cores : 2 cores
SMT mode : 8
Memory : 16GB
OS :  Ubuntu 16.04 LTS
Kernel version : 4.4.0-21-generic
I verified the same thing on x86 and could see that it is working fine.
After looking at the perf data on 2.19 I could observe that 90% of the time is spent in inner_thread
Attaching the perf data of 2.19 :

and whereas the annotations of inner_thread looks like this :
annotations.txt
I could the observe the same behavior on another Ubuntu machine as well but it works fine in RHEL.

Most of the POWER8-specific changes since 0.2.18 appear to have happened in a narrow timespan between April 19 and May 22 - while I cannot possibly comment on the power8 assembly, I think I understand that wernsaar also adjusted some thresholds for thread creation in the GEMM functions that may simply have made thread contention more likely, and 8310d4d3 also dropped the ALLOC_SHM define from Makefile.power without explanation (though it may have been spurious all along).
Unless somebody else comes up with a better idea, I wonder if you could try a snapshot from somewhere in the middle of what appears to have been the crucial period ( say  0551e57 from April 26 ), and/or try and see if limiting the number of threads created on each node via OMP_NUM_THREADS has any influence ?
Also do the Ubuntu and RHEL machines you mention use the exact same  binary, or was OpenBLAS
built separately on each (implying different compiler versions and/or options in use) ?

I have compiled openblas using just the make command to use default options on both Ubuntu and RHEL and used export LD_LIBRARY_PATH= and ran HPL using the following command : mpirun -np 2 -bind-to-core --allow-run-as-root --mca btl sm,self,tcp xhpl
I have used GCC and Gfortran version 5.3.1 on both Ubuntu and RHEL.
Also tried to collect the traces but I see the following error - [ No Source Available ]
Am I missing out something? Does it require to connect any debuggers? since it is a guest machine on KVM. Any other ways of collecting the traces?

You may need to build a debuggable version of OpenBLAS and HPL first to get any meaningful backtraces with gdb.

Just a quick peek into the problem - can you compare CPUID on compile machine and run machine?

Does it work out single threaded and/or without MPI binding options?
OpenBLAS by default spins up pthreads for all available CPUs, or compile-time detected CPUs, whichever smallest (set OPENBLAS_NUM_THREADS to less if needed) , maybe MPI binding/affinity somehow hurts default build that does not try to bind processors.
You can start gdb in build root directory where all source files are in place (sort of)

@rakshithprakash Can you provide me your make.inc from the HPL benchmark such that I can recompile it easily ?

Attaching the traces :
backtraces.txt

@brada4  Both the compiler and the run machine are the same in my case.
And I removed the binding in the mpi command and gave a run but I'm seeing the same issue again.
Please find the command that I used : mpirun  --allow-run-as-root --mca btl sm,self,tcp xhpl

@grisuthedragon  Please find the attached Makefile
Makefile_ppc.txt

I tested it with the current development version of OpenBLAS on an IBM Power8+ with CentOS 7.3 running on and everything works fine with the HPL benchmark.
I compiled OpenBLAS using
 make NUM_THREADS=1 USE_OPENMP=0

because for the HPL benchmark it is quite common to use only the parallelization coming from the MPI processes.

Probably related to #660, running with only one thread is bound to avoid any deadlocks from multithreading.

Even with enabled multithreading in OpenBLAS the hpl code works fine without running into a deadlock on my machine.

I was looking for
t a a bt (i.e backtrace from all threads)
Current backtrace looks like OpenMP-enabled system OpenBLAS (symlinked to /usr/lib/libblas.so.3 via update-alternatives)- are you sure HPL is linked against freshly built OpenBLAS? (check with ldd)

@brada4  Please find the attached backtrace for all threads :
backtraces_allthreads.txt
I also looked at the ldd and could see that libopenblas is linked to 2.19 version :
libopenblas.so.0 => /home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/libopenblas.so.0 (0x00003fffb4790000)

I have doubts about linked imports of xhpl (program that backtrace comes from)
You can try LD_PRELOAD=...../openblas.so.0 xhpl (with full path in hope to override system library)
Cleaner way would be to plant alternative for libblas.so.3 to work around HPL build system mistakes.
#2  exec_blas._omp_fn.0 () at blas_server_omp.c:312
#3  0x00003fff867be8a4 in GOMP_parallel () from /usr/lib/powerpc64le-linux-gnu/libgomp.so.1
#4  0x00003fff86ecc7e4 in exec_blas (num=<optimized out>, queue=<optimized out>) at blas_server_omp.c:305
---Type <return> to continue, or q <return> to quit---
#5  0x00003fff86dfbee0 in gemm_driver (args=<optimized out>, range_m=<optimized out>, range_n=<optimized out>, sa=<optimized out>, sb=<optimized out>, mypos=0) at level3_thread.c:672
#6  0x00003fff86dfc1f4 in dgemm_thread_nt (args=<optimized out>, range_m=<optimized out>, range_n=<optimized out>, sa=<optimized out>, sb=<optimized out>, mypos=<optimized out>)
    at level3_thread.c:733
#7  0x00003fff87ba7cd0 in dgemm_ () from /usr/lib/libblas.so.3
#8  0x00000000100121f0 in HPL_dgemm ()

I did a export LD_PRELOAD=/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19 and i'm seeing the following error :
ERROR: ld.so: object '/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.
And this is the ldd from export LD_LIBRARY_PATH=/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19 :
ldd ./xhpl
linux-vdso64.so.1 =>  (0x00003fffa0ef0000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00003fffa0e50000)
libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00003fffa0d20000)
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fffa0b40000)
libopenblas.so.0 => /home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/libopenblas.so.0 (0x00003fff9ff00000)
libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fff9fe10000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00003fff9fde0000)
libopen-rte.so.12 => /usr/lib/libopen-rte.so.12 (0x00003fff9fd30000)
libopen-pal.so.13 => /usr/lib/libopen-pal.so.13 (0x00003fff9fc60000)
libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff9fc20000)
/lib64/ld64.so.2 (0x0000000058cf0000)
libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fff9fae0000)
libgomp.so.1 => /usr/lib/powerpc64le-linux-gnu/libgomp.so.1 (0x00003fff9fa90000)
libdl.so.2 => /lib/powerpc64le-linux-gnu/libdl.so.2 (0x00003fff9fa60000)
libhwloc.so.5 => /usr/lib/powerpc64le-linux-gnu/libhwloc.so.5 (0x00003fff9f9f0000)
libutil.so.1 => /lib/powerpc64le-linux-gnu/libutil.so.1 (0x00003fff9f9c0000)
libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fff9f990000)
libnuma.so.1 => /usr/lib/powerpc64le-linux-gnu/libnuma.so.1 (0x00003fff9f960000)
libltdl.so.7 => /usr/lib/powerpc64le-linux-gnu/libltdl.so.7 (0x00003fff9f930000)

Unlike LD_LIBRARY_PATH you need to include the name of the library in the LD_PRELOAD, so:
export LD_PRELOAD=/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/libopenblas.so.0
(and it does look a bit strange that ldd shows separate entries for the libopenblas.so.0 and a libblas.so.3 although you mentioned that both link to the same file)

If I look at Makefile_ppc.txt attached earlier, it uses both -lblas and -lopenblas
Taking out -lblas will fix the issue of undefined result.

Users of Ubuntu 16.04 on POWER8 may also want to take note of Ubuntu Bug #1641241 here:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1641241
describing a misbehaviour of the hardware lock elision code included in recent versions of glibc that is apparently specific to the POWER platform (and worked around by the update linked at the end of the page)
(Found via a bug report by bhart in https://github.com/tensorflow/tensorflow/issues/5482)

I did an export LD_PRELOAD for both versions of openblas - 2.18 and 2.19. Below is the ldd for 2.19 :
ldd ./xhpl
linux-vdso64.so.1 =>  (0x00003fff87de0000)
/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/libopenblas.so.0 (0x00003fff871a0000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00003fff87100000)
libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00003fff86fd0000)
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff86df0000)
libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fff86d00000)
libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff86cc0000)
libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fff86b80000)
libgomp.so.1 => /usr/lib/powerpc64le-linux-gnu/libgomp.so.1 (0x00003fff86b30000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00003fff86b00000)
libopen-rte.so.12 => /usr/lib/libopen-rte.so.12 (0x00003fff86a50000)
libopen-pal.so.13 => /usr/lib/libopen-pal.so.13 (0x00003fff86980000)
/lib64/ld64.so.2 (0x0000000044b20000)
libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fff86950000)
libdl.so.2 => /lib/powerpc64le-linux-gnu/libdl.so.2 (0x00003fff86920000)
libhwloc.so.5 => /usr/lib/powerpc64le-linux-gnu/libhwloc.so.5 (0x00003fff868b0000)
libutil.so.1 => /lib/powerpc64le-linux-gnu/libutil.so.1 (0x00003fff86880000)
libnuma.so.1 => /usr/lib/powerpc64le-linux-gnu/libnuma.so.1 (0x00003fff86850000)
libltdl.so.7 => /usr/lib/powerpc64le-linux-gnu/libltdl.so.7 (0x00003fff86820000)
Both the versions 2.18 & 2.19 seem to work now after using LD_PRELOAD but I see that 2.19 is taking approximately 2.4x the time to complete when compared to 2.18. Please find the results below :
2.18 :
WR11R2C4        2000   140     1     2               2.44              2.190e+00
2.19 :
WR11R2C4        2000   140     1     2               5.98              8.922e-01
I collected the perf data and annotations for them, please find them below :
Samples: 293K of event 'cycles:ppp', Event count (approx.):<PHONE_NUMBER>00
Overhead  Command  Shared Object                   Symbol

28.97%  xhpl     libopenblas_power8p-r0.2.19.so  [.] inner_thread

24.71%  xhpl     libopenblas_power8p-r0.2.19.so  [.] inner_thread

6.20%  xhpl     libgomp.so.1.0.0                [.] 0x0000000000016350

5.89%  xhpl     libgomp.so.1.0.0                [.] 0x0000000000016790

5.27%  xhpl     libopenblas_power8p-r0.2.19.so  [.] LDGEMM_L4x16_LOOP

4.83%  xhpl     libgomp.so.1.0.0                [.] 0x000000000001635c

4.07%  xhpl     libgomp.so.1.0.0                [.] 0x000000000001679c

2.23%  xhpl     mca_btl_sm.so                   [.] mca_btl_sm_component_progress

1.73%  xhpl     mca_pml_ob1.so                  [.] mca_pml_ob1_progress

1.33%  xhpl     libopen-pal.so.13.0.2           [.] opal_progress

1.01%  xhpl     mca_pml_ob1.so                  [.] mca_pml_ob1_iprobe
│                  START_RPCC();
│
│                  /* thread has to wait */
│                  while(job[current].working[mypos][CACHE_LINE_SIZE * bufferside] == 0) {YIELDING;};
│1dad54:   add    r10,r26,r20
│1dad58:   rldicr r10,r10,3,60
│1dad5c:   ldx    r9,r30,r10
0.00 │1dad60:   cmpdi  cr7,r9,0
│1dad64:   bne    cr7,1dad9c <inner_thread+0x65c>
│1dad68:   nop
│1dad6c:   ori    r2,r2,0
7.92 │1dad70:   nop
│1dad74:   nop
│1dad78:   nop
10.82 │1dad7c:   nop
6.36 │1dad80:   nop
│1dad84:   nop
17.26 │1dad88:   nop
23.68 │1dad8c:   nop
│1dad90:   ldx    r9,r30,r10
3.85 │1dad94:   cmpdi  cr7,r9,0
│1dad98:   beq    cr7,1dad70 <inner_thread+0x630>
│
│                  STOP_RPCC(waiting2);

Not sure how to read the perf data (do you have the 2.18 values for comparison ?), are these results reproducible (and same workload etc on the machine during both runs) ? If the values are stable it could be that the changed thresholds mentioned above are not favorable for the matrix sizes in this particular benchmark. Perhaps @grisuthedragon has benchmark results from his machine easily available ?

@martin-frbg  Here are the perf data for 2.18 for comparison :
Samples: 301K of event 'cycles:ppp', Event count (approx.):<PHONE_NUMBER>00
Overhead  Command  Shared Object                   Symbol

14.60%  xhpl     libopenblas_power8p-r0.2.18.so  [.] dgemm_kernel
5.74%  xhpl     [kernel.kallsyms]               [k] update_curr
5.54%  xhpl     [kernel.kallsyms]               [k] __schedule
4.34%  xhpl     [kernel.kallsyms]               [k] pick_next_task_fair
4.11%  xhpl     [kernel.kallsyms]               [k] __calc_delta
3.93%  xhpl     [kernel.kallsyms]               [k] _switch
3.85%  xhpl     [kernel.kallsyms]               [k] pick_next_entity
3.35%  xhpl     [kernel.kallsyms]               [k] clear_buddies
3.04%  xhpl     [kernel.kallsyms]               [k] _raw_spin_lock
2.55%  xhpl     [kernel.kallsyms]               [k] __perf_event_task_sched_out
2.36%  xhpl     [kernel.kallsyms]               [k] __switch_to
2.18%  xhpl     [kernel.kallsyms]               [k] update_min_vruntime
2.12%  xhpl     [kernel.kallsyms]               [k] put_prev_entity

Yes it's reproducible.

can you fix conflicting libblas and libopenblas dependencies?
So far what I see is mistake building HPL, nothing more.

Is it conceivable that you built 0.2.18 with different options, something more like the NUM_THREADS=1 USE_OPENMP=0 that grisuthedragon recommended above for hpl ? (No libgomp and no reference to threading in its perf results would explain less overhead...)

Most likely /usr/lib/libblas.so.3 is ubuntu-supplied openblas 0.2.18 built with OPENMP, and without CBLAS or LAPACK...
# update-alternatives --list
Should confirm it

@martin-frbg Here is the result of my benchmark( gcc 4.8.5, glibc 2,17, CentOS 7.3, Kernel 4.8[from Fedora 25], current OpenBLAS 0.2.20dev , OpenMPI 1.8.)
I optimized the HPL.dat for my machine now having the following:
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
4            # of problems sizes (N)
10240 20480 30720 40960 30 34 35  Ns
1            # of NBs
96 32 64 96 128 160 192 224 256      NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1          # of process grids (P x Q)
4       Ps
5       Qs
16.0         threshold
1            # of panel fact
2 1 2        PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
2 4          NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
2 1 2        RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
0            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
0            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)

and for the largest experiment (N = 40960) I get:
OMP_NUM_THREADS=1 mpirun --allow-run-as-root  -np 20 ./xhpl
...
 ================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR00R2R2       40960    96     4     5             107.79              4.251e+02
HPL_pdgesv() start time Wed Jan  4 21:05:28 2017

HPL_pdgesv() end time   Wed Jan  4 21:07:16 2017

--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0028599 ...... PASSED
================================================================================
...

which is quite good comparing it to the pure DGEMM performance (490 GFlop/s) obtained by the DGEMM benchmark of OpenBLAS.

again check with LDD if you use fedora-supplied /usr/lib64/libopenblas(p/o).so or one you built or both.
it does not go anywhere if you mix in random combinations of libraries in the picture.

@brada4  Please find the results below for update-alternatives --list command for both LD_PRELOAD & LD_LIBRARY_PATH
update-alternatives --list libblas.so.3
/usr/lib/libblas/libblas.so.3
/usr/lib/openblas-base/libblas.so.3
update-alternatives --list libopenblas.so.0
update-alternatives: error: no alternatives for libopenblas.so.0

@martin-frbg I have used just the make command to use default options for both 2.18 & 2.19.

@grisuthedragon  Hi, can you please give it a try on Ubuntu 16.04 once?

Main idea is to avoid linking to system blas (remove -lblas option) , and use -L/where/openblas/is/built -lopenblas, and check with ldd that you actually test openblas build that you intended to test.

@rakshithprakash I do not have an Ubuntu 16.04 running on this machine. Furthermore, IBM suggest RHEL/CentOS on this type of machines.

@grisuthedragon On IBM site I find contrary statement.... This problem will not be fixed by Red Hat switch, since system default BLAS will be linked in addition to openblas.

@brada4 Removing the -lblas option doesn't seem to work for me. Please find the error below :
mpicc -L/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/ -lopenblas -L/opt/ibm/lib/ -lm -R/opt/ibm/lib  -o /home/hpl-2.2/hpl-2.2/bin/ppc64/xhpl HPL_pddriver.o         HPL_pdinfo.o           HPL_pdtest.o /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a
/home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_idamax.o): In function HPL_idamax': HPL_idamax.c:(.text+0x38): undefined reference to idamax_'
/home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dgemv.o): In function HPL_dgemv': HPL_dgemv.c:(.text+0xa8): undefined reference to dgemv_'
HPL_dgemv.c:(.text+0x12c): undefined reference to dgemv_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dcopy.o): In function HPL_dcopy':
HPL_dcopy.c:(.text+0x3c): undefined reference to dcopy_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_daxpy.o): In function HPL_daxpy':
HPL_daxpy.c:(.text+0x44): undefined reference to daxpy_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dscal.o): In function HPL_dscal':
HPL_dscal.c:(.text+0x3c): undefined reference to dscal_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dtrsv.o): In function HPL_dtrsv':
HPL_dtrsv.c:(.text+0xc0): undefined reference to dtrsv_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dger.o): In function HPL_dger':
HPL_dger.c:(.text+0x74): undefined reference to dger_' HPL_dger.c:(.text+0xbc): undefined reference to dger_'
/home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dgemm.o): In function HPL_dgemm': HPL_dgemm.c:(.text+0xd8): undefined reference to dgemm_'
HPL_dgemm.c:(.text+0x17c): undefined reference to dgemm_' /home/hpl-2.2/hpl-2.2/lib/ppc64/libhpl.a(HPL_dtrsm.o): In function HPL_dtrsm':
HPL_dtrsm.c:(.text+0xf4): undefined reference to dtrsm_' HPL_dtrsm.c:(.text+0x1c0): undefined reference to dtrsm_'
collect2: error: ld returned 1 exit status
Makefile:76: recipe for target 'dexe.grd' failed
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory '/home/hpl-2.2/hpl-2.2/testing/ptest/ppc64'
Make.top:64: recipe for target 'build_tst' failed
make[1]: *** [build_tst] Error 2
make[1]: Leaving directory '/home/hpl-2.2/hpl-2.2'
Makefile:72: recipe for target 'build' failed
make: *** [build] Error 2
And my make file is :
----------------------------------------------------------------------
- shell --------------------------------------------------------------
----------------------------------------------------------------------

SHELL        = /bin/sh

CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch

----------------------------------------------------------------------
- Platform identifier ------------------------------------------------
----------------------------------------------------------------------

ARCH         = ppc64

----------------------------------------------------------------------
- HPL Directory Structure / HPL library ------------------------------
----------------------------------------------------------------------

TOPdir       = /home/hpl-2.2/hpl-2.2
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)

HPLlib       = $(LIBdir)/libhpl.a

----------------------------------------------------------------------
- Message Passing library (MPI) --------------------------------------
----------------------------------------------------------------------
MPinc tells the  C  compiler where to find the Message Passing library
header files,  MPlib  is defined  to be the name of  the library to be
used. The variable MPdir is only used for defining MPinc and MPlib.

MPdir        =
MPinc        =
MPlib        =

----------------------------------------------------------------------
- Linear Algebra library (BLAS or VSIPL) -----------------------------
----------------------------------------------------------------------
LAinc tells the  C  compiler where to find the Linear Algebra  library
header files,  LAlib  is defined  to be the name of  the library to be
used. The variable LAdir is only used for defining LAinc and LAlib.

LAdir        =
LAinc        =
LAlib        =

----------------------------------------------------------------------
- F77 / C interface --------------------------------------------------
----------------------------------------------------------------------
You can skip this section  if and only if  you are not planning to use
a  BLAS  library featuring a Fortran 77 interface.  Otherwise,  it  is
necessary  to  fill out the  F2CDEFS  variable  with  the  appropriate
options.  One and only one  option should be chosen in each of
the 3 following categories:

1) name space (How C calls a Fortran 77 routine)

-DAdd_              : all lower case and a suffixed underscore  (Suns,
Intel, ...),                           [default]
-DNoChange          : all lower case (IBM RS6000),
-DUpCase            : all upper case (Cray),
-DAdd__             : the FORTRAN compiler in use is f2c.

2) C and Fortran 77 integer mapping

-DF77_INTEGER=int   : Fortran 77 INTEGER is a C int,         [default]
-DF77_INTEGER=long  : Fortran 77 INTEGER is a C long,
-DF77_INTEGER=short : Fortran 77 INTEGER is a C short.

3) Fortran 77 string handling

-DStringSunStyle    : The string address is passed at the string loca-
tion on the stack, and the string length is then
passed as  an  F77_INTEGER  after  all  explicit
stack arguments,                       [default]
-DStringStructPtr   : The address  of  a  structure  is  passed  by  a
Fortran 77  string,  and the structure is of the
form: struct {char *cp; F77_INTEGER len;},
-DStringStructVal   : A structure is passed by value for each  Fortran
77 string,  and  the  structure is  of the form:
struct {char *cp; F77_INTEGER len;},
-DStringCrayStyle   : Special option for  Cray  machines,  which  uses
Cray  fcd  (fortran  character  descriptor)  for
interoperation.

F2CDEFS      = -DAdd_ -DF77_INTEGER=int -DStringSunStyle

----------------------------------------------------------------------
- HPL includes / libraries / specifics -------------------------------
----------------------------------------------------------------------

HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)

- Compile time options -----------------------------------------------

-DHPL_COPY_L           force the copy of the panel L before bcast;
-DHPL_CALL_CBLAS       call the cblas interface;
-DHPL_CALL_VSIPL       call the vsip  library;
-DHPL_DETAILED_TIMING  enable detailed timers;

By default HPL will:
*) not copy L before broadcast,
*) call the BLAS Fortran 77 interface,
*) not display detailed timing information.

HPL_OPTS     =

----------------------------------------------------------------------

HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)

----------------------------------------------------------------------
- Compilers / linkers - Optimization flags ---------------------------
----------------------------------------------------------------------

export OMPI_CFLAGS:=
CC          = mpicc
CCNOOPT     = $(HPL_DEFS) -m64
CCFLAGS     = $(HPL_DEFS) -m64 -O3 -mcpu=power8 -mtune=power8
LINKER      = mpicc
LINKFLAGS   = -L/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/ -lopenblas -L/opt/ibm/lib/ -lm -R/opt/ibm/lib
ARCHIVER    = ar
ARFLAGS     = r
RANLIB      = echo
I tried adding another L/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/ -lblas and it didn't work. I can get the same make file compiled by using the -lblas option in LAlib

Please put the "-lopenblas" in the LAlib list where the -lblas was - the libhpl.a depends on it and the sequence within the library list matters.

@rakshithprakash
Or if you do not have OpenBLAS in the default search path of your compiler put
-L/home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/ -lopenblas to the LAlib variable. If /home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/ is the place where you have an compiled version of OpenBLAS. If the linker uses the shared library in this case, you may have to add /home/hpl-2.2/openblas-2.19/OpenBLAS-0.2.19/  to the LD_LIBRARY_PATH environment variable.
@brada4 If you're using the IBM XL compilers + ESSL + CUDA  than the IBM support told me the other around. But no more here. ;-)

It got compiled now after adding the entire path in LAlib variable. But I do not see the path in the ldd.
ldd ./xhpl
linux-vdso64.so.1 =>  (0x00003fff83f10000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00003fff83500000)
libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00003fff833d0000)
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff831f0000)
libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fff83100000)
libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff830c0000)
libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fff82f80000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00003fff82f50000)
libopen-rte.so.12 => /usr/lib/libopen-rte.so.12 (0x00003fff82ea0000)
libopen-pal.so.13 => /usr/lib/libopen-pal.so.13 (0x00003fff82dd0000)
/lib64/ld64.so.2 (0x00000000446b0000)
libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fff82da0000)
libdl.so.2 => /lib/powerpc64le-linux-gnu/libdl.so.2 (0x00003fff82d70000)
libhwloc.so.5 => /usr/lib/powerpc64le-linux-gnu/libhwloc.so.5 (0x00003fff82d00000)
libutil.so.1 => /lib/powerpc64le-linux-gnu/libutil.so.1 (0x00003fff82cd0000)
libnuma.so.1 => /usr/lib/powerpc64le-linux-gnu/libnuma.so.1 (0x00003fff82ca0000)
libltdl.so.7 => /usr/lib/powerpc64le-linux-gnu/libltdl.so.7 (0x00003fff82c70000)
But using export LD_LIBRARY_PATH I can see the output for both 2.18 & 2.19.
2.18:
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
WR11R2C4        2000   140     1     2               1.81              2.948e+00
2.19 :
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
WR11R2C4        2000   140     1     2               0.35              1.512e+01

Probably they add ESSL as alternative to libblas.so.3 and all works well by default. You could try that way with OpenBLAS too - 'make install' will install /opt/OpenBLAS/lib/libopenblas.so
then run
update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /opt/OpenBLAS/lib/libopenblas.so 1
then update-alternatives --config libblas.so.3
then you can easily switch between BLAS implementations as you go forward without hard-coding any implementation.

So if I read your most recent results correctly 2.19 is now performing better than 2.18 (Gflops went from 2.948 to 15.12 for that test) ?

@brada4  I do not think that they use the ESSL as alternative for libblas.so.3 because the ESSL is designed to work with the XL compiler and therefore the Fortran symbols does not have the underscore at the end. So installing ESSL as alternative will break all applications.

Just build HPL against -lblas and update alternatives. It is the easiest way


⚠ Bofrost.at has degraded performance

In 2cca270, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 844 ms

Resolved: Bofrost.at performance has improved in 246b93b.


[2.0.0] Add README and Change Log Documentation for WSO2 Open Banking Helm Resources

Description:
It is required to perform $subject for Helm resources of the following product deployment patterns.

Documentation for deployment pattern 2

Fixed.

Fixed.


Avoid Sierra and casm compilation

Now the cairo1-run crate receives the cairo program and compiled first to sierra and then to casm.
To improve the performance, we can avoid this by modifying the run function to receive the casm code

#1702 allows avoiding sierra compilation, but data from the sierra is still needed