cucumber-json-formatter (cucumber-gherkin-windows-amd64) not creating proper JSON outcome

👓 What did you see?
Cucumber-report.json created by the cucumber-json-formatter for windows is not a valid JSON and also the report  looks different from the Json report created in Mac system
✅ What did you expect to see?
Content of the json report for windows and Mac should be same and also json report should be valid
📦 Which tool/library version are you using?
@badeball/cypress-cucumber-preprocessor": "^9.2.1",
@cypress/browserify-preprocessor": "^3.0.2",
"cypress": "latest",
"typescript": "^4.6.4"
package.json file:
"cypress-cucumber-preprocessor": {
"json": {
"enabled": true,
"formatter": "formatter/cucumber-json-formatter.exe",
"output": "reports/cucumber-report.json"
}
}
🔬 How could we reproduce it?
Steps to reproduce the behavior:

Install cypress latest
Install cypress-cucumber package and add index.ts with below content:
import * as browserify from "@cypress/browserify-preprocessor";
import { preprocessor } from "@badeball/cypress-cucumber-preprocessor/browserify";
import { addCucumberPreprocessorPlugin } from "@badeball/cypress-cucumber-preprocessor";

export default async(
on: Cypress.PluginEvents,
config: Cypress.PluginConfigOptions
): Promise<Cypress.PluginConfigOptions> => {
await addCucumberPreprocessorPlugin(on, config);
on(
"file:preprocessor",
preprocessor(config, {
...browserify.defaultOptions,
typescript: require.resolve("typescript"),
})
);
return config;
};

Create a feature & spec file

Create a formatter folder at root and place the latest version of cucumber-json-formatter file from https://github.com/cucumber/common/releases for windows 64 and keep without .exe extension as mentioned in the link https://github.com/badeball/cypress-cucumber-preprocessor/blob/master/docs/json-report.md

Added below section in package.json file:
"cypress-cucumber-preprocessor": {
"json": {
"enabled": true,
"formatter": "formatter/cucumber-json-formatter.exe",
"output": "reports/cucumber-report.json"
}
}

Run the npx cypress run to execute the test and create the report (cucumber-report.json)

Content of the cucumber-report.json generated in Mac and Windows should be same.

Please let me know for any additional information.

Please provide an NDJSON input file that reproduces the incorrect JSON.

Please provide an NDJSON input file that reproduces the incorrect JSON.
Please find attached the ndjson file
cucumber-messages.zip

any update on this, having same issue.

Please refer https://www.npmjs.com/package/cucumber-json-report-formatter

I'm also facing this issue. any update?

@mattwynne can you remind me whether we decided to stop maintaining this tool? Personally I don’t see the need for it anymore since we decided to keep a built-in JSON formatter in all our Cucumber implementations.
I don’t understand why this is needed with Cypress. If Cypress runs Cucumber, can’t it just tell Cucumber to output JSON directly?

I don’t understand why this is needed with Cypress. If Cypress runs Cucumber, can’t it just tell Cucumber to output JSON directly?

Hi again, Aslak
I'm the author of https://github.com/badeball/cypress-cucumber-preprocessor, which is responsible for the (sudden) surge of users for this tool. Let me try to answer your question. cypress-cucumber-preprocessor is a plugin for Cypress, a testing framework, that enables behavior which mimics Cucumber.
Here's the thing, Cypress itself is a test runner, with a complete runtime and not something you can put inside of Cucumber like you can with EG. Selenium. The plugin will translate Gherkin into calls to describe() and it(), which you might know from EG. Mocha. These methods are provided by Cypress' runtime and is the only structure it recognize.
So the plugin doesn't actually "run Cucumber", but rather re-implements it (largely, but makes use of several of your modules), despite many referring to it as "Cypress + Cucumber". It's essentially just giving familiar behavior to a tool that's not actually even remotely compatible with Cucumber.
As to messages, JSON reports and this.. I developed outputting messages instead of JSON as it was significantly easier, with messages requiring little to no state in the implementation and being definitely typed. Thus appeared the sudden surge of users to https://github.com/cucumber/json-formatter.

So the plugin doesn't actually "run Cucumber", but rather re-implements it (largely, but makes use of several of your modules), despite many referring to it as "Cypress + Cucumber". It's essentially just giving familiar behavior to a tool that's not actually even remotely compatible with Cucumber.

Expanding to this...
The plugin is popular with the Cypress community. The typical user however, I suspect is fully unaware of the absence of relationship between it and Cucumber and you. The fact that it's re-implementing behavior (with feature identically being the target) remains unknown to most, I'm afraid. I know that the terminology is used carelessly in blogs, articles, guides, videos, etc.
I'm sorry that you got sucked in the Cypress vortex here. It's was never my intention. Feel totally free to close indiscriminately the issues where Cypress is even mentioned.

Ohhhhh I see! Thanks for explaining @badeball. I think it's really cool what you've done here.
I'm guessing that people want Cucumber JSON output because this format is more widely supported in 3rd party reporting tools.
I'm not sure how to best accommodate that need.
One strategy could be to implement the JSON formatter natively in your library. That might be easier if the JSON formatter in Cucumber.js could be reused, but I think it might be too tightly coupled to the internals of Cucumber.js. It might have been easier if it just consumed cucumber messages and produced JSON (like the go library does).
Another alternative is to keep maintaining this library. Since it's only used by users of your cypress plugin, I think it might be better to hand it over to that community.

I'm guessing that people want Cucumber JSON output because this format is more widely supported in 3rd party reporting tools.

Yeah, exactly.
 I benefited a lot by the existence of this though, despite no ones intention of maintaining it, it let me focus on other things for a while and I'm grateful for that.

@davidjgoss has done a lot of refactorings in cucumber-js lately to make it more usable as a library. I’m sure additional work would be needed to make it work in this extension, but maybe it’s worth pursuing still?
Fancy scheduling a call to discuss this?
I’d love to see 1st class support for Cucumber in Cypress in a way that is supported by the official implementation.

One of the initiatives we have going in cucumber-js is the reimagining of formatters as a simpler kind of plug-in focused on messages. I think we could reimplement the existing JSON formatter in this vein such that it could be used as a function by other projects I.e. just give it a message stream.
Happy to have a call!

I would also be happy to have a call! What TZ are you two in and what time would suit you the best?

I have used the below package to publish the cucumber report https://www.npmjs.com/package/zephyr-scale-results-publisher

This formatter could be a good integration point:
There I no well-defined json-schema for json reports - but this is a place, where it could be actually defined. Old tools could stick to it without huge refactorings(as it is needed for porting to messages).
This gives space and time to migrate tools to messages.

Sorry for the late reply @badeball - to get this scheduled, would you be able to drop into the Cucumber Slack and DM us?

Hey @davidjgoss, sorry myself for late replying. I'm super-busy nowadays with exams, but I'll message you there when I feel it's time to re-visit this issue on my end :)

There I no well-defined json-schema for json reports - but this is a place, where it could be actually defined.

Actually, there is. Sort of. See https://github.com/cucumber/cucumber-json-converter
We’re using this library in a commercial product to process the wildly inconsistent Cucumber JSON documents.

Actually, there is. Sort of. See https://github.com/cucumber/cucumber-json-converter

I do not see how a tool that can consume many disparate formats is any help for producing a correct document.
Perhaps you are saying that every one of these formats is correct but the ramifications of that aren't trivial. Could you clarify what you mean?

What I mean is that one of the schemas in this repo could be elected the official one. Then all the implementations could be updated to conform to that schema.
I don’t think it is possible to come up with a single schema that is conformant to what current Cucumbers currently output. They are too inconsistent.

I suspect we'll run into the same problems we had when we tried to deprecate the format. There is a lot of inertia in the ecosystem.
Though those problems should be less if we stay in the same ecosystem. So we could atleast share the implementation between cucumber-js and cypress.


Is Strike still maintained and open to contributions?

It doesn't feel so. It feels like people from Appiphony fix defects they need for internal projects but ignore stuff from outside. But the reactions and reaction time when I opened issues, and later fixed them in pull requests is frustrating. And it's confusing for people from outside who want to build on the code.
I know there is no guarantee with any Open Source. But this is not another small repo by an single developer. Appiphony promoted it actively and so did others (including myself). Can SalesforceDevs still rely on Strike components for development or not?
FYI:  @georgekenessey

@rsoesemann Thank you for posting the issue. I apologize about the delays you have experienced in regards to Appiphony accepting pull requests and we will strive to do better. I believe in full transparency so let me explain our perspective on the project and current situation in hopes that it will help you and others make an educated decision on if Strike is right for you.
Strike grew out of our own needs for a robust set of Lightning components for client work and it continues to be the main driver behind the project for us. We were fortunate to have some incredibly talented people available to spend thousands of hours on Strike and are very proud of the work they accomplished. In the end we decided to share it with the community and in full honesty did not completely appreciate the demands on our time this would take.
We have been incredibly fortunate to experience a large growth in our business and are very committed to giving each and every one of our clients the highest quality of work we can provide. The downside is that lately we have simply had no bandwidth to work on Strike.
Our propencity toward delivering extreme quality does not always make addressing issues and pull requests related to Strike easy for us. When they exist, we treat each one as a project that needs to be planned, executed and delivered. For example, a pull request needs to be reviewed by a senior developer and they need to determine if it makes sense for it to be included in the project as a whole. Often this is discussed with a lead architect. A QA person then needs to test the change and update the documentation (often in collaboration with the developer). We also need to evaluate if modifying the correlating fiddler is necessary. Unfortunately for us it is not just a push of a button due to the level of documentation we have put around the components. This has certainly had it’s benefits but come at an obvious cost.
All I can say is that we will strive to do better and are hiring more folks to support our increased demand for services. We hope to spend more time on to the project and will prioritize our client needs when we do. Hopefully this insight helps you make an informed decision around the use of Strike and as always we appreciate your support.

Depending on a costly architect for reviewing a few line of bug fix code might be an unrealistic expectation for an open source project. I wasn't talking about feature extension PRs. That can be done in forks.


HDDS-8451. [Snapshot] Catch Exception and log it in generateSnapshotDiffReport

What changes were proposed in this pull request?
This change is to add exception catch and log it if there is any failure in snap diff calculation. Also mark the job FAILED.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8451
How was this patch tested?
Existing snapshot tests.

LGTM


Multi stimuli and multi response of different type

Is there anyway I can implement multi stimuli and multi response, while two stimuli are of different two types (one audio, one text),  and two response are of different types (one slider and one  button) ?
Use-Case: i want the trial to play an audio stimuli while having the corresponding text, and have the subject rate on a sliding scale, and have a button for replaying audio stimuli

There doesn't seem to be any documentation regarding this issue?
Why is it that the multi-stim-multi-response plug-in doesn't seem to come with the jspsych-6 package? (I have only seen it mentioned in JsPsych-4 versions)
So far I have only been working with a workaround that is having all responses to be in buttons,  and I cannot insert any text stimuli : https://github.com/SearaChen/Psych_Experiment_Prototype/blob/master/audio_experiment.html

Thank you!

This sounds like you will need a custom plugin. You could start with the audio-slider-response plugin and use the prompt parameter for displaying text. The only thing you need to add is an extra button that will play the audio again. You can look at how the button is configured to continue the trial, and perhaps modify that to play the audio file again.

The multi-stim-multi-response plugin was removed because it was not widely used and supporting it in newer version of jsPsych became challenging. Most of the use cases for that plugin were achievable by combining other plugins together in a timeline.

Okay that is very clear, thank you josh!


ReferenceError: module is not defined

I'm having this issue with lucid-cardano 0.9.8 in Node.js env (with ESM)
ReferenceError: module is not defined

      at __wbg_static_accessor_NODE_MODULE_06b864c18e8ae506 (node_modules/lucid-cardano/esm/src/core/libs/cardano_multiplatform_lib/cardano_multiplatform_lib.generated.js:21973:25)
      at null.<anonymous> (wasm:/wasm/009949de:1:2246391)
      at null.<anonymous> (wasm:/wasm/009949de:1:2246688)
      at null.<anonymous> (wasm:/wasm/009949de:1:2203164)
      at null.<anonymous> (wasm:/wasm/009949de:1:2248789)
      at null.<anonymous> (wasm:/wasm/009949de:1:2175037)
      at null.<anonymous> (wasm:/wasm/009949de:1:1898472)
      at TransactionBuilder.add_inputs_from (node_modules/lucid-cardano/esm/src/core/libs/cardano_multiplatform_lib/cardano_multiplatform_lib.generated.js:17398:18)
      at Tx.complete (node_modules/lucid-cardano/esm/src/lucid/tx.js:331:28)

I'm trying to figure this out. Seems like pure ESM and WASM doesn't work with node.js. I probably need to generate a seprate file for node.js to interact with WASM properly, which is quite unfortunate.

Alright Lucid 0.9.9 should fix node.js issues once and for all.


[full-ci] chore: prepare changelog, bump version, update CD

Description

[ ] Run Full CI
[ ] Create Final Tag
[ ] Merge back into master

Related Issue

Fixes <issue_link>

Motivation and Context

How Has This Been Tested?

test environment:
test case 1:
test case 2:
...

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Technical debt
[ ] Tests only (no source changes)

Checklist:

[ ] Code changes
[ ] Unit tests added
[ ] Acceptance tests added
[ ] Documentation ticket raised:

you forgot to add the changelog folder ;-)

Ok, added it.

All changelog.md changes will be overwritten by CI after the merge. Have been commited accidentally.


Update README.md

Triggerless Backdoor Attack for NLP Tasks with Clean Labels has been accepted by NAACL.

Triggerless Backdoor Attack for NLP Tasks with Clean Labels has been accepted by NAACL.

Thanks for your contribution :)


Multiple Monitors

It looks like it's possible to configure multiple monitors from the config file, but I must have something wrong in my syntax (I'm guessing it's brackets). Can you add an example with a config for multiple monitors?

@thetechgy It works for me:
{
"api_url": "your_api_url",
"api_token": "token",
"insecure_api": false,
"monitors": [
{
"name": "Common name 1",
"url":  "http://example.com",
"threshold": 80,
"metric_id": 1,
"component_id": 1,
"interval": 60,
"timeout": 5,
"expected_status_code": 200,
"strict_tls": false
},
{
"name": "Common name 2",
"url":  "http://example2.com",
"threshold": 80,
"metric_id": 2,
"component_id": 2,
"interval": 60,
"timeout": 5,
"expected_status_code": 200,
"strict_tls": false
}
],
"system_name": "us-bot",
"log_path": "/var/log/cachet-monitor.log"
}

@therapy-lf, that works perfectly - I was missing the comma between the components. Thanks!


Add MQTT lock support

Description:
The MQTT lock platform is not adding support for a specific MQTT lock at the moment rather it's like a switch.
Example entry for configuration.yaml (if applicable):
lock:
  platform: mqtt
  name: Frontdoor
  state_topic: "home/frontdoor/"
  command_topic: "home/frontdoor/set"
  payload_lock: "LOCK"
  payload_unlock: "UNLOCK"
  optimistic: false
  qos: 0
  retain: true
  value_template: '{{ value.x }}'

Checklist:

[x] Local tests with tox run successfully.
[x] TravisCI does not fail. Your PR cannot be merged unless CI is green!
[x] Fork is up to date and was rebased on the dev branch before creating the PR.
[x] Commits have been squashed.
If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

Bueno! :dolphin:


Projects route shall not fetch if there is no authentication context

Technically, this issue is no longer present since the user, using the client, won't be able to go to the /projects route through the GUI if he is not logged in.
The issue is broadly referred as creating routing guards, as an unauthenticated user should not be able to navigate to /projects. Currently if a user does so through the navigation bar, the client will let him since there are no guards in place.
Moreover, even if a user navigates to the /projects page, or the other implemented, there will be an error handled by the client.
Probably this issue should evolve to others, more specific.

Agree, we can close this and open a new one about routing guards.


Add a allow_export flag to restrict exporting

On some PKCS11 implementations, it is not allowed to create a key that
can be exported. For that reason, prevent creating exportable keys from
the configuration file so that clients can have a better return value.
This probably should be written in the book as well, as a new possible return value for generate key.
Fix #462

But you're not exactly solving the problem in #462 - you're still setting the attributes for keys that have export as false.

As per the investigation in here, I checked that those attributes are actually allowed (in Layerscape) but they can't be set to unsecure mode (exportable=true, sensitive=false). It would still work if they explicitely set to the secure mode. That's why I "simplified" what we planned to this.
I believe Yubico will follow the same logic (as per this)

Oh, it's because we have a bug 😬 I was wondering wth is going on, because what you're saying and what Sahil reported are at odds - most keys we generate for tests have export set to false, so it made no sense that create_and_destroy would fail if what you said above were true.
However, see if you can spot something wrong with this:
https://github.com/parallaxsecond/parsec/blob/6018d25f072c674de6e6e2192bf18ba730a1baa7/src/providers/pkcs11/utils.rs#L97

Folks, if we only patch the entire world of buggy PKCS#11 drivers then this PR could be obsoleted! :partying_face:

Folks, if we only patch the entire world of buggy PKCS#11 drivers then this PR could be obsoleted! 🥳

Unfortunately not, see my comment above ☹ we set SENSITIVE and EXTRACTABLE to the same boolean value, when they should be "opposites"

Oh, right. That's totally something to improve. And what a coincidence that buggy PKCS#11 drivers exposed a bug in parsec :) Fortunately this one is easy to fix.

Should I add the fix in this PR?
Would have been funny if you did not spot the inverted logic and the double bug in Parsec and there made the key secure

Should I add the fix in this PR?

I think so - it's only one line and related to what's going on in here (and it was discovered via the same conversation).

Continuous Integration / Integration tests using Cypto Trusted Service provider

I think there's a typo in the test name that failed ^^ :)


Project 📃: QR Code generator using React JS

WebMasterLog Repository (Proposing new issue)
:red_circle: Project Title :QR Code Generator
:red_circle: Aim : QR Code generator takes input from the user and generates QR code which is fetched from the API.
:red_circle: Tech Stack Used : React
:red_circle: Category : React Js Projects
Screenshots 📷

:white_check_mark: To be Mentioned while taking the issue :

Full name : Sai Manjari
GitHub Profile Link : https://github.com/Saimanjari777
What is your participant role? GSSoC

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

Full name: Diya Sen
GitHub Profile Link: https://github.com/Diyaa0313
What is your participant role? GSSoC
I would like to work on this issue @Saimanjari777

@Saimanjari777 Assigned to you. Follow the contributing guidelines carefully.


Enforce order of fields in "author" in package.json

This order: https://github.com/sindresorhus/np/blob/9ea8f5aa9f05ef6e2c236977141eb8b2bc693e5d/package.json#L8-L10 name, email, url.

Same goes for maintainers if a maintainers list is found?

Yup, and contributors.

Any improvements on the error message? Tried a couple of things but this was the best I could come up with.

Looks great


Added advisers section in about.vue

I think it would be beneficial to include a link for each of them.  So click on the adviser's name and get taken to their university profile page or something.  In the absence of pictures or longer descriptions, this is a strategy to connect users w more info on advisers if they are looking for it.


K8SPS-57: Fix exposing primary service as LoadBalancer

Test name
Status

init-deploy
passed

semi-sync
passed

monitoring
passed

sidecars
passed

config
passed

users
passed

commit: https://github.com/percona/percona-server-mysql-operator/pull/44/commits/bce3762accb23b9a0eee7d439d3b50acef207ed4
image: perconalab/percona-server-mysql-operator:PR-44-bce3762


[bug] Can't connect to Mobile Metamask

Is there an existing issue for this?

[X] I have searched the existing issues

RainbowKit Version
0.5.1
wagmi Version
0.6.2
Current Behavior
User couldn't connect the dapp with the mobile metamask. After clicking the connect wallet and selecting the metamask, the user will redirect to the following page.

Expected Behavior
Connected to dapp normally.
Steps To Reproduce

Open https://www.rainbowkit.com/ in mobile metamask
Click Connect Wallet
Choose MetaMask
See the Error page

Link to Minimal Reproducible Example (CodeSandbox, StackBlitz, etc.)
https://www.rainbowkit.com/
Anything else?
No response

same issue on mobile, can connect to wallet perfectly with rainbowkit.com but can't connect with my personal site

Thank you for finding this! We should be using the injected provider here instead of Wallet Connect. Will get onto this soon! :)

This is now fixed in RainbowKit v0.5.3.


REST API to allow test bug filling templates

Add REST API that allows submission of bug filling template and files a bug from that template
REST API endpoint should be allowed to be called only by user with Admin permission
Use feature flags to control whether this API is active or not

onefuzz debug notification test_template
this is sufficient to support my use-case


Migrating existing application

Hey,
This looks like a great project!
It also looks much easier to migrate an existing app vs Nuxt, because you're changing the code in files, rather than moving files to directories, creating new routing folders, and changing the overall architecture.
Is this a correct assumption? (I'm no Nuxt expert).

Yep 👌

You win the award for quickest ticket response ever.
Thanks! I'll jump right in :)


bugs

for FireFox

[x] more information button does not lead to the new website
[x] upcoming event grid clicks do not proceed to the next level
[x] about us page's personal website hyperlink does not work
[x] taskforce hyperlink on the bottom of the page does not work
[x] "VIEW MORE EVENTS AND NEWS" should be "VIEW MORE NEWS AND EVENTS" on the bottom of page home

about us page's personal website hyperlink does not work
"VIEW MORE EVENTS AND NEWS" should be "VIEW MORE NEWS AND EVENTS" on the bottom of page home
pictures for the taskforce

[ ] Make Task Force to Bold + Diff Color

[ ] Does not support IE / Need to fix

[ ] favicon.ico


Consider inline scene drawing

The current method of getting the scene output in the editor GUI is to use a separate fullscreen render target, which acts as a back buffer texture. This is very wasteful with respect to GPU memory bandwidth, so consider drawing the scene inline with the editor GUI using ImDrawCallbacks.

Decided the current rendering separation is much better than what would be required to implement this feature, also the performance impact is not of significant concern.


Can not run vep

I can run normal vep tool with the same input file but errors occured when I run cannoli vep with following command:
cannoli-submit --master local --deploy-mode client -- vep /input/sample.vcf /output/out -executable /home/spark/ensembl-vep/vep -cache file:///home/spark/.vep/homo_sapiens/95_GRCh38/ -single
Log:
Caused by: htsjdk.tribble.TribbleException$InvalidHeader: Your input file has a malformed header: We never saw the required CHROM header line (starting with one #) for the input VCF file 	at htsjdk.variant.vcf.VCFCodec.readActualHeader(VCFCodec.java:119) 	at org.bdgenomics.adam.rdd.variant.VCFOutFormatter.read(VCFOutFormatter.scala:104) 	at org.bdgenomics.adam.rdd.OutFormatterRunner.<init>(OutFormatter.scala:32) 	at org.bdgenomics.adam.rdd.GenomicDataset$$anonfun$15.apply(GenomicDataset.scala:876) 	at org.bdgenomics.adam.rdd.GenomicDataset$$anonfun$15.apply(GenomicDataset.scala:838) 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) 	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) 	at org.apache.spark.scheduler.Task.run(Task.scala:121) 	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402) 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Please help to resolve the issue

Hello @toanddt!  Thank you for submitting this issue.
I am afraid the malformed header line error reported above might be masking the real issue.  Cannoli is expecting valid VCF on stdout from vep.
Might you be able to look for any error message(s) reported by vep itself in the logs?

Thank for your quick response @heuermh.
This a vep logs:
`-------------------- EXCEPTION --------------------
MSG: ERROR: "so" is not a valid value for --terms
Valid values: SO, display, NCBI
STACK Bio::EnsEMBL::VEP::Config::check_config /home/spark/ensembl-vep/modules/Bio/EnsEMBL/VEP/Config.pm:649
STACK Bio::EnsEMBL::VEP::Config::new /home/spark/ensembl-vep/modules/Bio/EnsEMBL/VEP/Config.pm:487
STACK Bio::EnsEMBL::VEP::BaseRunner::new /home/spark/ensembl-vep/modules/Bio/EnsEMBL/VEP/BaseRunner.pm:92
STACK toplevel /home/spark/ensembl-vep/vep:216
Date (localtime)    = Thu Jun 11 05:24:51 2020
Ensembl API version = 95
`
And I found out the rout cause, because cannoli hardcode vep args
`class Vep(
val args: VepArgs,
val stringency: ValidationStringency = ValidationStringency.LENIENT,
sc: SparkContext) extends CannoliFnVariantContextDataset, VariantContextDataset {
override def apply(variantContexts: VariantContextDataset): VariantContextDataset = {
val builder = CommandBuilders.create(args.useDocker, args.useSingularity)
  .setExecutable(args.executable)
  .add("--format")
  .add("vcf")
  .add("--output_file")
  .add("STDOUT")
  .add("--vcf_info_field")
  .add("ANN")
  .add("--terms")
  .add("so")
  .add("--no_stats")
  .add("--offline")
  .add("--dir_cache")
  .add(if (args.addFiles) "$0" else absolute(args.cachePath))`

What version of VEP you used for testing?

Thanks for the additional information!
Oh, that is frustrating -- why would they make an incompatible change so → SO?  We have to hard code some command line arguments in order for vep to output valid VCF ANN specification-formatted annotations.  I will have to find at what version they made this change.

Hi @heuermh
Did you find out the version of vep which can be used to run with cannoli?

Hello @toanddt, it appears that SO has been the default for --terms for a long while back, so I feel it is pretty safe to make the change in #251.  Please let me know if that works for you.

Thank @heuermh. But I am still not able to run cannoli - vep, another issue occured . Are you able to run cannoli - vep at your side?

I am also not able to run the current version of vep via its Bioconda/Biocontainers Docker image
...
20/07/08 21:30:29 INFO Vep: Piping DatasetBoundVariantContextDataset with 0 reference sequences and 0
samples to vep with command: [docker, run, -i, -v, /data:/data, --rm,
quay.io/biocontainers/ensembl-vep:100.2--pl526hecda079_0, vep, --format, vcf, --output_file, STDOUT,
--vcf_info_field, ANN, --no_stats, --offline, --dir_cache, /data, --species, homo_sapiens]
files: []
...
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.
20/07/08 21:30:34 ERROR Executor: Exception in task 3.0 in stage 1.0 (TID 4)
htsjdk.tribble.TribbleException$InvalidHeader: Your input file has a malformed header: We never saw a header line specifying VCF version
	at htsjdk.variant.vcf.VCFCodec.readActualHeader(VCFCodec.java:108)
	at org.bdgenomics.adam.rdd.variant.VCFOutFormatter.read(VCFOutFormatter.scala:104)
	at org.bdgenomics.adam.rdd.OutFormatterRunner.<init>(OutFormatter.scala:32)
	at org.bdgenomics.adam.rdd.GenomicDataset.$anonfun$pipe$8(GenomicDataset.scala:872)

I tried with both the indexed and non-indexed versions of the VEP cache, no difference.  It appears that vep is outputting invalid VCF or no VCF at all to stdout.
Will continue to investigate...

See also https://github.com/bigdatagenomics/adam/issues/2150

Hello,
The root cause of this issue is that currently we do not set  --vcf  flag to set output in VCF format.
After adding following changes: https://github.com/dmaziec1/cannoli/commit/5684640d81e720599a4200fbd8649ab4924e5fcb it is possible to save it as VCF

Hello,
The root cause of this issue is that currently --vcf flag, which is used to set output in VCF format, is missing.
After adding following changes: dmaziec1@5684640 it is possible to save it as VCF

Thank you @dmaziec1!  I left a few questions as comments on your commit

To answer some of my questions, from http://grch37.ensembl.org/info/docs/tools/vep/vep_formats.html#output

VCF output
The VEP script can also generate VCF output using the --vcf flag.
Main information about the specificity of the VEP VCF output format:

Consequences are added in the INFO field of the VCF file, using the key "CSQ" (you can change it using --vcf_info_field).

Thus it appears for Canolli we should be using
vep --vcf --vcf_info_field ANN --no_stats --offline --dir_cache ...

without --format vcf and --term SO arguments.

I've discovered multiple issues that will prevent mapping Ensembl VEP ANN VCF INFO field values to TranscriptEffect.
See https://github.com/bigdatagenomics/adam/issues/2279

Properly mapping Ensembl VEP ANN VCF INFO field values to TranscriptEffects will take some work, possibly upstream in VEP itself.  Meanwhile, I've merged #251 which retains VEP annotations in the default CSQ INFO field and does not attempt to map those to TranscriptEffects.


Installed and configured eslint into ts template.

This pull request makes the following changes

Fix https://github.com/code-collabo/node-mongo/issues/19

General checklist

[x] File or folder now contains changes as specified in the issue i worked on
[x] I have linked the issue I worked on to this pull request submitted by me

Testing checklist

[x] Only the ts template is modified to now contain eslint
[x]  Eslint shows error and console warnings after it has been configured
[x]  I certify that I ran my checklist

Ping @code-collabo/node-mongo

@Ifycode  Waiting for review.

@Dkingofcode You are not to make direct changes to your develop branch. It is not a good practice for a fork's develop branch to be commits ahead of the original repo's branch.

Close this Pull request.
Delete your current fork and re-fork the repo again.
Create a new branch from develop to make your changes (always).
Submit a new pull request.


pull request for e2e test at 12/04/2016 09:45:02

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Removing an installation

Executing the following command does not work:
$ phpbrew remove 5.6.6
It requires a specific directory, not just the version:
$ phpbrew remove php-5.6.6

Since users may have different build with the same version, the removing operation should specify the full qualified name to avoid confusion. :)

I see. But shouldn't the autocomplete complete the dir name instead of the
version?

well, it should. the completion works for me.
if it does not complete dir names for you, you might need to re-generate the zsh completion script and reload it.
https://github.com/phpbrew/phpbrew/blob/develop/src/PhpBrew/Command/RemoveCommand.php#L20
https://github.com/phpbrew/phpbrew/blob/develop/src/PhpBrew/Config.php#L190

I am using the bash autocompletion script. I've just updated it and it got even worse. Not it does not even complete the installed versions, but lists the files/folders in the current directory. See screenshot.
I am using the latest version from master branch (both the executable and the completion script).

looks like the completion is not hooked to your command, how do you generate the script and load it? can you show me the commands?
[Sent from iPhone]

Márk Sági-Kazár<EMAIL_ADDRESS>於 2015年5月3日 17:28 寫道：
I am using the bash autocompletion script. I've just updated it and it got even worse. Not it does not even complete the installed versions, but lists the files/folders in the current directory. See screenshot.
I am using the latest version from master branch (both the executable and the completion script).
—
Reply to this email directly or view it on GitHub.

I don't generate it. I download it from the repo and put it into /etc/bash_completion.d/

OK, I think that version is hand-written completion script, we have an auto-generated one:
https://github.com/phpbrew/phpbrew/blob/develop/completion/bash/_phpbrew
Please download it and retry again :)

You mean the one from the master branch? At least you updated that one.  I tried the one from both the master and the develop branch. No luck.

Hi, I authored the hand written completion file in the past but it got behind after many releases. So now I'm just generating it through the cli framework support https://github.com/phpbrew/phpbrew/blob/develop/Makefile#L42. The version name completion needs to be implemented again, hence why this is still on develop branch only.

@c9s help needed. Completion for phpbrew install|switch|remove are back in place, but completion for phpbrew ext intall|enable|disable|show... is not working. Am I missing something?

BTW, this cli completion generator is a killer feature!

@marcioAlmada only for bash completion? or both zsh and badh?

@c9s only bash AFAIK

I just tried, and found out command name, option name completion works, but the argument completion seems not working.
here is the meta command of the completion result generator:
php bin/phpbrew meta --bash extension.install arg 0 suggestions

@marcioAlmada  OK, i think it might be a bug of completing command arguments.
currently we only support valid-values, not suggestions:
https://github.com/phpbrew/phpbrew/blob/develop/completion/bash/_phpbrew#L2238
and here:
https://github.com/phpbrew/phpbrew/blob/develop/completion/bash/_phpbrew#L2258
it should call ..._complete_extension_install, however ..._complete_extension_install was not generated.

bash completion debug tips:
echo $var > bash.debug

And tail -f bash.debug in another shell to see the debug messages.

I have a project deadline before Friday, so I have to back to work. XD good luck

I think it should work now, I will roll out a patch release of cliframework today.

phpbrew remove ... completion now works too.

@marcioAlmada we need to run phprelease to run the steps defined in phprelease.ini, does it still work? since you moved the commands into Makefile, could you also add a MakeStep into phpreleasewhen you find some time? Thanks ;)

Thanks!

since you moved the commands into Makefile, could you also add a MakeStep into phpreleasewhen you find some time?

Yes sure ^^

Ok more debug info. The trace for php bin/phpbrew meta --bash extension.install arg 0 suggestions is resulting in:
#groups
declare -A groups
declare -A labels
groups[5.6]="5.6.8 5.6.7 5.6.6 5.6.5 5.6.4 5.6.3 5.6.2 5.6.1 5.6.0"
groups[5.5]="5.5.24 5.5.23 5.5.22 5.5.21 5.5.20 5.5.19 5.5.18 5.5.17 5.5.16 5.5.15 5.5.14 5.5.13 5.5.12 5.5.11 5.5.10 5.5.9 5.5.8 5.5.7 5.5.6 5.5.5 5.5.4 5.5.3 5.5.2 5.5.1 5.5.0"
groups[5.4]="5.4.40 5.4.39 5.4.38 5.4.37 5.4.36 5.4.35 5.4.34 5.4.33 5.4.32 5.4.31 5.4.30 5.4.29 5.4.28 5.4.27 5.4.26 5.4.25 5.4.24 5.4.23 5.4.22 5.4.21 5.4.20 5.4.19 5.4.18 5.4.17 5.4.16 5.4.15 5.4.14 5.4.13 5.4.12 5.4.11 5.4.10 5.4.9 5.4.8 5.4.7 5.4.6 5.4.5 5.4.4 5.4.3 5.4.2 5.4.1 5.4.0"
groups[5.3]="5.3.29 5.3.28 5.3.27 5.3.26 5.3.25 5.3.24 5.3.23 5.3.22 5.3.21 5.3.20 5.3.19 5.3.18 5.3.17 5.3.16 5.3.15 5.3.14 5.3.13 5.3.12 5.3.11 5.3.10 5.3.9 5.3.8 5.3.7 5.3.6 5.3.5"
labels[5.6]="PHP 5.6"
labels[5.5]="PHP 5.5"
labels[5.4]="PHP 5.4"
labels[5.3]="PHP 5.3"

The nested ext install command is being arg completed with the outermost install command.

@marcioAlmada seems like it's expected? no? bash completion doesn't support sections.

You don't have a formal section support in bash, but the manual completion script handled this ambiguity with: https://github.com/phpbrew/phpbrew/blob/8de802454666d913d61ca8030e703f6b1cbca4d7/completion/bash/_phpbrew#L30-L37

    case "${prev}" in
        install)
            case "${prev_prev}" in
                ext)
                # list available extensions
                opts="$(cd $PHPBREW_HOME/build/$PHPBREW_PHP/ext/ && ls -d */ | cut -d '/' -f 1)"
                COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
                return 0
                ;;
            esac

I'm not sure how hard it is to generate a completion file to handle this situation ^ though.

Got it, almost forgot the extension install command should be completed with extension name.

I think the bug is inside the meta command:
php bin/phpbrew meta --bash app.extension.install arg 0 suggestions

the above command should call the suggestion callback defined in ExtensionInstallCommand class.

I still get the same output as https://github.com/phpbrew/phpbrew/issues/496#issuecomment-101930270

I submitted another fix for the meta command. please give it a retry.

Fixed

Awesome! Works here \o/
I'm using corneltek/cliframework:dev-master as a composer dependency temporarily, please ping me when you release next minor.

CLIFramework 2.5.2 was released, and this one seems fixed?

Fixed phpbrew self-update


Changelog for 23.11.0

PR Description
Changelog for 23.11.0
Fixed Issue(s)

Documentation

[x] I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

[x] I thought about adding a changelog entry, and added one if I deemed necessary.

Testing

[x] I thought about testing these changes in a realistic/non-local environment.

Covered by #946


Organizing imports breaks undo history

VSCode Version: 1.25.1
OS Version: macos

Steps to Reproduce:

Enable

	"editor.codeActionsOnSave": {
      "source.organizeImports": true
    }

Edit some code, add an import, and make sure that vscode has to remove or reorder imports. Now try to undo. The cursor jumps to the top of the file instead of just changing the code that you were editing.

Maybe this is realated to the prettier plugin. I'm going to check this with plugins disabled.

Do you see this with all extensions disabled? I wasn't able to reproduce it in vanilla VS Code

yes, I tried it with code --disable-extensions. Let me try to make a video.

@mjbvz This could be a duplicate of #54117. @domoritz Please try with latest insiders (b48367a8652bd0aa862638bbc24555a70823d3bd or newer)


fix(autocomplete): optionSelections not emitting when the list of options changes

Fixes the MatAutocompleteTrigger.optionSelections stopping to emit when the list options has been swapped out.
Fixes #14777.

When is this going to be merged and release? I'm having same problems and looking at this fix.

any news about it? thanks and regards

Bumping up the priority since this is causing other issues.


Develop Delete Confirmation Modal for Curriculum Entries(Admin_Frontend)

Description: Implement a modal to confirm deletion of a curriculum entry.
Tasks:
Set up a confirmation modal that appears when an entry’s delete button is clicked.
Include options to confirm or cancel the deletion.
Link modal actions to backend delete endpoint to remove entries from the database.
Acceptance Criteria:
Confirmation modal displays when the delete action is triggered.
Selecting “Confirm” deletes the entry, while “Cancel” closes the modal without action.
Deleted entries are removed from the list view.

i will handle this task


Cannot resolve talker

Hi,
Please how to resolve this problem
Many Thank

Hi,
Please how to resolve this error : Cannot resolve talker
Many Thank


Migrating RC1 to RC2. Static files not working

I migrated a project from RC1 to RC2 to the point I could build and run.
I'm facing a problem where static files is not working. css, js, etc are not picked up.
I have the app.UseStaticFiles() middleware.
project json also has:
  "content": [
    "wwwroot",
    "Views"
  ],

any idea how can i debug this?

This should be a working sample: https://github.com/aspnet/Mvc/tree/dev/samples/MvcSandbox
Can you compare your app to this and see what differences there are?

Thank you. I figured it out. turned out to be folder structure error on my side.

Great!


gen_forward empty TensorList

Hi,
I've carried out the steps in the "Training your own model" section of the readme but can't run gen_forward.py:
`python gen_forward.py --alpha 1 --input_text "this is whatever you want it to be" griffinlim
Using device: cuda
Initialising Forward TTS Model...
+----------+--------------+----------+
| Tacotron | Vocoder Type | GL Iters |
+----------+--------------+----------+
|   10k    | Griffin-Lim  |    32    |
+----------+--------------+----------+
| Generating 1/1
Traceback (most recent call last):
File "gen_forward.py", line 142, in
_, m, _ = tts_model.generate(x, alpha=args.alpha)
File "/home/user/Documents/vocal_synthesis/models/forward_tacotron.py", line 165, in generate
x, _ = self.lstm(x)
File "/home/user/Documents/vocal_synthesis/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/user/Documents/vocal_synthesis/venv/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 570, in forward
self.dropout, self.training, self.bidirectional, self.batch_first)
RuntimeError: stack expects a non-empty TensorList`
I tried running gen_tacotron.py and it ran without error, but the file it produced seemed too long and sounded nothing like speaking.
If it's relevant, I didn't get too far in the 289,000 step section of train_forward but the loss wasn't reducing much anyway.
Thanks :)

Hi, this sounds as if your tacotron model hasn't built any attention and thus the durations are wrong. Could you check the attention plot of the tacotron tensorboard?

The attention image is pretty much a solid dark purple with some very faint lines so I think you're right! Should I start training a new model?

Yeah, you need to retrain the tacotron model. What dataset are you using? Taco has problems to build attention when the snippets begin with pauses.

I'm using my own dataset which definitely has some pauses at the start of clips. Thanks so much for the tip, I'll give it a shot.


fix issue 15914 - [REG 2.071] getopt doesn't accept anymore a character for a bool option

Discussed in #3859

I just remembered that regression fixes need to be made against the stable branch. Sorry, made that mistake in my PR too.

So I must open another PR in stable ?

Yes, unfortunately a new PR must be created to target a different branch.

Yes, regression fixes should go to the stable. For other bug fixes master is ok.


Versions are not reported for khmer or pigz from the UNPIGZ_MEGAHIT_CONTIGS-process

Check Documentation
I have checked the following places for your error:

[ ] nf-core website: troubleshooting
[ ] nf-core/metatdenovo pipeline documentation

Description of the bug
Versions are not reported in results/pipeline_info/software_versions.yml for diginorm/khmer or for pigz that is used in the UNPIGZ_MEGAHIT_CONTIGS-process.
Steps to reproduce
Steps to reproduce the behaviour:

Command line: nextflow run . -profile docker,test_diginorm --orf_caller prokka
See: results/pipeline_info/software_versions.yml

Expected behaviour
That all versions are reported, at least for the processes that are run by the tests.
Log files
Have you provided the following extra information/files:

[x] The command used to run the pipeline
[ ] The .nextflow.log file

System

Hardware: mac?
Executor: local
OS: macOS
Version 10.14.6

Nextflow Installation

Version: 21.10.6

Container engine

Engine: Docker
version: v20.10.11

Additional context

I should have fixed it in the last eggnog PR 🤞🏼

When I fixed the hmm classification, I noticed the module for versions was placed too early in the script. I fixed that, so I'm closing this, but let's keep an eye on this!


Update REVIEWERS

@googlebot I signed it!

Appreciate the offer but this system has been deprecated—see this message for info. Thanks


Adding more badges to the Readme page

RTD, appveyor and pypi.
The RTD is especially useful I think as there is not much feedback if the builds got stuck.

Thanks @bsipocz!


Unable to set icon

I have a project that I am currently running on Windows from the command line using nw.js (nw .).
I want to create an installation file and decided to make use of nwjs-builder-phoenix. Thanks for the capability!
I have followed the instructions to create a package.json. When I perform "npm run dist", I get the following error:
Error: rcedit.exe failed with exit code 1. Fatal error: Unable to set icon
at ChildProcess. (D:\AVBFlexSDKDistr\node_modules\rcedit\lib\rcedit.js:60:16)
I decided to try using the icon files in the example, pd.icns, but get the same error.
Given below is my package.json. Any help would be much appreciated! Should "npm run dist" create installation packages?
{
"name": "AVBFlex",
"version": "1.0.0",
"description": "AVBFlex",
"main": "index.html",
"scripts": {
"test": "echo "Error: no test specified" && exit 1",
"dist": "build --tasks win-x86,win-x64 --mirror https://dl.nwjs.io/ .",
"start": "run --x86 --mirror https://dl.nwjs.io/ ."
},
"author": "Richard Foss",
"license": "ISC",
"build": {
"nwVersion": "0.23.5",
"packed": true,
"targets": [
"nsis"
],
"win": {
"productName": "AVBFlex",
"companyName": "ImmersiveDSP",
"icon": "./assets/pd.icns"
},
"nsis": {
"installDirectory": "$PROGRAMFILES\${_ImmersiveDSP}\${_AVBFlex}"
}
},
"devDependencies": {
"nwjs-builder-phoenix": "^1.14.3"
},
"window": {
"icon": "./assets/pd.png"
}
}

For Windows you need an .ico file, the .icns file is used on macOS.

Oh no, sorry, that was careless of me! Thanks for the quick response.


MightyText recipe on Franz Mac doesn't appear

Hi guys
First of all thanks for Franz, it is such a great app! For me there is one service missing, which is MightyText. I wanted to make an own recipe to integrate it and then I found this repository:
https://github.com/brianjohnhanna/mightytext-franz-recipe
Everything seems included and from my point of view correct. I followed the instructions on how to make a recipe work on my local machine (dev) but unfortunately it doesn't appear in the service list at all.
I'm on a Mac High Sierra 10.13.3
The folder Franz under /Library/Application Support was missing in the first place, so I created the following folder structure
/Library/Application Support/Franz/recipes/dev/mightytext
and copied all files from the above mentioned repository.
Quit and restarted Franz, nothing happens. The service is not listed anywhere.
Am I missing something? Thanks in advance!

@MijoBarbaric There has been an open issue here for several months. Looks like it is currently tagged "Wait for Category Feature".
In the meantime, make sure you are adding it to the Library in your home folder, not the root library.
Try these steps:

With the Finder active, click on the "Go" menu in the toolbar, and then select "Go to Folder..."
Paste this in exactly: ~/Library/Application Support/Franz/recipes/
You will likely need to create the dev folder
Download the binary from here, unzip and add to the dev folder
Enjoy!

@brianjohnhanna That worked like a charm, thank you very much!


Add read region and tiff writer

This is a new attempt on top of #7. The difference now is that 1) I tried to fix any memory leaks (memorysanitizer is still not so happy)? 2) we can also set float offsets using pixman for bilinear interpolation (or whatever).
I am not sure we need the float interpolation, but I have tried using offset_in_pixels, but when using SlideScape with @Falcury's debug option, I still see a shift of the image happening. This doesn't happen if I load a real isyntax image, so somewhere I am using the wrong offset? See #17
Update: current version doesn't cause any shifts across levels.

Investigating further, this is an annotation using the Philips SDK:

This is the same region read from the tiff by the conversion:

So there appears to be a small offset, looking into it.

This is a new attempt on top of https://github.com/amspath/libisyntax/pull/7. The difference now is that 1) I tried to fix any memory leaks (memorysanitizer is still not so happy)?

I would like to help test this. Unfortunately though, the code still crashes for me on Linux, so I guess the problem is still there. I'll try to debug it, and I'll try on macOS as well to maybe more closely match your setup.

we can also set float offsets using pixman for bilinear interpolation (or whatever).

This is fine for testing of course, but once the offset problems are fixed, is it OK to get rid of this again? I don't really see this is as a good 'definite' solution, and I would really like to avoid adding external dependencies to the core library if at all possible...

I think the crashing issues in the TIFF converter could be due to out of bounds reading. We should probably add extra bounds checks in various places to catch these.
If I set the tile width and height to 256, and call libisyntax_tile_read() instead of libisyntax_read_region(), there is no crash and the conversion works:
// isytax_to_tiff.c line 117
    for (int32_t y_coord = 0; y_coord < height; y_coord += tile_height) {
        for (int32_t x_coord = 0; x_coord < width; x_coord += tile_width) {
            assert(tile_width == 256 && tile_height == 256);
            uint32_t *pixels = NULL;
            assert(libisyntax_tile_read(isyntax, isyntax_cache, scale, x_coord/256, y_coord/256, &pixels) == LIBISYNTAX_OK);

So one possibility is that libisyntax_read_region() may be reading out-of-bounds tiles (did not have time yet to fully test this).

Out of bounds reading perhaps could now be fixed by #21

@Falcury I've added a few asserts to make sure we don't access any tiles outside the region:
assert(tile_x >= 0);
assert(tile_y >= 0);
assert(tile_x < current_level->width_in_tiles);
assert(tile_y < current_level->height_in_tiles);

Does it crash on an assert for you?

I still see the issue, but it does not appear my code is writing beyond bounds (given the asserts). I'll run Valgrind over night (it takes quite a long time for the error to appear) and see what it comes up with.

I've tried adding all kinds of asserts to my code to check bounds but still the error is here according to Valgrind?

Could you try changing this at line 120 in isyntax_openslide_load_tile_coefficients_ll_or_h():
u8* codeblock_data = malloc(codeblock->block_size + 7); // allocate 7 extra safety bytes for isyntax_hulsken_decompress

It doesn't fix the leak for me, but maybe could solve that specific heap-buffer overflow error.

@Falcury Yes, that shuts up MemorySanitizer!

It no longer crashes for me, and the TIFF converter utility seems to produce correct output. :)
Incidentally, the results also look equally good for me with this formula for the offset (without subtracting 1.5f at the end):
float offset = ((PER_LEVEL_PADDING << (num_levels - level)) - PER_LEVEL_PADDING) * 0.5f;

It doesn't work for me in Release builds. We need to put statements outside the assert() statements to make sure they happen while not in Debug mode.
Some other observations and questions:

The percentage of the progress indicator can exceed 100%, maybe there is a bug in that calculation?
The ETA for pages > 0 also seems estimated too high?
Is there degradation of image quality due to the interpolation?

For merging:

If we need to keep the interpolation to get accurate offsets, how feasible is it to get rid of the pixman dependency again? For e.g. OpenSlide I guess it would be acceptable, but for e.g. Slidescape I would rather not use it.
To keep the git history somewhat clean, can we (eventually, when we want to merge) rebase and squash the commits?

@Falcury yes, the progress indicator is not 100% correct yet, this is one of those fiddly things that take a bit of tickering to get correct. I will fix it.
I can try to replace the interpolation, but pixman uses SSE and NEON where possible, it might be they optimized it further. Also, I now select bilinear (which seems to look good) but it has other options as well
I will rebase to main. Can we merge the height/width computation first? Perhaps you can also create a commit adding in the 7 safety bytes.

I will rebase to main. Can we merge the height/width computation first? Perhaps you can also create a commit adding in the 7 safety bytes.

Done!

Image degradation: not to me, but I’m not a pathologist. Do you have a few images you can inspect?

Here is a comparison, image sampled at level 2, without interpolation:

And with interpolation:

To me, the image without interpolation looks much crisper, and the interpolated image looks fuzzy.

Also when we are actually using interpolation, nothing is stopping us to do interpolation on float levels. You could then e.g write tiffs at arbitrary resolutions

True, I think this could be a valid use case. Still, for TIFF conversion specifically, I think the best way to do it is to only reconstruct level 0 (without any interpolation), and then rebuild the pyramid by resampling manually (bicubic should be fine for that, I think?)

 (E.g. using BEST)
To write to tiff based on level 0 has some drawbacks. We would need to handle anti-aliasing ourselves, which is not necessarily very trivial for these images as we’d also need to check against pathologist for quite a range of images I think.
Also using it for machine learning can be annoying as we would need to do the same thing during training as inference as we’d now be introducing different artefacts then the original isyntax. That would include a conversion step which I’d rather avoid.
Now, we could also figure out what the actual offset is based on the ‘selected mpp’ and apply that and don’t use interpolation (if we only select the levels themselves). That would require to add that to the deep learning pipelines but should be doable.
We typically use interpolation as we look at intermediate levels in the pyramid too.
that it looks less crisp is probably due to the low-pass filter effect of bilinear interpolation.

Fixed the counter

With checking against pathologists: these interpolation schemes are not necessarily the best for medical images or histopathology images, there are different approaches you can try as well. So you'd need a small test to see what is qualitatively better.

I did a small test comparing the offsets between levels in the Philips IMS-exported TIFF files vs. TIFF files converted by the SDK. For the SDK the offsets shift wildly, I guess the offset is not corrected for. For the TIFF files converted by IMS, the offsets still shift a bit, but roughly seem to correspond to:
int offset = ((PER_LEVEL_PADDING << (isyntax->images[0].level_count - level - 1)) - (PER_LEVEL_PADDING - 1));
This might be the 'least worst' option that Philips themselves arrived at for their TIFF conversion method. Between levels the shift amounts to 0,5-1 pixels each time. It's also visually very close to your own formula if you don't do the interpolation.

For the converted TIFF files, I notice that libjpeg-turbo is throwing warnings when decoding the JPEG streams:
Bogus JPEG colorspace
Apparently it's expecting 3 color components for PHOTOMETRIC_RGB, but getting 4. Not sure what's going on exactly.

I have seen that as well. If I read the tiff standard correctly I think what I did is correct (?) how are you using turbojpeg?

@Falcury I've tried to implement my own Mitchell-Netravali filtering, which is probably a good choice for images such as these. However, it's like 20x slower than pixman. I think it will be hard to avoid to pull in a dependency like that, unless we spent a lot of time on this all.

I have seen that as well. If I read the tiff standard correctly I think what I did is correct (?) how are you using turbojpeg?

I'm opening the file in Slidescape (which embeds the libjpeg-turbo code). Maybe in my TIFF interpreter, I am not dealing with the TIFFTAG_EXTRASAMPLES correctly. But I'm also not sure what libjpeg(-turbo) would expect here, if I try to set another color space than JCS_RGB it will throw an error complaining that the color conversion is not supported. I wonder how libtiff handles this.

@Falcury I've tried to implement my own Mitchell-Netravali filtering, which is probably a good choice for images such as these. However, it's like 20x slower than pixman. I think it will be hard to avoid to pull in a dependency like that, unless we spent a lot of time on this all.

For the default I'd like to match the way Philips IMS apparently exports TIFFs: no interpolation, using the 'least worst' integer shifts.
I think pulling in pixman is acceptable if it's unavoidable for your use case, as long as it's an optional dependency and not linked by default.
I'll try the Mitchell-Netravali filtering! I don't think this is available as a filter in pixman, though? (or is that PIXMAN_FILTER_BEST?)
I am still quite pessimistic that any interpolation technique that doesn't have access to lower-level pixels would be able to match the original image quality, though. I think ML performance on both training and inference is bound to suffer due to the information loss / blurry images, making the trade-off not worth it.
Have you considered resampling from (level - 1) for the random access use case? This would of course also incur a performance penalty, but may be worth it if you want the best of all worlds.

I think the approach resampling from (level - 1) might be a good option, indeed! But the shift is still there (but we could do that both in one go). If you want the default behavior of the Philips IMS export, then how would you want to do this with openslide? We can define how to calculate the shift; that might be an alright compromise.

@Falcury I will annotate the testimage.isyntax’s cells so we can inspect the drift and come up with mitigation strategies, or do you already have annotated isyntax images?

These are two boxes:
'[[[19932, 14987], [401, 322]], [[21336, 14843], [392, 351]]]'
This is what they look like (using Philips SDK in SlideScore):

And

Slightly worrying is that SlideScore has an offset of +1024 for this image, and +2048 for another one while they are directly reading from the SDK. I believe that's another offset we need to find somewhere. These points are corrected for that offset.
Here I use the Philips SDK:

Code:
from PIL import Image, ImageDraw

coords, size = w[1]
x = coords[0]
y = coords[1]
slide = OpenPhi("testslide.isyntax")

width = size[0]*2
height = size[1]*2

offset = 1024
x += offset
y += offset

level = 1
cells = [(a+offset-x, b+offset-y) for a, b in z]

data = slide.read_region((x, y), level, (width // 2**(level), height // 2**(level)))

# Create an ImageDraw object
draw = ImageDraw.Draw(data)

radius = 3
for cell in cells:
    xx, yy = cell
    xx /= 2**level
    yy /= 2**level
    draw.ellipse((xx-radius, yy-radius, xx+radius, yy+radius), outline=(255, 0, 0))

data

Will report back when I've converted the image using the above (integer) offset.

Initial findings: using the offset above gives a complete mismatch between SDK and tiff conversion, but it is indeed nearly constant

I am attempting to visualize the difference. This is a box:
def color_intensity(pixel1, pixel2):
    return int(sum(abs(p1 - p2) for p1, p2 in zip(pixel1, pixel2)) / 3)

def show_pixel_difference(image1, image2, highlight_color=(255, 0, 0)):
    if image1.size != image2.size:
        raise ValueError("Images must have the same dimensions.")

    diff_image = Image.new("RGB", image1.size)

    image1_rgb = image1.convert("RGB")
    image2_rgb = image2.convert("RGB")

    # Iterate through the pixels and highlight differences
    for y in range(image1.size[1]):
        for x in range(image1.size[0]):
            pixel1 = image1_rgb.getpixel((x, y))
            pixel2 = image2_rgb.getpixel((x, y))

            if pixel1 != pixel2:
                intensity = color_intensity(pixel1, pixel2)
                diff_image.putpixel((x, y), (intensity, 0, 0))
            else:
                diff_image.putpixel((x, y), (0, 255, 0))

    return diff_image

Level 0, box 1 (see above):

Setting an offset of 1 pixel:

You see more red appear = more offset. So I would say, is correct for level 0.
int32_t offset = ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING) >> level;
For completeness also offset of -1:

Let's go to level 1, offset 0 (I change the criteria for a hit to if if np.abs(pixel1[0]-pixel2[0]) + np.abs(pixel1[1]-pixel2[1]) <= 3:

Level 1 offset 1:

Level 1 offset 2:
Level 1 offset 3:

So it's not really correct yet, no.
Let's try with your formula + 128.

@Falcury I removed the interpolation as indeed the Philips SDK doesn't do it either (and has pixel shifts)

@Falcury @alex-virodov From the above I would like to conclude that the offset is this:
int32_t offset = ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING) >> level;
If would be interested to see what the float addition is to keep it 1-1 in SlideScape. Would it be possible to implement an offset in the debug window @Falcury?
I removed the dreaded pixman. If we can now find the proper offsets I can adjust my code as well to implement shifts in annotations across levels.

Thanks! I think the best would be to merge now and then fix any remaining issues. I'll rebase/fix the merge conflicts.
Some questions that I still have:

@Falcury @alex-virodov From the above I would like to conclude that the offset is this:
int32_t offset = ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING) >> level;

This is different than what you concluded above:

Initial findings: using the offset above gives a mismatch between SDK and tiff conversion, but it is indeed nearly constant. It required +128 (2**7) to end up like this:
This makes me believe the offset is this:
int num_levels = isyntax->images[0].level_count;
int offset = ((PER_LEVEL_PADDING << (num_levels - level - 1)) - (PER_LEVEL_PADDING - 1)) + (1 << num_levels);

Which is correct?

I am attempting to visualize the difference.

I am unsure how to interpret the images.
I was thinking to use phase correlation to get estimations of the optimal offsets (will try to do that).
Thanks for the annotations. Unfortunately I don't have the Philips SDK up and running in an easy way at the moment. I think instead I'll try to annotate an IMS-converted TIFF and try to match that back with the corresponding iSyntax.
# Add threads library for linux
find_package(Threads REQUIRED)

if(UNIX AND NOT APPLE)
    set(LIBRT rt)
endif()

# ...

target_link_libraries(isyntax_example PRIVATE Threads::Threads ${LIBRT})

Why is this needed in CMakeLists.txt? Doesn't seem required for me.
if(NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE Debug)
endif()

Why Debug by default? I am thinking for the average user, if they download the code from GitHub they will want an optimized build. You can easily override by passing -DCMAKE_BUILD_TYPE=Debug to CMake.

I think I finally figured out what is happening with the offsets, at least in the iSyntax files (not sure about the extra power-of-two offsets you mentioned). Let me know if you think the below code is correct.
def get_offset(level, num_levels):
    # The base image (level 0) has padding pixels added around it.
    # The formula determining the offset of the actual image is given by:
    # ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING)
    # For 9 levels (maxscale == 8) this is 1533 pixels.
    padding_offset = (PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING

    # After applying the DWT to get levels > 0, the pixels need to be visually shifted to the
    # bottom right in order to line up neatly with the lower level.
    # The amount for this visual shift is given by:
    if level == 0:
        visual_shift_in_terms_of_base_pixels = 0
    else:
        visual_shift_in_terms_of_base_pixels = (PER_LEVEL_PADDING << (level - 1)) - (PER_LEVEL_PADDING - 1)
    # For 9 levels this is: [0, 1, 4, 10, 22, 46, 94, 190, 382]

    # Note that for levels > 0, there will also be wavelet coefficients ending up 'beyond'
    # the top-left corner of the image into the padding area, as a kind of ringing artefact.
    # As a result, the 'first valid pixel' actually shifts to the top left for levels > 0
    # and does NOT correspond to the visual shift, which goes to the bottom right.
    # This can be seen e.g. when zooming in very far on top left of the topmost layer.

    # To shift each level so that they neatly line up with level 0, they need to be shifted
    # RELATIVE to the level 0 padding by the following amount:
    visual_shift_in_terms_of_level_pixels = visual_shift_in_terms_of_base_pixels / (1 << level)
    # For 9 levels: [0.0, 0.5, 1.0, 1.25, 1.375, 1.4375, 1.46875, 1.484375, 1.4921875]
    # Note that this offset seems to asymptotically approach 1.5 pixels.

    # Taking the above into account, to get the images of all levels to visually line up,
    # you would have to sample each layer at the following (subpixel) offset:
    effective_offset = (padding_offset - visual_shift_in_terms_of_level_pixels) / (2**level)
    # For 9 levels: [1533.0, 766.0, 382.25, 190.375, 94.4375, 46.46875, 22.484375, 10.4921875, 4.49609375]
    # For 8 levels: [765.0, 382.0, 190.25, 94.375, 46.4375, 22.46875, 10.484375, 4.4921875]
    # For 7 levels: [381.0, 190.0, 94.25, 46.375, 22.4375, 10.46875, 4.484375]

    return effective_offset

num_levels = 7
offsets = [0] * num_levels
for i in range(num_levels):
    offsets[i] = get_offset(i, num_levels)
print(offsets)

Rebased and merged: 7972197a943a93f58df6ed365714a23323dae859

If would be interested to see what the float addition is to keep it 1-1 in SlideScape. Would it be possible to implement an offset in the debug window @Falcury?

@jonasteuwen Done, hopefully this is what you had in mind: https://github.com/amspath/slidescape/commit/5639921c9bdcca343119ea6af0567cf5f2e9fa72
Screenshot:

Wow, @Falcury looks good! Yes, this way, we can try to see the offsets. I will check the merged code and compare with my previous setup to see if it's still matching nicely.
Phase plot might be interesting to see the difference. Here green = pixel matches within ~1.5 diff and red mismatch.

@Falcury What did you do, conversion is suddenly 5 times faster ;-)

Thanks!
If you're interested, I recently added phase correlation code to Slidescape for doing image registration:
https://github.com/amspath/slidescape/blob/master/src/utils/phasecorrelate.c#L159
https://github.com/amspath/slidescape/blob/master/src/core/image_registration.c#L271

You made some useful changes, it's suddenly ~1.5 minutes to convert.

I think that might be because the number of memory allocations got reduced by a lot after the recent API change by @alex-virodov. Now we can pass in our own buffer instead, which we only need to allocate once and then re-use.

@Falcury So basically if I run your code, and then do:
[_ - (1533 >> idx) for idx, _ in enumerate(offsets)] (1533 being the offset for level 0 calculation), I should be getting the relative offsets?
[0.0,
 0.25,
 0.0,
 0.46875,
 0.7265625,
 0.861328125,
 0.93017578125,
 0.9649658203125,
 0.982452392578125]

I can try that.

@jonasteuwen Hmm, then maybe the last part should be + instead of -? I could have made a mistake there. I don't really have a super thorough way to test it...
effective_offset = (padding_offset + visual_shift_in_terms_of_level_pixels) / (1 << level)
    # For 9 levels: [1533.0, 766.75, 383.5, 191.78125, 95.8984375, 47.951171875, 23.97607421875, 11.9881591796875, 5.994110107421875]
    # For 8 levels: [765.0, 382.75, 191.5, 95.78125, 47.8984375, 23.951171875, 11.97607421875, 5.9881591796875]
    # For 7 levels: [381.0, 190.75, 95.5, 47.78125, 23.8984375, 11.951171875, 5.97607421875]

Right, but 1533 should be the default offset now (that’s what matches with SDK), so the addition should be very small. 0.25 for the first layer seems correct.
What I do is convert the images to tiff and then fill in the small values in your new debug window

Right, but 1533 should be the default offset now (that’s what matches with SDK), so the addition should be very small. 0.25 for the first layer seems correct.
What I do is convert the images to tiff and then fill in the small values in your new debug window

The offsets in the debug window are in terms of pixels at the base level, so you would have to multiply them with the downsampling factor again.
This seems to come closer:
[(int(1533 / 2**idx) - _) * 2**(idx) for idx, _ in enumerate(offsets)]

For some reason, now the 0.25 is gone again. But the rest of the levels seem to look good.

@Falcury Can you give the full code please? The above seems to work to stabilize the image, except the first level.
How I test it is this: isyntax-to-tiff: check if it matches level 0 in Philips SDK (answer yes). Other levels also seem to work! But we would like to know the actual offset so we can offset annotations (or have a better viewer).

@jonasteuwen Here is the full code:
PER_LEVEL_PADDING = 3

def get_padding_downscaled(level, num_levels):
    offset = ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING) >> level
    return offset

def get_padding_downscaled_no_floor(level, num_levels):
    offset = ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING) / 2**level
    return offset

def get_offset(level, num_levels):
    # The base image (level 0) has padding pixels added around it.
    # The formula determining the offset of the actual image is given by:
    # ((PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING)
    # For 9 levels (maxscale == 8) this is 1533 pixels.
    # For 8 levels (maxscale == 7) this is 765 pixels.
    padding_offset = (PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING

    # After applying the DWT to get levels > 0, the pixels need to be visually shifted to the
    # bottom right in order to line up neatly with the lower level.
    # The amount for this visual shift is given by:
    if level == 0:
        visual_shift_in_terms_of_base_pixels = 0
    else:
        visual_shift_in_terms_of_base_pixels = (PER_LEVEL_PADDING << (level - 1)) - (PER_LEVEL_PADDING - 1)
    # For 9 levels this is: [0, 1, 4, 10, 22, 46, 94, 190, 382]

    # Note that for levels > 0, there will also be wavelet coefficients ending up 'beyond'
    # the top-left corner of the image into the padding area, as a kind of ringing artefact.
    # As a result, the 'first valid pixel' actually shifts to the top left for levels > 0
    # and does NOT correspond to the visual shift, which goes to the bottom right.
    # This can be seen e.g. when zooming in very far on top left of the topmost layer.

    # To shift each level so that they neatly line up with level 0, they need to be shifted
    # RELATIVE to the level 0 padding by the following amount:
    visual_shift_in_terms_of_level_pixels = visual_shift_in_terms_of_base_pixels / (1 << level)
    # For 9 levels: [0.0, 0.5, 1.0, 1.25, 1.375, 1.4375, 1.46875, 1.484375, 1.4921875]
    # Note that this offset seems to asymptotically approach 1.5 pixels.

    # Taking the above into account, to get the images of all levels to visually line up,
    # you would have to sample each layer at the following (subpixel) offset:
    effective_offset = (padding_offset - visual_shift_in_terms_of_base_pixels) / (1 << level)
    # For 9 levels: [1533.0, 766.0, 382.25, 190.375, 94.4375, 46.46875, 22.484375, 10.4921875, 4.49609375]
    # For 8 levels: [765.0, 382.0, 190.25, 94.375, 46.4375, 22.46875, 10.484375, 4.4921875]
    # For 7 levels: [381.0, 190.0, 94.25, 46.375, 22.4375, 10.46875, 4.484375]

    return effective_offset

num_levels = 9
offsets = [0] * num_levels
offsets2 = [0] * num_levels
offsets3 = [0] * num_levels
for i in range(num_levels):
    offsets[i] = get_offset(i, num_levels)
    offsets2[i] = get_padding_downscaled_no_floor(i, num_levels)
    offsets3[i] = get_padding_downscaled(i, num_levels)
print(offsets)
print(offsets2)
print(offsets3)

padding_offset = (PER_LEVEL_PADDING << num_levels) - PER_LEVEL_PADDING # 1533 for num_levels == 9

# Difference between the calculations in terms of pixels for each individual level
# (assuming we are doing interpolation)
print([((padding_offset / 2**idx) - _) for idx, _ in enumerate(offsets)])

# Difference between the calculations in terms of pixels for each individual level
# int() because we are currently snapping to whole numbers for TIFF conversion, without interpolation
print([(int(padding_offset / 2**idx) - _) for idx, _ in enumerate(offsets)])

# Difference expressed as pixels at the base level (multiply by downsampling factor again)
print([(int(padding_offset / 2**idx) - _) * 2**(idx) for idx, _ in enumerate(offsets)])

But we would like to know the actual offset so we can offset annotations (or have a better viewer).

Maybe it would make sense to embed the XPosition and YPosition TIFF tags in the converted TIFF? Then viewers could choose to use those to align the image on screen.
https://www.awaresystems.be/imaging/tiff/tifftags/xposition.html

Great plan! Let's verify if we can get it to work with the above code :-). How would we pass that through OpenSlide? Perhaps in the properties.

How would we pass that through OpenSlide? Perhaps in the properties?

I think it can be queried using "tiff.XPosition" and "tiff.YPosition":
https://openslide.org/docs/properties/

Right! Alright, let's do it this way. I'd be interested in finding a formula which gives the shift as function of the requested mpp (so we can do resampling and proper shifts).

@Falcury Let me see I got this right. The offset shown above in your example is defined in level 0. So to get it down to the level we want to compute the offset for we need to divide by 2**level. This then needs to be expressed in micrometers (or inches depending on the setting).
Do you interpret it the same way?

Yes! I think ResolutionUnits would be centimeters, not micrometes (that seems the only option in the TIFF standard apart from inches). We can keep it in terms of level 0, the conversion to ResolutionUnits is just a constant (should be 0.25 most of the time).

Right, indeed. I'll implement it after we merge your other PR.


remove the other category in the GitHub dashboards

Is your feature request related to a problem? Please describe.
The 'Other' category within the GitHub dashboards is currently aggregating a large number of events, which skews the data representation on the graphs. This makes it difficult to derive meaningful insights from the visualisations since the 'Other' category dominates the scale and detail of the graph.
Describe the feature you'd like
An enhancement to the data visualisation tool is required, where we can either filter out or break down the 'Other' category into more specific sub-categories. This would allow for a more granular analysis and a balanced view of event.
Additional context
In the current graphs, the 'Other' category eclipses other data points and renders the graph less useful for analysis. A feature to adjust the scale dynamically based on selected categories or an option to exclude certain categories would be beneficial. See attached screenshot for reference.

Thanks for opening this issue. This is a display limit controlled by QuickSight service. For line chart, QS displays 25 data points for color and the rest would fall in Other category. See https://docs.aws.amazon.com/quicksight/latest/user/line-charts.html
I would recommend you to filter data using the provided filters including author, repository, branch, AWS account, AWS Region, tag, or time period. If you tagged your resources, you can enter the tags in the solution's cloudformation stack then the solution would only show the tagged resources by default (https://docs.aws.amazon.com/solutions/latest/devops-monitoring-dashboard-on-aws/step-2.html). You can also customize the QS analysis and publish it to a new dashboard in your account as needed.

close the ticket due to inactivity.


micro cleanup

use sort_by since it is easier to read and faster for large lists
remove unnecessary assignment and return

Closing in favor of #677


Add finally block feature

Hi,
I developed a feature that I found missing.
The problem is that I want to continue the execution of a coroutine block even if there is an error, for example a loader must disappear after the request whether there is an error or not.
With current framework:
DispatchQueue.main.startCoroutine(in: self.scope) {

isLoading = true

    do {
        try self.service.getAll().await()
        try self.service.getAll().await() // Error occurred go to catch block
        try self.service.getAll().await()

        isLoading = false
    } catch {
        // Error occurred
         isLoading = false
    }
}

With this version as soon as there is an error we do not continue the block we go into the catch block. But the problem is that isLoading = false is written twice
I know I can also do this which will not stop the block but it is not what I reach, if there is an error I want the block to be stopped
DispatchQueue.main.startCoroutine(in: self.scope) {

isLoading = true

        try? self.service.getAll().await()
        try? self.service.getAll().await() // Error occurred but I dont want continue to the next request, I want hide the loader
        try? self.service.getAll().await()

        isLoading = false
}

So my solution is as follows, I wrap the coroutine block to make it clearer, Async / Await. And I added 2 blocks to catch the error, and a finally block which will be executed even if we get an error.
isLoading = true

async {
    try self.service.getAll().await()
    try self.service.getAll().await() // Error go to catch
    try self.service.getAll().await()
}.catch { error in
    Log.s("Error occurred:  \(error.localizedDescription)")
}.finally {
   // This block will be executed at the end of the block even if there is an error
   isLoading = false
 }

I already wrote this wrapper, if you are interested in integrating it into the framework I can make a pull request. This way will not replace the old way, it will just be a new possibility way

I like your approach but the main idea of this framework is to reduce the use of chains and other reactive programming approaches (although the framework includes everything you need to build chains, but it's like extra features).
The native Swift version should look like this:
DispatchQueue.main.startCoroutine {
   isLoading = true

   do {
      try self.service.getAll().await()
   } catch {
      Log.s("Error 1 occurred:  \(error.localizedDescription)")
   }

   do {
      try self.service.getAll().await()
   } catch {
      Log.s("Error 2 occurred:  \(error.localizedDescription)")
   }

   isLoading = false
}

You can do it a little better with extensions:
extension CoFuture where Value == Void {

   func await(catchBlock: (Error) -> Void) {
      do { try await() } catch { catchBlock(error) }
   }

}

DispatchQueue.main.startCoroutine {
   var isLoading = true

   self.service.getAll().await { error in
      Log.s("Error 1 occurred:  \(error.localizedDescription)")
   }

   self.service.getAll().await { error in
      Log.s("Error 2 occurred:  \(error.localizedDescription)")
   }

   isLoading = false
}

You can also create an extension for Sequence where Element: CoFuture<Void>, etc.

@mickaelhero You can also do so, it all depends on what errors you want to handle and how:
DispatchQueue.main.startCoroutine {
   isLoading = true
   defer { isLoading = false }

   do {
      try self.service.getAll().await()
      try self.service.getAll().await()
      try self.service.getAll().await()
   } catch {
      Log.s("Error occurred:  \(error.localizedDescription)")
   }
}

@mickaelhero Also, you can use the available methods to build chains:
DispatchQueue.main.coroutineFuture {
   try dd.await()
   try dd.await()
   try dd.await()
}.recover { error in
   Log.s("Error occurred:  \(error.localizedDescription)")
}.whenComplete {
   isLoading = false
}

But I'm not a fan of chains for a number of reasons. The idea of the async/await pattern is to reduce or avoid their use.

Thanks for your answer :)


loss function

A(code): loss = (1-self.lamb)*F.mse_loss(output, target) + self.lamb*dot_loss
B(paper): 𝐿𝑜𝑠𝑠 = 𝑀𝑆𝐸(𝑦, 𝑦̂) + 𝑆𝑂𝑅(ℎ̃x), 𝑆𝑂𝑅(𝑀) = 𝜆‖𝑀 ∙ 𝑀𝑇 − `𝐼‖^2

Can I think that A and B are the same? If so, I wonder what role (self.lamb) plays and what is the performance change depending on (self.lamb).

What does 𝑀 mean in the paper?

Yes you can think A and B are the same.
Dot loss refers to the value of soft orthogonality regularization (SOR), which calculates how well h, a two-dimensional matrix, grouped or orthogonalized the knobs. (self.lamb) acts as a weight to indicate how much importance to give to dot loss when training the model. Higher values of (self.lamb) result in better grouping of knobs. However, if (self.lamb) is too high, mse_loss will be reflected less, which can lead to poor learning. For this reason, you should set lamb high enough to take dot_loss into account, but not so high that the influence of mse_loss is ignored.

M is a two-dimensional matrix consisting of several vectors. The SOR is to determine whether the vectors in that matrix are orthogonal. In this paper, M can be regarded as a reshaped h.

Thank you for the comments!


Pass PSR-7 integration tests

We should pass https://github.com/php-http/psr7-integration-tests

@akrabat we don't now?
Will this not be a BC break for Slim 3?
Or is this targettng 4?

We don't, I think. Slim 4 thing.


Merge dev into master

Travis automatic deployment:
https://pr1076--safereact.review.gnosisdev.com/app

Travis automatic deployment:
https://pr1076--safereact.review.gnosisdev.com/app

Travis automatic deployment:
https://pr1076--safereact.review.gnosisdev.com/app

Build got stuck

Build got stuck


HIVE-23737: Reuse dagDelete Feature Of Tez Custom Shuffle Handler Instead Of LLAP's dagDelete

LLAP have a dagDelete feature added as part of HIVE-9911, But now that Tez have added support for dagDelete in custom shuffle handler (TEZ-3362) we could re-use that feature in LLAP.
There are some added advantages of using Tez's dagDelete feature rather than the current LLAP's dagDelete feature.

We can easily extend this feature to accommodate the upcoming features such as vertex and failed task attempt shuffle data clean up. Refer TEZ-3363 and TEZ-4129

It will be more easier to maintain this feature by separating it out from the Hive's code path.

@t3rmin4t0r Could you please take a look?

@prasanthj @rbalamohan Could you please take a look?

@rbalamohan Could you please review?


[HDX-41] Alert on aggregated values (P95, etc.)

Today we can only alert on the occurrence of log lines, it would be great if the user can create an alert on a chart in a dashboard as well (ex. if the p95 of duration exceeds X threshold for Y minutes, then alert).
From SyncLinear.com | HDX-41

Ah looks like the sync was disconnected for some reason on this issue - this was shipped a while ago!


Mountain buffers should only be initialized once.

The vertex and index buffer for the mountains should only be initialized once before the rendering loop starts. Recreating the buffers on every frame is costly and useless.

Grid vertex and index buffer are created once before the rendering loop starts. But the buffers themselves are still created per frame. This is another performance killer.

Done


feat: integrate label parsing to kili.labels()

This PR integrates the label parsing feature in the kili.labels() method
Also adds a tutorial for the label parsing
I'm open to suggestions when it comes to the method and parameter names
also add tests for poseestimation labels

@PierreLeveau how would you want to add the support for parsed label in other kili methods?
I'm not sure if it's a good idea, since other kili methods take json response arrays, not "labels"
so we would need to add another argument "label_array" for each label mutation method
I think it's reasonable to let the user do label["jsonResponse"], which is already the current state
However, we can add a .json_response property to the ParsedLabel class, in order to avoid doing parsed_label.to_dict()["jsonResponse"] which is quite ugly. We would have instead parsed_label.json_response
wdyt @kili-technology/ml ?

@PierreLeveau how would you want to add the support for parsed label in other kili methods?
I'm not sure if it's a good idea, since other kili methods take json response arrays, not "labels"
so we would need to add another argument "label_array" for each label mutation method
I think it's reasonable to let the user do label["jsonResponse"], which is already the current state
However, we can add a .json_response property to the ParsedLabel class, in order to avoid doing parsed_label.to_dict()["jsonResponse"] which is quite ugly. We would have instead parsed_label.json_response
wdyt @kili-technology/ml ?

I agree with you for all your remarks


Re-enable GPU Fusion pass.

It was accidentally disabled during the weekly sync a few weeks ago
/cc @ekuznetsov139

Can one of the admins verify this patch?


Add a project setting to enable stdout flushing in release builds

This can be used in server builds for journalctl compatibility.
This closes https://github.com/godotengine/godot-proposals/issues/1912.

Thanks!

Cherry-picked for 3.2.4.

Cherry-picked for 3.2.4.

Reverted before 3.2.4 beta 5 due to #44850.

Cherry-picked for 3.2.4.

Reverted before 3.2.4 beta 5 due to #44850.

Now that we've found a fix in master, can this be cherry-picked to 3.2 again with https://github.com/godotengine/godot/pull/45853 applied on top? Thanks in advance :slightly_smiling_face:

Cherry-picked for 3.2.4. (Together with #45853 squashed in the cherry-pick.)

I am still getting this error on ubuntu 20.04
Godot Engine v3.2.4.rc3.mono.official - https://godotengine.org

ERROR: get_language_code: Condition "p_locale.length() < 2" is true. Returned: p_locale
   At: core/translation.cpp:945.
ERROR: set_locale: Unsupported locale 'C', falling back to 'en'.
   At: core/translation.cpp:969.


[Quote] Gray 90/100 theme link wrong color

Description
For React version, in Gray 90 and Gray 100 themes, the link color for the CTA is wrong. The color should match Web components. Currently the React version shows $interactive01 as the color token, while WC uses $hoverPrimaryText.
React:

Web Components:

Component(s) impacted
Quote
Browser
Chrome, Safari
Carbon for IBM.com version
v1.33-rc.0
Severity
Severity 4 = The problem is not visible to or noticeable to an average user. Affects minor functionality, no workaround needed.
Application/website
C4IBM.com
Package
@carbon/ibmdotcom-react
CodeSandbox example
https://codesandbox.io/s/github/carbon-design-system/carbon-for-ibm-dotcom/tree/main/packages/web-components/examples/codesandbox/components/quote
Steps to reproduce the issue (if applicable)
No response
Release date (if applicable)
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

@RichKummer double checking specs

Updated reference to $link-01 per the specs


Add HTTP timeout option

These changes allows to define the timeout option of HTTPotion (as requested here).
I kept the same default value (5 seconds).

Coverage remained the same at 100.0% when pulling 3dcd5ba70e16c4c31361c2ccaa491a8a18859517 on primait:ISSUE-17 into 26c3622dbab08e19028ff7d73ef564da0d3e0b70 on SparkPost:master.

Thanks for the PR @davidefedrigo!


[AMD] Fixed instruction reorder

Fixed instruction reorder

improved pure-gemm pattern matcher

[x] I am not making a trivial change, such as fixing a typo in a comment.

[ ] I have written a PR description following these
rules.

[x] I have run pre-commit run --from-ref origin/main --to-ref HEAD.

Select one of the following.

[ ] I have added tests.

/test for lit tests
/unittest for C++ tests
/python/test for end-to-end tests

[x] This PR does not need a test because the current tests covers introduced changes

Select one of the following.

[x] I have not added any lit tests.
[ ] The lit tests I have added follow these best practices,
including the "tests should be minimal" section. (Usually running Python code
and using the instructions it generates is not minimal.)

@antiagainst, @zhanglx13 could you review the code?

Closing given superseded by https://github.com/triton-lang/triton/pull/5346.


Linux dependency installer fails for lttng-ust dependency.

Agent Version and Platform
Version of your agent? 2.179.0
OS of the machine running the agent? CentOS 8
Azure DevOps Type and Version
dev.azure.com (formerly visualstudio.com) or on-premises TFS/Azure DevOps Server? n/a
If on-premises, which release? n/a
If dev.azure.com, what is your organization name? n/a
What's not working?
Agent fails to pull the repo file at https://packages.efficios.com/repo.files/EfficiOS-RHEL7-x86-64.repo . This will fail for agent OSes that require this step. You should investigate how to mitigate any possible unreliability with this package source.
My particular agent will run on CentOS 8, which has the package lttng-ust as part of the CentOS/RHEL distro's AppStream repo. It appears that you're not making the distinction for CentOS/RHEL "8" in installdependencies.sh. Using the distro's lttng-ust will mitigate this issue for CentOS/RHEL "8"
This will likely affect other RHEL based distros (e.g. Amazon Linux). edit: installdependencies.sh doesn't/fails to run on AL2.
Error messages:
'lttng-ust' installation failed with exit code '0'
Can't install dotnet core dependencies.
failed: Connection timed out.
Connecting to packages.efficios.com (packages.efficios.com)|2607:5300:203:5aae::ef3d|:443... failed: Network is unreachable.
You can manually install all required dependencies based on following documentation
https://docs.microsoft.com/en-us/dotnet/core/dependencies?pivots=os-linux&tabs=netcore31

Hi @matthewhembree this problem is still actual to you? It looks like a temporary network problem with the repository server.

Hi @matthewhembree this problem is still actual to you? It looks like a temporary network problem with the repository server.

@sdobrodeev This is working now, but my concern shifted to being unable to deploy agents when that server is unavailable. Is there a way to mirror that dependency under Microsoft?

@matthewhembree Unfortunately, we can't provide mirror for CentOS and other internet resources due to maintenance concerns.


Fix the contents function on the repository API.

It appears that Github either changed their way of querying files
or directory or the bug was always there but say you want to do :
repo.contents(branch, "MYFILE", function(err, contents) {}, false);
it currently doesn't work and list only the root directory of the
repository. Changing slightly the way Github.js query the contents
make things works correctly as expected.

Both #92 and #127 solve this. If you can beat them to updating your PR, I'll merge this instead of those. :checkered_flag:

Closing as duplicate of merged PR.


Add SimpleApiClient in Networking category.

Project URL

https://github.com/jaychang0917/SimpleApiClient-ios
Description

Add SimpleApiClient in Networking category.
Why it should be included to awesome-ios (optional)
Checklist

[x] Only one project/change is in this pull request
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 9 / tvOS 10 or later
[x] Supports Swift 3 or later
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Warning

:warning:
Found 1 link issue, a project collaborator will take care of these, thanks :)

Link issue by awesome_bot

Line
Status
Link

1269
301
https://github.com/west0r/SwiftyVK redirects tohttps://github.com/SwiftyVK/SwiftyVK

  Generated by :no_entry_sign: Danger

Hey @jaychang0917, thanks for contributing! What do you think about the Reative Programming category?

Hi @lfarah sure, I updated:)

Thank you