Tesseract prints characters differ from lstmeval

My system info:

OS: Ubuntu Desktop 18.04 LTS (4.15.0-55-generic)

Hi.
I am beginner and am trying to train some Korean character images for Korean recognition.
To understand how to train with Tesseract 4.0 LSTM, I have trained my data from scratch by following lines of Makefile in this Tesstrain step by step, and most of steps seemed to work fine until creating traineddata.
These steps are what I did until now. I manually followed the steps instead of running make:

I made box files and unicharset by following this lines.

I made lstmf files by following this lines.

I made two split file lists for training and evaluation by following this lines.

Before combining lang model, I downloaded radical-stroke.txt by following this line, and 3 langdata files (kor.punc, kor.numbers, and kor.wordlist) from this link.
I didn't download kor.config file because it cause an error that chi_tra.traineddata is needed.

I combined lang model by following this lines.

Then I started LSTM training by following this lines.

I tested them. The results are like:

lim@ubuntu:~/tools/tesstrain$ usr/bin/lstmeval --traineddata data/kor/kor.traineddata --model data/kor/checkpoints/kor_checkpoint --eval_listfile data/kor/list.eval
data/kor/checkpoints/kor_checkpoint is not a recognition model, trying training checkpoint...
Loaded 1/1 lines (1-1) of document data/ground-truth/kor.malgun.exp249.lstmf
Loaded 1/1 lines (1-1) of document data/ground-truth/kor.malgun.exp228.lstmf
Truth:먹
OCR  :이
Truth:독
OCR  :이
Loaded 1/1 lines (1-1) of document data/ground-truth/kor.malgun.exp197.lstmf
Loaded 1/1 lines (1-1) of document data/ground-truth/kor.malgun.exp41.lstmf
Truth:파
OCR  :이
Truth:신
OCR  :열
... (skip)
At iteration 0, stage 0, Eval Char error rate=133.33333, Word error rate=96.875

There seems to be no problem with the results.
I know WER is abnormally high but I thought it doesn't matter because I just wanted to check whether the characters recognized by usr/bin/lstmeval are equal with the characters recognized by usr/bin/tesseract for a same image.

I made traineddata output file.

lim@ubuntu:~/tools/tesstrain$ usr/bin/lstmtraining --stop_training \
--continue_from data/kor/checkpoints/kor_checkpoint \
--traineddata data/kor/kor.traineddata \
--model_output usr/share/tessdata/kor.traineddata

Then I used tesseract with kor.malgun.exp197.tif. the TIF file was shown to '이' when I followed step 7 (testing with lstmeval). So I expected the same result.
lim@ubuntu:~/tools/tesstrain$ usr/bin/tesseract data/ground-truth/kor.malgun.exp197.tif stdout -l kor --psm 6 > result

But the real result was totally mess. As I was concerned, the recognized characters differed from each other.
Why the characters recognized by lstmeval and tesseract are different? Is it normal?
Thank you...

Why the characters recognized by lstmeval and tesseract are different?

I can confirm this with a test for Devanagari:
Loaded 1/1 lines (1-1) of document data/deva-lstmf/162.deva1.Sanskrit_2003,.exp0.lstmf
Loaded 1/1 lines (1-1) of document data/deva-lstmf/2214.deva1.Aksharyogini.exp0.lstmf
Truth:गूहितुं चित्रांश कुक्कुटनाडीयन्त्र असम्भवस्तु सतोऽनुपपत्तेः ॐ ॥२.३.९॥
OCR  :गृहितुं चित्रांश कुकुटनाडीयन्त्र असम्भवस्तु सतोऽनुपपत्तेः ॐ ॥२.३.९॥
Truth:। ददाशदस्मै दारिद्र्याद्ध्रियम् यकृत्कोपः महाराज धिष्ण्येमे सर्वा
OCR  :। ददाशदस्मै दारिद्र्याद्ध्रियम् यकृत्कोपः महाराज धिष्ण्येमे सर्वा
At iteration 0, stage 0, Eval Char error rate=2.8985507, Word error rate=14.285714

ubuntu@tesseract-ocr:~/tesstrain$ tesseract data/deva-boxtiff/162.deva1.Sanskrit_2003,.exp0.tif - -l devaLayer2.131 --tessdata-dir ./
Page 1
गूहितुं चित्रांश कुकुटनाडीयन्त्र असम्भवस्तु सतोऽनुपपत्तेः ॐ ॥२.३.९॥
ubuntu@tesseract-ocr:~/tesstrain$ tesseract data/deva-boxtiff/2214.deva1.Aksharyogini.exp0.tif - -l devaLayer2.131 --tessdata-dir ./
Page 1
। ददाशदस्मै दारिद्र्याद्ध्रियम् यकृत्कोपः महाराज धिष्ण्येमे सर्वा

lstmeval OCR:
OCR  :गृहितुं चित्रांश कुकुटनाडीयन्त्र असम्भवस्तु सतोऽनुपपत्तेः ॐ ॥२.३.९॥

tesseract OCR:
गूहितुं चित्रांश कुकुटनाडीयन्त्र असम्भवस्तु सतोऽनुपपत्तेः ॐ ॥२.३.९॥

Word which is different: गृहितुं vs गूहितुं

Thank you for

@stweil Should this issue be kept open?
Should lstmeval and tesseract give same output?

Yes, I think this needs more examination.

@stweil I bet that this is a PSM issue again. @JihwanLim @Shreeshrii Could you rerun your tests on the line level setting PSM to 13?

@wrznr I tried it with --psm 13 but it still gives same result. Only '^L' which is FORM FEED is shown in most cases.

@JihwanLim Could you provide us with a snippet from your test data? I'd like to reproduce your results...

@wrznr Sorry for making you wait.. I attached my Makefile. You can see new targets such as prepare, images, labels, fonts, and etc in the file but probably you don't need to care about them because they are just for generating new TIFF images.
Makefile.zip

@JihwanLim Many thanks. I'll have a look into your data set within the week and get back to you here.

@wrznr Thank you!

@JihwanLim What I meant with snippet from your test data was a small set of image-text pairs in order to examine the deviant behavior of lstmeval.
(Even if I download hangul.ttf I am missing the scripts get_data.py  and most importantly hangul-image-generator.py to reproduce your training setup.)

Okay I attached my project.

Extract tesstrain.tar.gz
Required packages are [here].(https://github.com/IBM/tensorflow-hangul-recognition/blob/master/requirements.txt)
Extract fonts.tar.gz and place malgun.ttf into tesstrain/fonts.
Build your Tesseract into tesstrain/usr.
Start from make unicharset.

And let me know if something is missing at any time, thank you.
tesstrain.tar.gz
fonts.tar.gz

Have you solved this problem? Since I met the same problem. I have a quite different OCR output compared with I use the "tesseract" command, using the same model. Could you please resond to me? Thanks

No. We have not. Thanks for pinging. I try to find some time for it next week!

No. We have not. Thanks for pinging. I try to find some time for it next week!

Thanks. I'm struggling with this problem for such a long time, but cannot find the reason/solution. Can you help me with this? Thank you so much

I think the problem is when you have to stop lstmtraining and convert to integer traineddata. you may have used traineddata generated by tesstrain.sh. you should better use traineddata best
lstmtraining
--stop_training
--convert_to_int
--continue_from ../tesstutorial/impact_from_full/impact_checkpoint
--traineddata tessdata/best/eng.traineddata
--model_output ../tesstutorial/impact_from_full/eng_impact_int.traineddata

any updates on this, I have the same problem, lstmeval and tesseract with psm -13 and the same traineddata do not match.

It seems I have the same issue on the training for couple of old russian glyphs (which makes it plus char training from russian). Actually this issue slaughters all the fun from the tesseract since I suspect that recognition with this bug fixed would be dramatically better. Can issue be prioritized, please?
By the way, are there any embedded debug support for the tesseract app which can be activated?

By the way, are there any embedded debug support for the tesseract app which can be activated?

yes, you can: build with debugging enabled and then enable any of the debug parameters you can see in tesseract --print-parameters (the most important of which is debug_file – must be non-empty to see any debug messages).

Why the characters recognized by lstmeval and tesseract are different? Is it normal?

Yes, it's not unlikely, since the latter is much more complex – e.g. because it contains image preprocessing, page segmentation, multi-model/lang and legacy engine code. The basic function is the same though:

lstmeval →
lstmtester →
lstmtrainer →
LSTMTrainer::PrepareForwardBackward →
LSTMRecognizer::RecognizeLine + LabelsFromOutputs
tesseractmain →
TessBaseAPI::ProcessPage →
TessBaseAPI::Recognize →
Tesseract::recog_all_words →
Tesseract::classify_word_and_language →
Tesseract::classify_word_pass1 →
Tesseract::LSTMRecognizeWord →
LSTMRecognizer::RecognizeLine + LabelsFromOutputs

I concur with @wrznr in surmising this is a PSM issue, but the OP already refuted that by trying PSM 13 to no effect.
@bilal-rachik brings in model finalization (esp. float→int conversion) which could play a role, esp. since the differences IIUC are rather small. Can anyone confirm this by trying without --convert_to_int?
It could also be related to thresholding or image normalization...

@bilal-rachik @bertsky Is this really a tesstrain issue?

Is this really a tesstrain issue?

You are right, this should probably be transferred to the tesseract repo.

is there any update here? I'm having this issue where I'm using eng.traineddata and I'm getting accurate results for some test .png's using tesseract, but nonsense using lstmeval. This is messing up my training. I'm wondering if, like mentioned above, I have some configs set incorrectly

@jhartungBE all we have at this point are suspicions (what to look for). Have you tried …

PSM=13 / --psm 13
with traineddata from tessdata_best /  without --convert_to_int
with/out adding language models, or
with --configfile <(echo thresholding_method 2) / -c thresholding_method=2

… yet?

Thanks for the quick response. Yes I have tried the first two options, but not sure what you mean on the latter two. Here's a simple example that explains my issue.
I have these two example image/text pairs in test-ground-truth. I can generate the box/lstmf files using  "make lists MODEL_NAME=test PSM=7".
I can then run "lstmeval --model eng_tessdata_best/eng.traineddata  --eval_listfile data/test/all-lstmf" and I get
Loaded 1/1 lines (1-1) of document data/test-ground-truth/Message_20210917_163.lstmf
Loaded 1/1 lines (1-1) of document data/test-ground-truth/Message_20211012_281.lstmf
Warning: LSTMTrainer deserialized an LSTMRecognizer!
Truth:any 19.25 legs?
OCR  :L
Truth:I got nothing better then 1
OCR  :BE te e  c
At iteration 0, stage 0, Eval Char error rate=94.444444, Word error rate=100

When I run in my tesseract repo, using, for example:
tesseract Message_20211012_281.png test.txt --psm 7
I get perfect match of I got nothing better then 1
Message_20211012_281.gt.txt

Message_20210917_163.gt.txt

@jhartungBE, like I said in my first comment, the Tesseract standalone CLI has much more than just the bare recognition of lstmeval – and that includes a check and compensation for inverse colours, like in your example.
So that's another issue (in fact, it's no issue IMO).

Okay understood. Thank you. However, I'm failing to understand how I can train tesseract if this is the case and lstm training doesn't really apply to my images the same way that the tesseract engine will? Does this just mean I have to modify my images to pass to lstm training so that they are received the same way the LSTMRecognizer will receive them when I'm running tesseract?

Yes, that's what it means. Just install ImageMagick and do a convert input.png -negate output.png

Great, thanks. Appreciate your help


Adding the ability to annotate services

This will enable configuration of the load balancer in various cloud providers.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: sudermanjr
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: linsun
Assign the PR to them by writing /assign @linsun in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Hi @sudermanjr. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I signed it!

/ok-to-test

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: rshriram, sudermanjr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS [rshriram]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sudermanjr: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
fe8c3bcfa9ff99a47f0f5df870b581497181aceb
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

I signed it!

force merging because the mixer test changes went in after this PR was submitted and this PR has no impact on any code.


Fixed typo in README.md

Supported file formats for fragment shader said .gs instead of .fs.

Thanks!


SCC example is missing a profile attachement

The example should show how to attach a profile for a scan to be complete.
I would add something like this:
## SCC Profile Attachment
##############################################################################
resource "ibm_scc_profile_attachment" "scc_profile_attachment_instance" {
  name        = "my-profile-attachement"
  depends_on  = [ibm_scc_instance_settings.scc_instance_settings]
  profile_id  = "a0bd1ee2-1ed3-407e-a2f4-ce7a1a38f54d" # CIS IBM Foundations v1.0.0
  instance_id = ibm_resource_instance.scc_instance.guid
  description = "scc-profile-attachment"
  scope {
    environment = "ibm-cloud"
    properties {
      name = "scope_id"
      # value = local.account_id
      value = ibm_resource_group.group.id
    }
    properties {
      name = "scope_type"
      # value = "account"
      value = "account.resource_group"
    }
    # properties {
    #   name = "exclusions"
    #   value = []
    # }
  }
  schedule = "daily"
  status   = "enabled"
  notifications {
    enabled = false
    controls {
      failed_control_ids = []
      threshold_limit    = 14
    }
  }
  attachment_parameters {
    parameter_name         = "tls_version"
    parameter_display_name = "IBM Cloud Internet Services TLS version"
    parameter_type         = "string"
    parameter_value        = "1.3"
    assessment_type        = "automated"
    assessment_id          = "rule-e16fcfea-fe21-4d30-a721-423611481fea"
  }
  attachment_parameters {
    parameter_name         = "ssh_port"
    parameter_display_name = "Network ACL rule for allowed IPs to SSH port"
    parameter_type         = "numeric"
    parameter_value        = "22"
    assessment_type        = "automated"
    assessment_id          = "rule-f9137be8-2490-4afb-8cd5-a201cb167eb2"
  }
  attachment_parameters {
    parameter_name         = "rdp_port"
    parameter_display_name = "Security group rule RDP allow port number"
    parameter_type         = "numeric"
    parameter_value        = "3389"
    assessment_type        = "automated"
    assessment_id          = "rule-9653d2c7-6290-4128-a5a3-65487ba40370"
  }
  attachment_parameters {
    parameter_name         = "ssh_port"
    parameter_display_name = "Security group rule SSH allow port number"
    parameter_type         = "numeric"
    parameter_value        = "22"
    assessment_type        = "automated"
    assessment_id          = "rule-7c5f6385-67e4-4edf-bec8-c722558b2dec"
  }
  attachment_parameters {
    parameter_name         = "rdp_port"
    parameter_display_name = "Disallowed IPs for ingress to RDP port"
    parameter_type         = "numeric"
    parameter_value        = "3389"
    assessment_type        = "automated"
    assessment_id          = "rule-f1e80ee7-88d5-4bf2-b42f-c863bb24601c"
  }
}

@lionelmace A new profiles submodule is being added in https://github.com/terraform-ibm-modules/terraform-ibm-scc/pull/30

A new attachments submodule was released in https://github.com/terraform-ibm-modules/terraform-ibm-scc/releases/tag/v1.2.0 and the complete example now shows how to use it.
For now it takes in a profile ID as input, but once the provider adds support to do a profile lookup by name (https://github.com/IBM-Cloud/terraform-provider-ibm/issues/5185) we will update the module to take a profile name as input


Typo in doc comments for Set.keys and ReadonlySet.keys

TypeScript Version:  4.0.2

Search Terms: typo lib Set ReadonlySet keys
Code
const set = new Set()
set.keys()
;(set as ReadonlySet<unknown>).keys()

Expected behavior:
Documentation for Set.keys and ReadonlySet.keys should be

Despite its name, returns an iterable of the values in the set.

(full stop/period at the end)
Actual behavior:

Despite its name, returns an iterable of the values in the set,

(comma at the end)
Playground Link: link
Related Issues:

Should just be a matter of changing the comments in src/lib


No Highlight in VSCode

Summary
SystemVerilog code highlighting is not working in VSCode (Verible extension installed).
Test cases and examples

The file type is SystemVerilog.
The formater is working, the linter is working, but code highlight not.
Additional context
I downloaded released bin from here, Ubuntu-20.04-focal-x86_64, and copy bin files to /usr/bin.
I built VS Code extension as the guideline says.
Thanks for everyone's help.

You can try with another extension that provides highlighting, and wouldn’t interfere with verible ls. I use this one.

Any other extension I try which activates on System Verilog code seems to disable the Verible extension. Is this a way round this @gasrodriguez ?

You can try with another extension that provides highlighting, and wouldn’t interfere with verible ls. I use this one.


Audit Log missing configuration instructions

I was wondering if there were some configuration instructions for the Audit log.
I'm running it in Strapi 3.4.6. From the Navigation plugin I managed to figure out this config:
{
    "audit-log": {
      "enabled": true,
      "exclude": [],
      "map": []
    }
  }

However nothing happens when I change any content records.
I couldn't quite figure out if I was meant to put things in map and if so, what it was.

Hi @AlexStansfield,
Great to hear from you! Thanks for the feedback and I will add instruction for this package soon.


[REUQEST] Also hosting ZNet EZSP NCP Zigbee coordinator firmware for ICC-A-1

@basilfx FYI @mtx512 now also build custom ZNet EZSP NCP Zigbee coordinator firmware for ICC-A-1

https://github.com/mtx512/efr32

https://github.com/mtx512/efr32/tree/master/icc-a-1

@MattWestb has posted documentation on how-to flash that firmware on ICC-A-1

https://github.com/MattWestb/IKEA-TRADFRI-ICC-A-1-Modul

I added this to the README.md: https://github.com/basilfx/TRADFRI-Hacking#ezsp

I added this to the README.md: https://github.com/basilfx/TRADFRI-Hacking#ezsp


check embedded code blocks

Description
This PR will add a new short python script that will find all embedded code blocks in our docs and parse them to see if there are any very obvious errors. We may even be able to lint the python ones, but this is unclear as of now.
What it does now:

Forces you to mark a code block with the type of language, if it is not a language, setting it to text is allowed
Will parse python, toml, yaml and json blocks and fail if they are not parseable

LGTM! @sh-rp but npm run start still runs some python right? didn't we plan to decouple this?
@rudolfix yes, I want to do these things step by step. In the next pr i plan to remove python stuff from the npm commands (except for the main build command) and remove duplicated snippet code, but for the latter I still have to do some tests.


[support] Remove "router state" messages from cell monitors

What
These descriptions are misleading because the presence of the rep or
guardian processes shouldn't be affected by the routers. I think they may
have been copy/pastad from another monitor by accident.
How to review
Code review only.
Who can review
Not @dcarley

works, merging


Sema: Fix two bugs with type resolution of generic arguments [5.10]

Description: When resolving the generic argument list of a generic type, we didn't check for certain invalid situations involving occurrences of pack expansion types. As a result we would crash instead of producing a diagnostic.
Risk: Very low.
Tested: New test cases added.
Radar: rdar://116713961 and rdar://116716014.
Reviewed by: @hborla

@swift-ci Please test


vi test fails when using trackRequestProgress

When running the vi test getting the below error:

For test case we are trying to mock the function trackRequestProgress but getting the below error
TypeError: Cannot redefine property: trackRequestProgress

@subhashreerautray, Blob.stream() is supported since Node.js 15.7.0. Which version of Node do you use? What is inside request.body that you pass to trackRequestProgress?

@subhashreerautray, it seems there is something wrong in the request.body that you pass to trackRequestProgress.
You can debug the code at https://github.com/teil-one/fetch-api-progress/blob/main/src/trackRequestProgress.ts#L84. Or simply inspect the value of request.body before passing the request to trackRequestProgress.


PC - Edge and IE Browsers - if there are multiple players in the same html document, collision can occur if one player uses the IMA plugin and the other uses the MailOnline plugin

During our internal testing, a collision was observed when there is more than one player in the same html document and at least one is loading the Brightcove IMA plugin and another loads the MailOnline plugin.  In this case, some of the players will lose their src attribute and are unable to play the content video.

One way to get around this problem is to put your players into their own iframe.
Another way to get around this problem is to use the adRenderer option to specify the 'ima' renderer regardless of the presence of DFP parameters in your prebid configuration options if ANY of the players in the same document load the IMA plugin.  Setting the adRenderer option to 'ima' will prevent the loading of the MailOnline plugin.

We have attempted to address this issue programmatically in the following way: If more than one Brightcove Player is detected within the same html document AND we detect that the Brightcove IMA plugin as been loaded into the document, then we will attempt to always use the IMA plugin as the renderer for the prebid plugin regardless of the presence of DFP parameters in the prebid configuration options.  See this issue:  xxx

The fix to this issue was made in the Brightcove prebid plugin.  See https://github.com/prebid/prebid-js-plugin-brightcove/issues/115


Question: How to unbind collapsible event listeners?

I am making a long-lived application (e.g. electron desktop app) that uses materialize-css. When I render a view with a collapsible component it runs $('.collapsible').collapsible(); after the element has been inserted. What should I run when it's about to be removed from the DOM?
Looking at the current code it looks like I need to do something like $('.collapsible').off('click.collapse'), but it would be nice if I could so something like $('.collapsible').collapsible(false); If there's not already a way to do this, would you please consider adding one? (Otherwise it may lead to user code coupling to implementation details, like my example calling .off)
(I think this only applies to dynamically generated fields since .collapsible is automatically called on DOM ready.)

This relates to https://github.com/Dogfalo/materialize/issues/4522

Added in 79cb76e


The compiled kernel fails to boot

When I use kernel-config-vmware to compile the Linux Kernel, I find that the compiled kernel cannot boot the OS successfully.
The linux kernel which I use is the latest development version downloaded from GitHub。
I compiled the Kernel with Ubuntu 22.04 Daily Build Operating System, also tested under it.

I think it may be the issue from the development branch of the kernel.
You should use Linux Kernel 5.15.x with the kernel config. (We have only adapted the 5.15.x at present.)
Kenji Mouri


pull request for e2e test at 11/12/2016 14:26:10

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Update Deploy_DNS Jenkins job URL

The Jenkins job is being moved from Carrenza Production to AWS Production with new URL:
https://deploy.blue.production.govuk.digital/job/Deploy_DNS
This PR updates the URL in this repo.
Ref:

Trello Card
govuk-puppet

I just fixed at the blue in the URL. The new gds-cli works as is.


support for react native

Your package is what i need to use for my react native project, but when i add the package and try to run my app through it, it gives me this error, I tried reinstalling and clearing my cache but that didn't help.
I am using react-native => 0.60, is there any requirement of the latest version for it?

Hi @AsadShah1995 ,
Thanks so much for your contribution. unfortunately, this library is made only for Reactjs not native.
Thank you.


Add MessagePack serialization format.

Sorry for the rather late review. Does this need a one-byte code or can we use a 2 byte code (0x80+)? We have at most 127 of them and we've already wasted too many without thinking.

I thought MessagePack is popular enough of a format that it was best placed with all of the other serialization formats.
I'm fine putting it anywhere though. I'll try 0x0200 unless you have another preference.

I thought MessagePack is popular enough of a format that it was best placed with all of the other serialization formats.

Unfortunately, it's not about placement, it's just that we have 127 codes to assign and have already assigned way too many. I'd prefer to move out 90% of the codes in that range but it may be a bit late for that.
0x0200 works.

Basically, we try to reserve single-byte codes for codes used in addresses (multiaddr, ipld, and multihash). Really, cbor, protobuf, rlp, and bencode should be moved elsewhere.

(and can you rebase on master?)

Got it, thanks for the explanation!
I noticed JSON is also missing and I was wondering how you feel about adding that first:
https://github.com/multiformats/multicodec/pull/141

Merge the json one.

Thanks, I've updated this PR to use 0x0201


Post processing example doesn't support resizing

How can Bevy's documentation be improved?
The post_processing example doesn't currently support resizing. Its stated intent is to be:

useful to implement your own post-processing effect such as
edge detection, blur, pixelization, vignette... and countless others.

If people copy this code to their own projects it would be really nice if it properly supported resizing.
I think there may be two viable solutions to this:

Write a system that resizes the full screen quad whenever the window size changes
Write a custom vertex shader that takes the view.width and height uniform values into account and is always snapped to the edges of the viewport.

I couldn't reproduce any issue with resizing, but after playing with values in the example a little more it's possibly due to having a 2x screen scale, and the mesh being sized with physical screen size. This should also be fixed.

I's a bit subtle in the example because there is nothing near the edges of the screen. If you set a small initial size using WindowDescriptor and then scale the window up, you can see it more clearly, though.

I just hit this problem and I couldn't figure out how to write this resizing system. From what I understand we need to resize render target image as well as the quad.
When I'm calling resize on the image I'm getting an error that "color" and "resolve" attachments have different sizes, and I can't understand why since attachments should be synced on the extract stage, right? And for the quad, I guess we should just create a new quad every time?

For future reference:

https://github.com/bevyengine/bevy/pull/5902 adds a full screen triangle with high level Material2d bevy functions
https://github.com/bevyengine/bevy/pull/5084 seems to implement a fullscreen triangle, but it's doing so via a render pipeline

In case it might be useful, for this example resizing works on Mac, but on Windows it doesn't and I needed to write a system to support it.

@IceSentry this issue can probably be closed now that we removed the old post processing example?

Yep, this isn't an issue anymore


refactor(gateway): send internal tx instead of thin tx

This change is

crates/gateway/src/utils.rs line 34 at r1 (raw file):
Previously, MohammadNassar1 (mohammad-starkware) wrote…
Let's discuss it F2F.

Why not external_tx_to_executable_tx?


Branch Content Discover

Branch Content discover Implementation
@aaustin @dmitrig01 @shekar9

Other than my concerns above, this is looking great!

👍 let's do this - get it in BMF first?


Object.isEmpty([]) returns false

I found this a little unintuitive:
Object.isEmpty([]) // return false

I was hoping for an isEmpty that worked regardless of type, that would behave something like:
Object.isEmpty([]) // return false
Object.isEmpty({}) // return false
Object.isEmpty('') // return false

If this is intentional, perhaps a mention in the docs with an example could reduce confusion.

No, you're definitely right, but Object.isEmpty('') is true... check that one again...
You're right about the array of course.

oh, Object.isEmpty('') works fine, I was just giving an example of other 'empty' types

Ah ok I see.
Now a question for you: Is the number 8 empty?
Technically it is, and should be....

Since calling isEmpty on a number doesn't really make a lot of sense, I think the safest, easiest to understand route would be to assume every number, including 0, is not empty.

Simply that as an object, a number "has no properties". I think that's a reasonable definition of isEmpty. Whatever case the method definition needs to be something equally easy to understand so as not to cause confusion.

Ok let's do that then as I think it makes the most sense... I'll put in the change for arrays....


build: convert build-related scripts to POSIX shell (#172)

Clean up scripts that are used throughout the build and use bash syntax in order to allow running them on platforms that don't have bash installed, since they don't really require any bash-specific features and can easily be converted to POSIX shell.

I am not sure I understand the purpose of this change. What platform does not have bash available for it that we want this change? It also adds another set of changes that will have to be merged and maintained btw Speedb and RocksDB.

Alpine Linux, which is used in CI, uses BusyBox by default, not Bash. This is easily solvable there by simply installing Bash, but I wanted to support platforms where that might not be possible. Though I agree that it adds unnecessary maintenance burden, so as long as there's no concrete requirement we probably shouldn't do it. Closing this PR for now.


Add lifecycle.prevent_destroy best practice to data-carrying resource's documentation

Change Description

Add lifecycle.prevent_destroy best practice to data-carrying resource's documentation

Required SDK Upgrades

N/a

Testing
N/a

Documentation change related to #688


Automate release process

For now, if you wish to publish a new release of the CLI you need to:

git checkout main
npm run release -- 2.0.0 for a full release (or  npm run release -- 2.0.0-beta --tag beta for a prerelease where --tag is the dist tag of your choice)
The script above will do all this for you

Check your in the right place
Check the version your trying to publish doesn't exist yet
Re-install npm dependencies from scratch
Run tests
Bump version with npm version ...
Publish package on npm
Push a git tag v<version>

Wait for the CI to run on the pushed git tag (the CI will create a draft github release with standalone tarballs attached to it)
Edit the newly created draft release to write release notes :memo: and publish the github release :rocket:

Package distribution
With the above process we will only distribute a NPM package + standalone tarballs on the Github Release. For the future we will need to distribute packages in common package managers (Homebrew, .deb, .rpm & .aur, Docker, Nix, Snap ...?)

I've had a lot of success with semantic-release, automatically creating major, minor, or patch releases based on the commits that come in. I'd be happy to take a swing at setting that up if you like.


Mention installing LiveServer globally in docs

The instructions are quite verbose now. I was writing it with newish Julia users in mind. Without being explicit, there is the risk to lose people who are like "... I don't understand, let's do something else"

Good stuff, thanks @rikhuijzer


Write Separator Suggestion Drop Down

A Write Separator introduced in #7. When selecting one of the drop down values (eg NUL Null), the UI is taken to the home page AKA http://home IP.

The JavaScript wasn't added 😔


Switch to contact sensor instead of outlet with a stateless switch for mute.

Now with ios 17 having built-in native history for contact sensors, i think it makes sense to use that instead of the outlet (and eve history). The other benefit is support for activity notifications which aren't supported on outlet. I think this would be a huge enhancement with the other being a mute stateless switch instead of an outlet. I can probably do this in a pull request if you think it makes sense.

Now with ios 17 having built-in native history for contact sensors

Huh?  Where do I find this?  Do you have documentation and/or screenshots?

here it is: https://support.apple.com/en-us/HT213743
@ebaauw

activity history is supported for garage doors, door locks, security systems, and contact sensors.

Ah yes, the Activity History_ under Security.  That seems to be a Home special, not available to other HomeKit apps.  I don't like that, as it lacks any graphical display, but it works in parallel to Eve history (for contact sensors).
Outlet seems to be the natural choice, for providing combined history for On/Off and Consumption, .  I agree, mapping the UPS state to On/Off doesn't make much sense, especially since the UPS state is read-only.  Also, I like the idea of push notification when my UPS switches to battery or reports an alarm (last time that happened, I heard the beep on the device, but hadn't noticed the Status Fault).
I don't think I've tried combining contact sensor and consumption history in a single accessory before.  Eve generally doesn't like combined history for sensors and actuators (lights, switches, outlets, where consumption is reported), but it wouldn't hurt to try.  Combined history for stateless switches with anything else doesn't work, as Eve thinks the device is an Eve button.

another option: add a contact sensor for ups status, and notifications, limit the outlet to enabling/disabling the beeper, and consumption history?

In v1.2.0.

Erik you are the man. I tested works perfect. Thank you so much!!! Amazing.


build: Upgrade pipelines to Node18 + upgrade nvm recommendation

Update the recommended versions of Node.js for developers to use, as well as the version used by CI, from version 14 to 18.

package-lock.json was changed from lock file format 1 to 3. This seems like a bad change since you are trying to stay on the same npm version. Probably revert that file.

Just a few things to comment on or make sure are required changes from me. Without doing new npm, package-lock.json should probably go back too as Craig suggested.

I was able to revert all of these file to their lts versions and everything worked locally:
package-lock.json
examples/apps/view-framework-sampler/src/views/reactView.tsx
examples/data-objects/diceroller/src/main.tsx
examples/hosts/app-integration/container-views/src/view.tsx
packages/test/test-end-to-end-tests/src/test/deRehydrateContainerTests.spec.ts
I suggest reverting those files and seeing if CI still passes.

CI did fail on examples/apps/view-framework-sampler/src/views/reactView.tsx which is what I had seen too. I don't know how your build worked, Tyler. Of course I don't know why it actually fails either. Mysteries of lerna managed dependencies. Home stretch.

In reply to:<PHONE_NUMBER>

The packages/test/test-end-to-end-tests/src/test/deRehydrateContainerTests.spec.ts file was not reverted because that fix and e2e test that hangs.
For details this is the comment that Abram did in the fix:
0.58.x doesn't have the fix to routerlicious-driver in this PR: https://github.com/microsoft/FluidFramework/pull/8913
(also see related issue for more context on the problem: https://github.com/microsoft/FluidFramework/issues/9163)
This causes this test to hang while loading container2, as the snapshot is over 16KB.


Unsupported processor model (151)

I am trying to measure (and optimise) power consumption on my FreeBSD server.
I have tried turbostat --show PkgWatt,CorWatt but it doesn't print any metrics.
Next I tried your PCM tools:
# pcm-power

 Intel(r) Performance Counter Monitor (2024-05-23 09:12:02 +0200 ID=e9a1f396)

 Power Monitoring Utility

=====  Processor information  =====
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Max CPUID level          : 32
CPU model number         : 151
Number of physical cores: 4
Number of logical cores: 8
Number of online logical cores: 8
Threads (logical cores) per physical core: 2
Num sockets: 1
Physical cores per socket: 4
Last level cache slices per socket: 4
Core PMU (perfmon) version: 5
Number of core PMU generic (programmable) counters: 8
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 4
Width of fixed counters: 48 bits
Nominal core frequency:<PHONE_NUMBER> Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: yes
The processor supports enhanced IBRS                     : yes
Package thermal spec power: 35 Watt; Package minimum power: 0 Watt; Package maximum power: 0 Watt;

Socket 0: 0 PCU units detected. 0 IIO units detected. 0 IRP units detected. 0 CHA/CBO units detected. 0 MDF units detected. 0 UBOX units detected. 0 CXL units detected. 0 PCIE_GEN5x16 units detected. 0 PCIE_GEN5x8 units detected.

Unsupported processor model (151).
Cleaning up
 Zeroed PMU registers
 Zeroed uncore PMU registers

My CPU is a "Intel Core i3 12100T".
Thank you for your help!

Hi,
I'm sorry to say this but this is the case here:
https://github.com/intel/pcm/blob/master/doc/FAQ.md#q5
You have a client CPU, nothing we can do because there is no HW support for that in client CPUs.
Cheers,
Otto

OK, thanks


Move cursor to the end of the input during onFocus={true}

I am currently using the onFocus={true} on my inputProps for this component, how can I go about setting the cursor to the end of the auto-suggest input?
It is currently focusing on the input of the autosuggest component once the component has been added to the DOM, but it is placing the cursor at the start of the input.

To get the input field see this example. Then, use inputProps.onFocus to move the cursor to the end. This StackOverflow question may help. Closing this issue as it seems more like a generic Javascript question than a react-autosuggest question.


Install from source on Ubuntu: internal compiler error: Segmentation fault case RankType::k2... RankType::kK: {

Hello,
I am trying to install open_spiel from source (after several failed attempts using pip directly).
Ubuntu 20.04.2 LTS
cmake version 3.16.3
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Everything runs fine per the installation instructions until
./open_spiel/scripts/build_and_run_tests.sh
[ 57%] Building CXX object games/CMakeFiles/games.dir/stones_and_gems.cc.o
/home/awinter/Documents/open_spiel-master/open_spiel/games/solitaire.cc: In member function 'std::vector<open_spiel::solitaire::Card> open_spiel::solitaire::Card::LegalChildren() const':
/home/awinter/Documents/open_spiel-master/open_spiel/games/solitaire.cc:394:42: internal compiler error: Segmentation fault
           case RankType::k2... RankType::kK: {
                                          ^~
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://gcc.gnu.org/bugs/> for instructions.
make[2]: *** [games/CMakeFiles/games.dir/build.make:752: games/CMakeFiles/games.dir/solitaire.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:7445: games/CMakeFiles/games.dir/all] Error 2
make: *** [Makefile:95: all] Error 2

This is a bit beyond my skill set to try and solve. I was wondering if anyone had an idea on how to fix this.
Thanks!

Digging through more of the output from the builf script I found this too
[ 36%] Linking CXX executable matrix_example
/home/awinter/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: ../CMakeFiles/open_spiel_core.dir/spiel.cc.o: in function `open_spiel::Game::ToString[abi:cxx11]() const':
spiel.cc:(.text._ZNK10open_spiel4Game8ToStringB5cxx11Ev+0x3d2): undefined reference to `open_spiel::GameParametersToString(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, open_spiel::GameParameter, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, open_spiel::GameParameter> > > const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/CMakeFiles/matrix_example.dir/build.make:160: examples/matrix_example] Error 1
make[1]: *** [CMakeFiles/Makefile2:5319: examples/CMakeFiles/matrix_example.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 36%] Building CXX object games/CMakeFiles/bridge_double_dummy_solver.dir/bridge/double_dummy_solver/src/dump.cpp.o

Seems like g++ is being used. Is clang++ in your path? Can you try prepending the build script command with CXX=clang++ ?

Ok thanks! I tried what you said and got this error, which is really weird
./open_spiel/scripts/build_and_run_tests.sh
clang++ not found (the clang C++ compiler is needed to compile OpenSpiel). Exiting...

but I have clang installed.

Can you run the following and report the output:
clang++ --version
which clang++

clang++ --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
which clang++
/usr/bin/clang++
Also thank you for the help over the weekend.

No problem, problems like these are usually easy.
Can you now run
rm -rf build venv
CXX=/usr/bin/clang++ ./open_spiel/scripts/build_and_run_tests

Same error
rm -rf build venv
CXX=/usr/bin/clang++ ./open_spiel/scripts/build_and_run_tests.sh
clang++ not found (the clang C++ compiler is needed to compile OpenSpiel). Exiting..

Can you try editing the build script and just commenting out that check? Should be near the top of the file.

So that works to a point and then the whole system hangs up during the test phase and doesn't recover.

I have to do a hard shutdown. Not sure where the log would be at to see what's going wring.

But this runs examples/example --game=tic_tac_toe just fine in the build directory.

But trying this outside of the build dir
python3 open_spiel/python/examples/example.py --game=breakthrough
Traceback (most recent call last):
File "open_spiel/python/examples/example.py", line 26, in
import pyspiel
ImportError: /home/awinter/Documents/open_spiel-master/build/python/pyspiel.so: undefined symbol: PyThread_tss_set

Tried again and it gives me the those weird green bars, locks up everything but after 3 mins everything is fine again. Ran the example from build and then went up one dir and ran the python3 ... examples.
Looks like we are good to go. Thanks!

Great, enjoy!


Add Devcontainer/Codespaces Specifications

Describe the solution you would like
Devcontainer specifications provide a reproducible environment with pre-installed software dependencies. They make it easier for new contributors / collaborators to jump into a project and make contributions instead of spending time setting up their dev environment.
Additional context
If this feature is of interest, I would be willing to contribute the necessary code.

Hi @kenibrewer, thanks for the suggestion! I see devcontainers being used more and more around me in the companies I work for, so I'd fully support this feature. I look forward to your contribution and I will try to review quickly once you have a proposal!


Getting error "No module named 'groundingdino'"

I get this module import error when I attempt to run the grounded_sam.ipynb notebook

ModuleNotFoundError                       Traceback (most recent call last)
 in <cell line: 11>()
9
10 # Grounding DINO
---> 11 import GroundingDINO.groundingdino.datasets.transforms as T
12 from GroundingDINO.groundingdino.models import build_model
13 from GroundingDINO.groundingdino.util import box_ops
/content/Grounded-Segment-Anything/GroundingDINO/groundingdino/datasets/transforms.py in
11 import torchvision.transforms.functional as F
12
---> 13 from groundingdino.util.box_ops import box_xyxy_to_cxcywh
14 from groundingdino.util.misc import interpolate
15
ModuleNotFoundError: No module named 'groundingdino'

It seems like you fail to install the grounding dino library. Please make sure you perform each cells from the top of the notebooks. Or you can run python -m pip install -e GroundingDINO .

I ran that code block but same error. I even tried running pip install -e . from within the GroundingDINO folder...same issue

Can you help to check if the lib is installed successfully? By pasting the log after performing the python -m pip install -e GroundingDINO

If I run it a second time this is the output
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Obtaining file:///content/Grounded-Segment-Anything/GroundingDINO
Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (2.0.0+cu118)
Requirement already satisfied: torchvision in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (0.15.1+cu118)
Requirement already satisfied: transformers in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (4.27.4)
Requirement already satisfied: addict in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (2.4.0)
Requirement already satisfied: yapf in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (0.32.0)
Requirement already satisfied: timm in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (0.6.13)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (1.22.4)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (<IP_ADDRESS>)
Requirement already satisfied: supervision==0.3.2 in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (0.3.2)
Requirement already satisfied: pycocotools in /usr/local/lib/python3.9/dist-packages (from groundingdino==0.1.0) (2.0.6)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from supervision==0.3.2->groundingdino==0.1.0) (3.7.1)
Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.9/dist-packages (from timm->groundingdino==0.1.0) (0.13.4)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/dist-packages (from timm->groundingdino==0.1.0) (6.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (3.0)
Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (2.0.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (1.11.1)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (4.5.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (3.10.7)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from torch->groundingdino==0.1.0) (3.1.2)
Requirement already satisfied: cmake in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch->groundingdino==0.1.0) (3.25.2)
Requirement already satisfied: lit in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch->groundingdino==0.1.0) (16.0.0)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from torchvision->groundingdino==0.1.0) (2.27.1)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.9/dist-packages (from torchvision->groundingdino==0.1.0) (8.4.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from transformers->groundingdino==0.1.0) (23.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers->groundingdino==0.1.0) (2022.10.31)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/dist-packages (from transformers->groundingdino==0.1.0) (4.65.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.9/dist-packages (from transformers->groundingdino==0.1.0) (0.13.3)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (5.12.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (1.0.7)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (4.39.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (0.11.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib->supervision==0.3.2->groundingdino==0.1.0) (2.8.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->torch->groundingdino==0.1.0) (2.1.2)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision->groundingdino==0.1.0) (1.26.15)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision->groundingdino==0.1.0) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision->groundingdino==0.1.0) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision->groundingdino==0.1.0) (3.4)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch->groundingdino==0.1.0) (1.3.0)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib->supervision==0.3.2->groundingdino==0.1.0) (3.15.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/dist-packages (from python-dateutil>=2.7->matplotlib->supervision==0.3.2->groundingdino==0.1.0) (1.16.0)
Installing collected packages: groundingdino
Attempting uninstall: groundingdino
Found existing installation: groundingdino 0.1.0
Can't uninstall 'groundingdino'. No files were found to uninstall.
Running setup.py develop for groundingdino
Successfully installed groundingdino-0.1.0

same issue here

Same issue, the first cell in the notebook ran without errors and groundingdino-0.1.0 was installed successfully.

Fixed the problem by installing the modules from their respective git repos (didn't bother cloning this one).
!pip install git+https://github.com/facebookresearch/segment-anything.git
!git clone https://github.com/IDEA-Research/GroundingDINO   %cd /content/GroundingDINO   !pip install -e .
! pip install diffusers transformers accelerate scipy safetensors
I also changed how I import the models (I also commented out the transforms import because it's not used in the notebook.
import argparse
import os
import copy

import numpy as np
import torch
from PIL import Image, ImageDraw, ImageFont
from torchvision.ops import box_convert

# Grounding DINO

from groundingdino.models import build_model
from groundingdino.util import box_ops
from groundingdino.util.slconfig import SLConfig
from groundingdino.util.utils import clean_state_dict, get_phrases_from_posmap
from groundingdino.util.inference import annotate, load_image, predict
# import groundingdino.datasets.transforms as T

import supervision as sv

# segment anything
from segment_anything import build_sam, SamPredictor
import cv2
import numpy as np
import matplotlib.pyplot as plt

# diffusers
import PIL
import requests
import torch
from io import BytesIO
from diffusers import StableDiffusionInpaintPipeline

from huggingface_hub import hf_hub_download

Also, ensure you're in the GroundingDINO folder before importing

Also, ensure you're in the GroundingDINO folder before importing

Sorry for reopening, but I have a question. Which versions of Python, PyTorch, Torchvision and CUDA are you using?

Also, ensure you're in the GroundingDINO folder before importing

Sorry for reopening, but I have a question. Which versions of Python, PyTorch, Torchvision and CUDA are you using?

No worries, here's the output
Python version: 3.9.16 (main, Dec  7 2022, 01:11:51)
[GCC 9.4.0]
PyTorch version: 2.0.0+cu118
Torchvision version: 0.15.1+cu118
CUDA version: 11.8

Thank you!

Nope I'm not
On Mon, Apr 10, 2023 at 5:44 PM Stefan Jaspers @.***>
wrote:

Thank you!
—
Reply to this email directly, view it on GitHub
https://github.com/IDEA-Research/Grounded-Segment-Anything/issues/43#issuecomment-1502429423,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AI4IAWCMASYFHFX5FEZMLWDXASEMPANCNFSM6AAAAAAWY5PS3A
.
You are receiving this because you modified the open/close state.Message
ID: @.***
com>

Thanks for pointing out the issue, we will fix it soon.
The simple way to solve this is to add:
sys.path.append(os.path.join(os.getcwd(), "GroundingDINO"))

after the installations.


Implemented macOS file drag and drop

Added support for dragging osz files into the osu!lazer window from Finder, part of which includes copying the Linux namespace into a separate MacOS namespace.
The native calls are implemented using reflection by hooking into the internal interop classes provided by OpenTK.
I noticed that OpenTK 3.0.5 officially supports drag and drop, so if the projects get updated to peppy's latest fork, this can be replaced with the FileDrop event.
TODO: Implement macOS clipboard support.

The guy responsible for adding support to OpenTK is one of the contributors on this project. We've already on a version of opentk which should support using the implemented method, so please use that.

Awesome, will do so. :)

You may want to jump on discord if you aren't already (https://discord.gg/ppy)

I jumped on the osu!dev Discord a couple months back but I never got around to doing the verification process.

@swoolcock Did you test it with different languages? How its working?

I'll probably redo it once osu! moves to OpenTK 3.0.5 and I can use the built-in event handlers.

It's because method that are in code is similar to what OpenTK uses so it will be nice to get some feedback and to fix mistakes in OpenTK before moving to new version. Drag and drop code already get many bug reports(X11 and Windows) and only one that didn't get any is OSX so I suspect it has a lot of them.

OpenTK (and hence my implementation) uses the NSFilenamesPboardType which I believe is now deprecated.  I'll have a mess around in Xcode and see if I can get a good implementation working, then I'll look at replicating it with interop.

I think the best way to handle this is to accept file URLs as provided by Finder, since they should not be affected by localisation.
Example (Swift 3):
// register the dragged types with the NSWindow instance referenced by the OpenTK Window.WindowInfo.Handle
window.register(forDraggedTypes: [NSURLPboardType])

// override functions in NSWindow
override func draggingEntered(_ draggingInfo: NSDraggingInfo) -> NSDragOperation {
  return draggingInfo.draggingPasteboard().canReadItem(withDataConformingToTypes: ["public.file-url"]) ? .copy : NSDragOperation()
}

override func prepareForDragOperation(_ draggingInfo: NSDraggingInfo) -> Bool {
  return draggingInfo.draggingPasteboard().canReadItem(withDataConformingToTypes: ["public.file-url"])
}

override func performDragOperation(_ draggingInfo: NSDraggingInfo) -> Bool {
  let filenames = draggingInfo.draggingPasteboard().pasteboardItems?.flatMap{$0.string(forType: "public.file-url")}
  // TODO: handle files here
  return true
}

The file URLs will be in the format file:///.file/id=x.y where x and y are numbers rather than the filename (thus avoiding localisation problems).
I could implement this myself into osu-framework, or it could be added to OpenTK.  If I add it to osu-framework I'll try to avoid unnecessary reflection if possible.

I'll most likely force push a cleaner version of this.

As I see OpenTK reference now point to myget OpenTK build, so you can try to use d&d things from it. Also osu win implementation of d&d should be removed, as it must become platform-independent from osu-framework look.  I can help with this if you want.

I have some concerns with using the current OpenTK implementation of D&D.  I disagree with the idea of firing off individual events per file, and I feel that having strings as file paths may bring in localisation problems.
Additionally, I believe the NSFilenamesPboardType has been deprecated in favour of file URLs (I am aware that my implementation also uses NSFilenamesPboardType, but I plan on changing this).

What do you mean about localization? String is UTF-16 and you can convert it to whatever you want, if you cannot, than it OpenTK bug in conversion and will be solved at their side(correct me if not). It's easier to make os dependent things at OpenTK side because it handle main event loop(X11 heacy dependent on that as well as SDL2 I suppose).  About sending one string per event, if it become hell to work with it may be changed, api is still new.

Ultimately, i see the framework being able to handle dragging anything onto it, not just files.  Examples for osu!:

Drag an osz link from the osu! website and have osu!direct automatically download it
Drag any plain text and have it appear in the currently focused text box, as if it had been typed
Drag an image from a website or image editor into the storyboard editor without having to save it to disk first
Drag a sound snippet from a sound editor into the storyboard editor
Anything else that's not actually a file...

In this case, the use of NSFilenamesPboardType will not provide the functionality required, so the usage (and naming) of FileDrop is inappropriate.  If OpenTK cannot provide this at the moment, I would suggest rolling our own into the framework, then possibly making a pull request to OpenTK and using that if it is accepted.

It's already can be implemented and it's not connected to file drop
Same as above
good idea
also nice

But it's not file drop thing, it's data(links, text, raw data.

Should be reimplemented in a more platform-agnostic way, now that we're running an OpenTK version with native support.


[Diag] Create frontend flags for localization and write tests

In the last PR #32483, I introduced localization support for diagnostics via file-per-language store in YAML format.
In this PR I'm introducing the following:

Retrieve the diagnostic messages path from the compiler toolchain.
Create a -locale compiler frontend flag for choosing the language.
Create a -localization-path compiler frontend flag to provide a path for the YAML files (to use in testing).
Write tests.

The tests handle these cases:

If the language locale code isn't supported. In this case, we won't crash the compiler but instead show an error message of the supported locale codes.
If the path provided to -localization-path is incorrect i.e. the directory is missing the YAML files. We won't crash the compiler but instead, we should fall back to the default messages from the current .def files.
If the current <lang>.yaml doesn't have a translation for a particular diagnostic message it should fall back to the English message found in .def files.
Finally, if the message is translated and available it should, of course, show the translated message.

In the future PRs, I should handle lint tools for YAML files and prune tools to prune the diagnostics that gets removed from .def files.
cc @xedin

@LucianoPAlmeida in #32483 you were justing using an assertion rather than an if statement here:
https://github.com/apple/swift/blob/master/include/swift/AST/DiagnosticEngine.h#L743
Can you please, explain more why to do that?

@HassanElDesouky I can just on that an tell you why - since it's expected that locale and path are always there when setLocalization is called it's usually good to validate that assertion with assert to make sure that it's always true :)

@HassanElDesouky I can just on that an tell you why - since it's expected that locale and path are always there when setLocalization is called it's usually good to validate that assertion with assert to make sure that it's always true :)

Exactly how @xedin said :)
It is a good way to make it easier for the caller to spot the error if it passed the wrong value by mistake. In this case, just silent not setting the locale based on an implementation detail condition not known by the caller, could lead to a "why is not setting?" and some time after that trying to figure out.

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - 935ac0c8e7a8f48a7eae241ee3acf5898961fdd7

Build failed
Swift Test OS X Platform
Git Sha - 935ac0c8e7a8f48a7eae241ee3acf5898961fdd7

I'm going to need to revert this. The driver is double faulting on my Linux box (Fedora 32 / x86-64). It crashes due to heap corruption, then hangs trying to introspect and print the stack. There also appears to be an unrelated(?) uninitialized memory error that valgrind detects (as opposed to malloc's own self diagnostics):
 $ env SDKROOT=/ valgrind '/home/dave/b/q/t/bin/swift' -module-cache-path '/home/dave/b/q/t/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache' -swift-version 4  -Xfrontend -ignore-module-source-info -locale NOT_A_LOCALE
==2090498== Memcheck, a memory error detector
==2090498== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2090498== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==2090498== Command: /home/dave/b/q/t/bin/swift -module-cache-path /home/dave/b/q/t/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache -swift-version 4 -Xfrontend -ignore-module-source-info -locale NOT_A_LOCALE
==2090498==
==2090498== Syscall param execve(envp[i]) points to uninitialised byte(s)
==2090498==    at 0x983B45B: execve (in /usr/lib64/libc-2.31.so)
==2090498==    by 0x256386A: swift::driver::Compilation::performSingleCommand(swift::driver::Job const*) (in /home/dave/b/q/t/bin/swift-frontend)
==2090498==    by 0x256411A: swift::driver::Compilation::performJobs(std::__1::unique_ptr<swift::sys::TaskQueue, std::__1::default_delete<swift::sys::TaskQueue> >&&) (in /home/dave/b/q/t/bin/swift-frontend)
==2090498==    by 0x2550584: run_driver(llvm::StringRef, llvm::ArrayRef<char const*>) (in /home/dave/b/q/t/bin/swift-frontend)
==2090498==    by 0x254F547: main (in /home/dave/b/q/t/bin/swift-frontend)
==2090498==  Address 0x72ea02f is in the brk data segment 0x72ea000-0x72ea02f
==2090498==
malloc(): invalid next size (unsorted)
Stack dump:
0.      Program arguments: /home/dave/b/q/t/bin/lldb --repl=-disable-objc-interop -sdk / -color-diagnostics -module-cache-path /home/dave/b/q/t/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache -swift-version 4 -locale NOT_A_LOCALE -ignore-module-source-info
^C
$

That being said, if I bypass the driver, it works.


DOMException: Failed to execute 'pushState' on 'History': A history state object with URL 'file://./' cannot be created in a document with origin 'null' and URL 'file:///dist/index.html'.

Hi Guys,
While taking dist, im hitting this error which constantly refreshes the page.
vendor.bundle.js?dc17209…:10 DOMException: Failed to execute 'pushState' on 'History': A history state object with URL 'file://./' cannot be created in a document with origin 'null' and URL 'file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/index.html'.
    at ve.u.url (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:25567)
    at s (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:9:30702)
    at file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:10:448
    at d.$digest (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:10:12923)
    at d.$apply (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:10:14737)
    at file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:9141
    at Object.a [as invoke] (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:22674)
    at a (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:9062)
    at ct (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:9346)
    at lt (file:///Users/TheRedDevil/Projects/ng6/Source/Web/dist/vendor.bundle.js?dc172094010e4fd21817:8:8584)

I have tried adding the same in a web server also.
Starting Chrome in --allow-file-access-from-files mode is not possible as this would be deployed to client server.
Also i cannot find in my code base where this pushState is being used.
Note: Ive tried cloning this repo freshly and the error exist for me there too.

Also i cannot find in my code base where this pushState is being used.

This is part of $location service.

I have tried adding the same in a web server also.

Did you check locally with webpack-dev-server for example? You could start it by running npm start.

as this would be deployed to client server.

In this case you will have some document origin.
Feel free to ask if you still have problems. But I closing this right now.


recreate perfmetrics portrait plot in python

Description

Closes #3550
Link to documentation:

Before you get started

[x] ☝ Create an issue to discuss what you are going to do #3550

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🛠 This pull request has a descriptive title
[x] 🛠 Code is written according to the code quality guidelines
[ ] 🛠 Documentation is available
[ ] 🛠 Tests run successfully
[ ] 🛠 The list of authors is up to date
[ ] 🛠 Any changed dependencies have been added or removed correctly
[ ] 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

[ ] 🧪 Recipe runs successfully
[ ] 🧪 Recipe is well documented
[ ] 🧪 Figure(s) and data look as expected from literature
[ ] 🛠 Provenance information has been added

New or updated data reformatting script

[ ] 🛠 Documentation is available
[ ] 🛠 The dataset has been added to the CMOR check recipe
[ ] 🛠 The dataset has been added to the shared data pools of DKRZ and Jasmin by the @ESMValGroup/OBS-maintainers team
[ ] 🧪 Numbers and units of the data look physically meaningful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

what a noble venture, Lukas! Godspeed :checkered_flag:  :beer:

Hi, we are currently working on the ESMValTool release for v2.11.0. We're wondering if you'd be able to complete this PR by the end of next week (Friday 10th May).
If we don't hear anything from you by then we'll move this into the next milestone.

I just came across this. It looks great. Any progress?

I just came across this. It looks great. Any progress?

We are working on it 😄 It's planned to include this in v2.12

I am not in favor of removing the old recipe as long as we do not reproduce the full output (e.g. including Taylor diagrams, etc.). and therefore have a new and complete "perfmetrics". So long, this PR just includes the portrait diagram. For this reason and to avoid confusion between "perfmetrics" and "portrait diagram", the new recipe (and documentation) should be simply called e.g. "recipe_portrait_plot.yml" and not contain anything like "perfmetrics" in the filenames.
As a side note, I am not sure helper scripts such as "compare_netCDFs.py" need to go into the "diagnostics" folder. Maybe they do not need to go into the repository at all?

I just talked to @bettina-gier and I think I misunderstood the removal of the recipe. So this is fine. My points about the naming remain.

I would propose to add a note to the existing documentation of the perfmetrics (https://github.com/ESMValGroup/ESMValTool/blob/main/doc/sphinx/source/recipes/recipe_perfmetrics.rst) that the portrait diagram alone can also be produced with the new recipe + diagnostic.
In addition, a more detailed description would ideally go into the new "benchmarking" documentation, which is still waiting to be merged: https://github.com/ESMValGroup/ESMValTool/blob/benchmarking_maps4monitoring/doc/sphinx/source/recipes/recipe_benchmarking.rst


fix [dev-8944] and feature [dev-8912] hidden methodology

Testing Steps

For dev-8912: Storybook: Pages/Dashboard/Regression Hidden Filter
For dev-8944: Storybook. Pages/Dashboard/Regression API Filters Chart

Self Review

I have added testing steps for reviewers
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing unit tests are passing

Screenshots (if applicable)

Additional Notes

@adamdoe it would help me tremendously if we could get the Stories in Dashboard that say "Regression ..." to be run off of test before every release. If we could set up the trigger to build storybook off of test whenever there is a merge into test that would make storybook a reliable place to run automated test scripts. All QA would need to do is upon the deployed storybook and check the regressions.
I think our blocker before was some token thing?

@adamdoe all the tests are passing for me, I rebased off dev maybe that fixed it.

The map data isn't loading for me in storys.


Freshness scans shows 0 value in the Soda Cloud UI

Wanted to notify with v3.0.3 in the cloud I just see a straight line with 0 as the value every time, even though the output of the scans shows the actual time difference.
Scans and screenshots of the result attached. It would be great if it gets resolved quickly, when any failure happens it would be easier to understand from the UI (Even with ms) and not go into the logs to understand how far behind our datasets are.
checks for reservation_browser_debug_v:
  - freshness(res_date, CUST_VAR):
      warn: when > 70m
      fail: when > 150m

SODA-1023

Fixed in 3.0.7


Infer User Capabilities from REST API Responses

Feature Description

In the editor and dashboard, we currently pass the current user's capabilities that we need in the app via inline JS / the ConfigProvider. This works OK-ish, but does not really scale well.

Just checking for can user edit stories does not work for cases like the dashboard where we need to individually check for capabilities for the stories being displayed.
The user could be allowed to edit story X but not Y, so the checks would not be accurate.
Every enhancement to the React apps requires PHP changes to pass the necessary capability information.

Lucky for us, the WordPress REST API incorporates hyperlinking throughout the API to allow discoverability and browsability, loosely based on the HAL standard.
For example, the _links property of the response object contains a map of links to other API resources, grouped by “relation”. The relation specifies how the linked resource relates to the primary resource.
This includes capabilities! For each story, there is a wp:action-assign-author entry under _links if the user has the capability. So instead of passing hasAssignAuthorAction manually from PHP we can simply look for this entry.
Note: the WP Rest API does not currently add links for all actions that we need (see this example, but extending the REST API to do that is trivial.
So my suggestion:

Add new actions to _links for things we need
When fetching stories from the REST API, check for the _links to get the user's capability for each story
Use that information to have more granular capability checks

Alternatives Considered

targetHints are an interesting idea as well, but not yet available.

Additional Context

Gutenberg has implemented a really neat approach for this.
Essentially, one calls a Redux selector to see whether the current user can perform the given action on the given REST resource.
If the information hasn't been fetched yet, an OPTIONS request to the REST API is triggered via the canUser() resolver (basically a middleware that only runs once).

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

@spacedmonkey would love your input on this / the suggested approach

Verified in QA


Crash bug in virtiofs for files over 4M triggered by Backblaze personal backup

Issue description
In v0.1.215-1, v0.1.215-2, v0.1.217-1, and v0.1.217-2 Backblaze client causes the Virtio-FS Service to crash on files > 4190000 bytes (-ish). With v0.1.204, the service survives, but the Backblaze client reported an error of 'TEMPORARY_OTHER' for the file and cannot be backed up.
To Reproduce
Attempting to use the Backblaze personal backup client to back up the attached G0011522.JPG from a virtio-fs directory will reliably trigger the failure. The Virtio-FS service crashes, and the Z: drive disappears. The backblaze client treats the situation as if Z: was a USB drive that disconnected, and carries on ignoring the rest of the Z: drive.
Using explorer to copy the file into the VM succeeds. Using Backblaze to back up the file from the copy inside the VM also succeeds.
Expected behavior
The service should remain up and going, and the backup client should be able to read the file and back it up.
What appears to happen instead is the client successfully reads the file in 1MB chunks (presumably to scan for blocks that need to be backed up), then it comes around for a second pass and load the entire file all at once -- but instead of reading 4190259 bytes, the number of bytes returned on the read is 0, which appears to trigger a fault.
Screenshots
The attached files:
G0011522.JPG is the file that will cause the failure
dir.txt is a listing of the directory of the virtio filesystem. BB backs files up from smallest to largest -- G0011525.JPG succeeds, at 4181721 bytes. G0011522.JPG fails at 4190259 bytes.
VFS.log is the logging from virtiofs (DebugFlag set to 0xFFFFFFFF)
Dante.xml.txt is the qemu configuration file.
Host:

Distro: Ubuntu 22.04
Kernel version: 5.15.0-33-generic
QEMU version: 6.2.0    (6.2+dfsg-2ubuntu6)
QEMU command line -- standard as launched from virt-manager -- see bottom
libvirt version: 8.0  (8.0.0-1ubuntu7)
libvirt XML file: attached -- Dante.xml.txt

VM:

Windows version: 10 - 21H2
Which driver has a problem: Virtio-FS
Driver version or commit hash that was used to build the driver: 1.215-1, 1.215-2, 1.217-1, 1.217-2   (1-204 does not crash but also does not work correctly -- see above)

Additional context
Dante.xml.txt
dir.txt
VFS.log

/usr/bin/qemu-system-x86_64 -name guest=Dante,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-6-Dante/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/Dante_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-4.2,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram -accel kvm -cpu Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,sha-ni=on,umip=on,waitpkg=on,gfni=on,vaes=on,vpclmulqdq=on,rdpid=on,movdiri=on,movdir64b=on,fsrm=on,md-clear=on,avx-vnni=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff -m 8192 -object {"qom-type":"memory-backend-file","id":"pc.ram","mem-path":"/var/lib/libvirt/qemu/ram/libvirt/qemu/6-Dante/pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":8589934592} -overcommit mem-lock=off -smp 4,sockets=1,dies=1,cores=4,threads=1 -uuid 095d4640-0dec-45bc-bfca-a527554ad2ff -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=34,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot menu=off,strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -blockdev {"driver":"host_device","filename":"/dev/ezekiel-00/dante","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-3-format","read-only":false,"driver":"raw","file":"libvirt-3-storage"} -device virtio-blk-pci,bus=pci.6,addr=0x0,drive=libvirt-3-format,id=virtio-disk0,bootindex=2 -blockdev {"driver":"file","filename":"/mnt/iso/template/iso/Windows21H2.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"} -device ide-cd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=1 -blockdev {"driver":"file","filename":"/mnt/iso/template/iso/virtio-win-0.1.217-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"} -device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=3 -chardev socket,id=chr-vu-fs0,path=/var/lib/libvirt/qemu/domain-6-Dante/fs0-fs.sock -device vhost-user-fs-pci,ats=on,id=fs0,chardev=chr-vu-fs0,queue-size=1024,tag=Backups,bus=pci.5,addr=0x0 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a2:e5:95,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"spice"} -spice port=5901,addr=<IP_ADDRESS>,disable-ticketing=on,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Also.... tried multiple things inside the XML configuration with no effect:

with and without queue=1024
with and without ats=on
with and without xattr=on
with each of the caching mode options
with lock posix=on/flock=on and posix=off/flock=off

Tried with virtio-win-prewhql-0.1-217 on Win10-64(21h2) guest, can upload the photo(you provided) to backblaze from Z:(virtiofs shared volume).Also I tried to uploda/download a 488M size file,it works fine.

host info:
qemu-kvm-4.2.0-60.module+el8.5.0+14545
4.18.0-348.23.1.el8_5.x86_64

Tried with virtio-win-prewhql-0.1-217 on Win10-64(21h2) guest, can upload the photo(you provided) to backblaze from Z:(virtiofs shared volume).Also I tried to uploda/download a 488M size file,it works fine.
host info: qemu-kvm-4.2.0-60.module+el8.5.0+14545 4.18.0-348.23.1.el8_5.x86_64

Interesting that RH8 didnt have an issue, but Ubuntu 22.04 did. The difference may be as simple as the host kernel.
That said... one of the updates somewhere in the past couple months seems to have addressed the issue. I was able to successfully back up the underlying Linux filesystems without a problem. Some of the files were well over 10GB+.
Host:
Distro: Ubuntu 22.04
Kernel version: 5.15.0-47-generic    -- updated kernel version
QEMU version: 6.2.0 (6.2+dfsg-2ubuntu6.3)   -- updated
libvirt version: 8.0 (8.0.0-1ubuntu7.1)   -- updated

VM:
Windows version: 10 - 21H2
Which driver has a problem: Virtio-FS
Driver version: 0.1-221

Thanks for listening. I'm not sure which update actually fixed the issue, but I'm not sure it matters any more, either.

Hi,
What version of virtiofsd is used on Ubuntu?

/usr/lib/qemu/virtiofsd  --version
virtiofsd version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
using FUSE kernel interface version 7.33

Also I would like to add my host info.
kernel: kernel-4.18.0-348.23.1.el8_5.x86_64
qemu-kvm: qemu-kvm-4.2.0-60.module+el8.5.0+14545+9e40c7b1.2.x86_64
virtiofsd:
rpm -qf /usr/libexec/virtiofsd
qemu-kvm-core-4.2.0-60.module+el8.5.0+14545+9e40c7b1.2.x86_64

I was having a similar issue with Backblaze on a virtiofs share.  The Z: drive would disappear shortly after starting the Backblaze indexing and I would have to reboot the Windows guest to be able to access it again.  I assumed it was a problem with the virtio-win drivers and I tried all the same things that @jmbezekiel did to no avail.
Then I closely monitored the debug log for virtiofsd on the host and I noticed the following:

[2022-11-20 04:22:35.654885+0000] [ID: 00000004] unique: 160215, opcode: READDIRPLUS (44), nodeid: 947806, insize: 80, pid: 5100
[2022-11-20 04:22:35.654889+0000] [ID: 00000004]    unique: 160215, error: -24 (Too many open files), outsize: 16

Apparently it was trying to open too many files.  I then tried running virtiofsd with --rlimit-nofile=1000000 and that did not help either.  Then I tried --inode-file-handles=mandatory and my version of virtiofsd did not support this:

nox:/home/christophocles # /usr/libexec/virtiofsd --version
virtiofsd version 7.1.0 (openSUSE Tumbleweed)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
using FUSE kernel interface version 7.36

So then I compiled the Rust version of virtiofsd from this link https://gitlab.com/virtio-fs/virtiofsd and ran it with --inode-file-handles=mandatory and this finally solved the problem!!

I was having a similar issue with Backblaze on a virtiofs share. The Z: drive would disappear shortly after starting the Backblaze indexing and I would have to reboot the Windows guest to be able to access it again. I assumed it was a problem with the virtio-win drivers and I tried all the same things that @jmbezekiel did to no avail.
Then I closely monitored the debug log for virtiofsd on the host and I noticed the following:

[2022-11-20 04:22:35.654885+0000] [ID: 00000004] unique: 160215, opcode: READDIRPLUS (44), nodeid: 947806, insize: 80, pid: 5100
[2022-11-20 04:22:35.654889+0000] [ID: 00000004]    unique: 160215, error: -24 (Too many open files), outsize: 16

Apparently it was trying to open too many files. I then tried running virtiofsd with --rlimit-nofile=1000000 and that did not help either. Then I tried --inode-file-handles=mandatory and my version of virtiofsd did not support this:

nox:/home/christophocles # /usr/libexec/virtiofsd --version
virtiofsd version 7.1.0 (openSUSE Tumbleweed)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
using FUSE kernel interface version 7.36

So then I compiled the Rust version of virtiofsd from this link https://gitlab.com/virtio-fs/virtiofsd and ran it with --inode-file-handles=mandatory and this finally solved the problem!!

Hi @christophocles,
Thank you for sharing.
Rust version is the one that gets the latest fixes and should be used going forward.
Best regards,
Yan.

Rust virtiofsd is a default version. Closing.


Keystore should also expose encryption/decryption

Right now keystore only has message signing and signature verification but doesn't have anything related to encryption/decryption.

@bowenwang1996 is there a customer use case for this? It would be helpful in order to assign priority properly.

@janedegtiareva I think we just asked people to use tweenacl directly, so not very urgent.

NEAR client libraries are for working with NEAR. Developers needing encryption/decryption should use a different library designed for that.
We shouldn't use NEAR account keys for things other than signing transactions, see nearprotocol/nearcore#2833.


feat : add ability to log to a file

Description
This PR fixes #841
This PR adds the ability to create a log file.
This is an example of how to do it:

Should we add the ability to add more configurations, like timestamps, levels, etc?

Hey @carlosm27 👋
How would you use this new interface?

Hey @sansyrox. I would use it to create log files with timestamps, log level and with and add rotation policy.
What features are more appropiate for this interface?

API is very practical. Does it support asynchronous operation?


Unable to setup ghost on ubuntu desktop 16.04 x64

I tried installing ghost on my pc. I followed the steps here>. I got the following error. Please help.

Hi @kskarthik, sorry to hear you're having trouble. As mentioned in the readme & contrib guide, if you have questions about Ghost or need help, please head on over to slack - https://ghost.org/slack/ - and post in the #help channel. There you can get help & support from the Ghost community, which we're not able to do here in the GitHub bug tracker.

I am unable to login to slack :(