Wrong sorting of ROSE chrom_sizes and bed

Description of the bug
In the ROSE workflow the bed file is sorted using SORT_BED (which uses gnu_sort).
This results in the chromosomes of the bed file being sorted in this order:
chr1
chr10
chr11
chr12
chr13
chr14
chr15
chr16
chr17
chr18
chr19
chr2
chr3
chr4
chr5
chr6
chr7
chr8
chr9
chrM
chrX
chrY

The chrom_sizes gtf (we use the genome index for that), however, looks like that:
chr1    195471971       8
chr2    182113224       198729854
chr3    160039680       383878307
chr4    156508116       546585323
chr5    151834684       705701916
chr6    149736546       860067187
chr7    145441459<PHONE_NUMBER>
chr8    129401213<PHONE_NUMBER>
chr9    124595110<PHONE_NUMBER>
chr10   130694993<PHONE_NUMBER>
chr11   122082543<PHONE_NUMBER>
chr12   120129022<PHONE_NUMBER>
chr13   120421639<PHONE_NUMBER>
chr14   124902244<PHONE_NUMBER>
chr15   104043685<PHONE_NUMBER>
chr16   98207768<PHONE_NUMBER>
chr17   94987271<PHONE_NUMBER>
chr18   90702639<PHONE_NUMBER>
chr19   61431566<PHONE_NUMBER>
chrX    171031299<PHONE_NUMBER>
chrY    91744698<PHONE_NUMBER>
chrM    16299<PHONE_NUMBER>

This leads to an error thrown in the process INVERT_TSS, since sorted files are expected here.
We didn't catch this before since we are only testing on chr1.
One option would be to add the -V flag to the SORT_BED process, for sorting the bed file.
This would bring the normal chromosomes in the right order, but we would still get an error with chromosome X, Y and M, since they are not in the natural sorting order in the chrom_sizes file.
Therefore, I would propose sorting the chrom_sizes file with GNU_SORT as well to make sure we always get the right order.
Command used and terminal output
No response
Relevant files
No response
System information
No response

Sounds reasonable


25 epochs yields sequence of  at generation

Hello! Thanks for the repo.
I want to train a good enough LM on the Hebrew language.
I will later want to use the model in order to get the probability of a given word in a specific context.
I've trained your second model - 'words.py' first with 5 epochs. It seems ok, then I've taken a larger dataset (650MB of Hebrew Wikipedia) and it took around 2 days on Azure VM.
Sadly, the output now at the generation is
....
I'm trying to understand what went wrong.
The output model is ~65MB.
i2w & w21 are at length 10K (because of the limit).
The real vocab size is more towards 1.2 million words.
I suspect that this big gap is what causes bad performance.
What is good practice in order to train good LM?
Epochs? Dataset size? Words limit? Which model do you recommend out of the three suggested?
Thanks a lot :)

Hi Yonatan,
Looks like Hebrew is a language with a pretty high morphological structure (lot of grammatical structure inside the words). That explains the high vocabulary. It also means that if you cut down the vocabulary, you will get a  lot of pad tokens, which is probably why the models mode collapses to just predicting the pad token.
I don't expect you'll get very far with a word-level model. You need to model the structure inside the words. The character level model may help, but it's very slow. I recommend using a character-level transformer instead. It should work just as well, and train much faster. See my blog post on the subject for an explanation and some code samples (in pytorch rather than keras).

Thank you. That sounds true. Thanks for the blog post - I will read it.
I did consider char-level LMs for the reasons you said, and i'm convinced that it will work better.
My problem with it is the computing the probability of a word given a context.
For example P(red | This car is).
In a word level model, it's easy:
You feed "The car is" into the model, it predicts scores for each word in your vocab (let's say 10K dimension vector - one for each word), you normalize those scores to a probability distribution, and then you use the word-to-index in order to extract the probability for the word you want.
I'm not sure how it's supposed to work in a character-level model, because it samples a character each time instead of word. Then I believe that multiplying the probabilities of the sampled characters may be problematic.

You can multiply the probabilities of the characters, so long as you condition on everything before the character:
p(r, e, d|'this car is ') = p(r | 'this car is ') p(e|'this car is r')p(d|'this car is re')

This follows from the chain rule of probability. If work the same in the LSTM and the transformer model (they both model conditional probability in the same way).

Sounds good, it worth a try. Thanks :)


Fix test for null flutter root

Description
With the usageValues added we can fail to find a flutter root and crash, though this is only in tests.

Landing on red to fix build


chore: fix demo app compilation and make sure to build it on CI to not break it again

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (1e90976) 92.00% compared to head (0b63e97) 92.00%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #72   +/-   ##
=======================================
  Coverage   92.00%   92.00%
=======================================
  Files           1        1
  Lines          25       25
  Branches       11       11
=======================================
  Hits           23       23
  Misses          1        1
  Partials        1        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


older versions of the site shall be captured and published similar to how we did with aegis.ist

Done!


[BUG] ha-floorplan breaks the automatic dashboard/view

You can also check the discussion on the HA forum.
To reproduce this you need the default/automatic view and a floorplan view.
When you use the floorplan view before opening the default/automatic view in the same client page session, the following error appears:

It seems that this is because the floorplan view seems to add an entry to the hass.entities that has "*" as the key instead of having the entity_id as the key.
When the floorplan was not visited, this '' key does not exist in 'entities' and the issue does not occur.  So the floorplan is adding the '' entry somewhere while it should add entities["sensor.16996282-911ec145"]   (entities seems to be from hass.entities).

What version of HA floorplan are you using?

I am on version 1.0.35, which is the latest AFAICS.

BTW, I found this in the code (debugger):
                    this.hass.states["*"] = {
                        entity_id: `sensor.${t}`,
                        state: "on",
                        last_changed: (new Date).toString(),
                        last_updated: (new Date).toString(),
                        attributes: {
                            device_class: "ha-floorplan",
                            friendly_name: "ha-floorplan - Floorplan for Home Assistant",
                            icon: "mdi:floor-plan",
                            assumed_state: !1,
                            hidden: !0
                        },
                        context: {}
                    },

Changing to
                    this.hass.states[`sensor.${t}`] = {

possibly fixes the issue, but there may be a reason for the use of index "*" such as limiting the number of "virtual sensors" that seem to be defined this way to only 1.
Further there are other locations in the code that seem to rely on this (last line) that would require another means for identifying this special case:
            getChangedEntities(t) {
                const e = new Set
                  , i = Object.keys(this.hass.states);
                for (const o of i)
                    if ("*" !== o || e.has(o)) {

We had an issue in the past with the * entity so I thought this was fixed. Let me flag it with the devs @petarkozul @exetico

We are already aware of the issues related to this bug. Thank you for flagging it, in a issus 👍. Let's keep this open, until it's fixed.
Petar originally did the logics of that part of the code. So I'll need to find a bit of time, to see what it's related to, as of today, if there's any relations.
As you pointed out, there's a few areas of the code, which possibly should be rebaked a bit 😊.

Kindly check with: https://github.com/ExperienceLovelace/ha-floorplan/blob/dev/dist/floorplan.js
It's currently in the dev-branch.

@exetico I've updated my floorplan.js and floorplan.js.gz with that version and I do not observe the issue anymore.


ISSUE-24: fix normalization of weight values in Mod156 export.

Fixes #24
Remove 'hack' for vertices with more than 4 bone influences (this would probably needs more more work and might break some meshes, but will be done in another issue
Add test that the sum of all weights is always 255

Coverage decreased (-0.04%) to 93.176% when pulling 0bfca596bfc61677c878bd2c52e43e811f20fd57 on bugfix/zerodivisionerror-in-bone-weights-exporting into 93ef3212991c8dc548472c64422ccb956a1d2b36 on develop.


rubico/x/always

always(value any) -> ()=>value

Create a function that always returns a value.
const getHey = always('hey')

console.log(getHey()) // hey

Done with https://github.com/a-synchronous/rubico/commit/a1f5e9dfae8e608af6c29f5b7b8ba6883f849f8a


CIV-9256 update endpoint

https://tools.hmcts.net/jira/browse/CIV-9256
update endpoint
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x ] No

Merged as a part of CIV-9253-test changes


Error Boundary Audit

fix error boundaries:
make it not look like 💩..  if a page error boundary gets triggered have a nice page wide error display.  if it's a leaf node, have a reasonable default.
never let navigation disappear
define sli level boundaries so we can answer questions like "how many times did a user not see a dashboard cell because of an error"

@joelfried was this completed? if so, could you link to a pr an close?

This was completed in https://github.com/influxdata/ui/pull/364.


strconv: optimize ParseFloat with new refloat algorithm

Floating point conversions are used in many places such as encoding/json. While the current Eisel-Lemire is efficient, you can actually use smaller lookup table and reduce execution time at the same time.
I implemented this algorithm in Go from start and now it's available through  BSD-3-Clause license: https://github.com/sugawarayuuta/refloat

Benchmarks

$ go test -bench "\d$" -benchmem -benchtime 1m
goos: linux
goarch: amd64
pkg: github.com/sugawarayuuta/refloat
cpu: Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz
BenchmarkParseFloat64/strconv/bits-4            648877453              112.3 ns/op             0 B/op          0 allocs/op
BenchmarkParseFloat64/strconv/norm-4            729803588               95.37 ns/op            0 B/op          0 allocs/op
BenchmarkParseFloat64/refloat/bits-4            773428314               93.41 ns/op            0 B/op          0 allocs/op
BenchmarkParseFloat64/refloat/norm-4<PHONE_NUMBER>              67.75 ns/op            0 B/op          0 allocs/op
BenchmarkParseFloat32/strconv/bits-4            894441672               82.30 ns/op            0 B/op          0 allocs/op
BenchmarkParseFloat32/strconv/norm-4<PHONE_NUMBER>              67.97 ns/op            0 B/op          0 allocs/op
BenchmarkParseFloat32/refloat/bits-4            870175646               83.18 ns/op            3 B/op          0 allocs/op
BenchmarkParseFloat32/refloat/norm-4<PHONE_NUMBER>              50.24 ns/op            0 B/op          0 allocs/op
PASS
ok      github.com/sugawarayuuta/refloat        612.972s

Observation

We can see that it performs mostly better on normally distributed (norm) floats and bit-wise uniform floats (bits).
Benchmarks are done on multiple machines - they did produce results very similarly.
Especially normally distributed float64 has a significant change (~41%) which will fit in many real use-cases.

Tests

passes parse-number-fxx-test-data, the standard libraries' tests and I'm doing active fuzz testing in addition comparing outputs to the current strconv. so far the only detected problem seems to be the issue 42436 which is a problem with strconv only.

Notes

I actually used an external tool to generate polynomial approximations (Which is not distributed through the same license as the project itself). Do tell me if it can be a problem...

Please add the output of golang.org/x/perf/cmd/benchstat comparing the benchmark output, run with -test.count=10, for both the current package and the package with your change.  Thanks.

Hi, thank you for commenting. I had to rerun the benchmark since I didn't specify -count 10 in the benchmark above, if you're OK with that. The left side is the standard library and the right side is my library.
                    │ ./strconv.txt │            ./refloat.txt            │
                    │    sec/op     │   sec/op     vs base                │
ParseFloat64/bits-4    104.85n ± 1%   92.11n ± 1%  -12.16% (p=0.000 n=10)
ParseFloat64/norm-4     95.04n ± 5%   68.89n ± 3%  -27.51% (p=0.000 n=10)
ParseFloat32/bits-4     80.86n ± 2%   84.44n ± 1%   +4.41% (p=0.000 n=10)
ParseFloat32/norm-4     65.97n ± 2%   48.88n ± 1%  -25.91% (p=0.000 n=10)
geomean                 85.39n        71.53n       -16.22%

                    │ ./strconv.txt │         ./refloat.txt          │
                    │     B/op      │    B/op     vs base            │
ParseFloat64/bits-4    0.000 ± 0%     0.000 ± 0%  ~ (p=1.000 n=10) ¹
ParseFloat64/norm-4    0.000 ± 0%     0.000 ± 0%  ~ (p=1.000 n=10) ¹
ParseFloat32/bits-4    0.000 ± 0%     3.000 ± 0%  ? (p=0.000 n=10)
ParseFloat32/norm-4    0.000 ± 0%     0.000 ± 0%  ~ (p=1.000 n=10) ¹
geomean                           ²               ?                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │ ./strconv.txt │            ./refloat.txt            │
                    │   allocs/op   │ allocs/op   vs base                 │
ParseFloat64/bits-4    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFloat64/norm-4    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFloat32/bits-4    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFloat32/norm-4    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                           ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

@sugawarayuuta this is great! Thanks for sharing. It would be great if you send a change to optimize the strconv package. See https://go.dev/doc/contribute for how to contribute to Go, if you haven't seen it. Thanks!

CC @griesemer

Hi, kind of unfortunate. I tried to apply my parser to the strconv, and indeed it became faster. However after some hours of experimenting, It seemed that it's not the transformation (main algorithm) that's faster, but the reading part...
I'm sorry I didn't notice this issue, but since I confirmed only changing readFloat can speed up all of benchmarks in atof_test.go by considerable amount, can I send a change which covers the reading part? Should I edit this issue or create a new one? Thanks...

@sugawarayuuta nice work and thank you, please go ahead with firstly sending the change for readFloat, we appreciate it and look forward to interacting with you on your change! Later on, with enough time we can figure out how to completely transform your work for the whole of ParseFloat.


needextend throws exception if id is modified

needextend shall not allow modification of id of the need.
.. needextend:: id == "MY_NEED_ID"
   :id: MY_NEW_NEED_ID

At the moment, it does not allow it, but throws an exception.

Extension error (sphinxcontrib.needs.roles.need_incoming):
Handler <function process_need_incoming at 0x7f4172c051f0> for event 'doctree-resolved' threw an exception (exception: 'BuildEnvironment' object has no attribute 'warn_node')

It would be nice if the id modification is ignored (and an error is thrown).

The change does not register internal options (currently "id" only") for needextend in Sphinx.


Misc estimator steps for scz bp

Added a bunch of estimator steps - some (probably) mainly for scz bp, but others may be more generally useful if you fancy e.g. augmentation or data-centric processing.
Let me know if you could see yourself using any of them, then I'll add them in a separate PR for common (and review).
Added:

Simple imputer
Iterative imputer with random forest (to mimic missforest)
Standard z-score normalizer that infers numeric columns
Augmentation with synthetic data using synthcity
Data-centric processing with cleanlab

@MartinBernstorff, @sarakolding, @bokajgd, @erikperfalk
The reason for the additional pipeline constructor (using imblearn's Pipeline), is that sklearn does not allow one to modify the number of rows in X and Y in a pipeline, but imblearn to the rescue. Their Pipeline is just a subclass of sklearn's which adds functionality for resamplers (such as the one I added for synthetic data) and is a drop-in replacement.

Looks interesting! Would love to hear whether they add any performance 👍


[FLINK-21203]Don’t collect -U&+U Row When they are equals In the LastRowFunction

What is the purpose of the change
In the  LastRowFunction , the -U&+U Row will be collected even if they are the same, which will  increase calculation pressure of the next Operator.
To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also, a config to enable the optimization will be added.
Brief change log

add equaliser and isStateTtlEnabled in ProcTimeDeduplicateKeepLastRowFunction
optimize the logic of DeduplicateFunctionHelper#processLastRowOnProcTime and  DeduplicateFunctionHelper#processLastRowOnChangelog  by equaliser and isStateTtlEnabled
add unit test in ProcTimeDeduplicateKeepLastRowFunctionTest

Verifying this change
This change added tests and can be verified as follows:

Added test that validates that Duplicate row is Deduplicated when state ttl disabled In  ProcTimeDeduplicateKeepLastRowFunctionTest#testDeduplicateWithGenerateUpdateBefore

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): ( no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (/no)
If yes, how is the feature documented? (not documented)

cc @leonardBang , could you help to review this?

cc @leonardBang , could you help to review this?

ok， I'll take a look

@wangpeibin713 Could you rebase your code to master rather than use  git merge command?  I found you used git merge which lead to current branch has two parents.

@wangpeibin713 Could you rebase your code to master rather than use  git merge command? I found you used git merge which lead to current branch has two parents.
@leonardBang I sorry for that. I have rebase the code. Could you help to review this pr now, 3Q

Hi @leonardBang could you help to review this pr.  3Q

Hi @leonardBang could you help to review this pr. 3Q

Sorry for the late response because Spring Festival vocation, I'll review this soon

LGTM @wangpeibin713 could you fix the build failure and rebase to latest master branch?

@leonardBang  Thanks, I have fixed the build failure by mvn spotless:apply and rebase to latest master branch


Datadog

rebuilt datadog content
site builds
no broken links

Made the change to the heading.
Be advised I was unable to get Datadog running on my linux VM so someone should test this to make sure it works on Mac/Linux native


Add make target for local installation of osdctl

make install builds a binary and copies it to /bin
Comparable behavior to go install

jira: OSD-16141

/approve


Changing Twitter logo in the Website

As seen in the image below, website has same old twitter logo, I want to change it to new one.
Please assign me.

@sumn2u  I couldn't figure the how css is being attach to the website. Can you help me out?

i would like to work on this issue please assign to me!

@sumn2u  It seems that we are using lower version of fontawesome. Is there any way to upgrade it?

It seems that the issue has been fixed, looking at the website.

Yes @Arnalv. Thanks for having a look into it. 🙏


[Releases] Latest executable file for windows doesn't  work well

Which build was it, is it on #36?

yes

Huh thats weird, I've just created a VM and it worked. Can you share the url you're trying to download?

i'm using my tower to use it and i compiled it and i have this error now

Yeah that part I understood, but I can't reproduce it, that why I'm asking you for the series URL, so I can try to reproduce it.

oh yes https://www.crunchyroll.com/es-es/the-worlds-finest-assassin-gets-reincarnated-in-another-world-as-an-aristocrat

i was testing without headless and i'm having the same problem so might be something that we don't know

Can you try and start it with the argument --headless false and check if chromium gets stuck somewhere?

i've compiled in code is the same but i ended with the same result

I meant --headless false as an argument, or edit BrowserFactory.cs on line 18 and change headless to false

I meant --headless false as an argument, or edit BrowserFactory.cs on line 18 and change headless to false

I've dome that yesterday and t'he result was the same

And where does chromium gets stuck?

It login but when is trying to download It vives me a timeout

It logs in normally? Closes the tab and all?

It logs in normally? Closes the tab and all?

yes and then i have the timeout

Ahmm it shouldn't have closed any tabs, since by your screenshot it is waiting for chromium to navigate somewhere.
After it types in your credentials and press the login button, where does chromium stays?

if i do not close it i'm having this error

i'm having this without closing anything

Huh that weird, the JSON containing the media ID is malformed:

I've changed it to use regex instead of JSON parsing, can you try the latest build and print the results again?

Huh that weird, the JSON containing the media ID is malformed:

I've changed it to use regex instead of JSON parsing, can you try the latest build and print the results again?

as latest build i'm having this issue

I've just released another build which skips the API authentication if not necessary (which is your case), can you try again with build #40?

Oops, I had mixed up the rules, can you try again on #41?

Now they say that my credentials where not provided and were provided

Fixed that too, can you try on #42?

Finally working


Crash with SyntaxError: Unexpected token C in JSON at position 1644

I saw a case very similar to https://github.com/microsoft/vscode-languageserver-node/issues/436
SyntaxError: Unexpected token C in JSON at position 1644
    at JSON.parse (<anonymous>)
    at StreamMessageReader.onData (node_modules/vscode-jsonrpc/lib/messageReader.js:182:29)
    at Socket.listen.readable.on (node_modules/vscode-jsonrpc/lib/messageReader.js:148:18)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

It looks like with certain characters the Emacs lsp-mode LSP client sends a bogus request. I've entered an issue with them to improve the situation https://github.com/emacs-lsp/lsp-mode/issues/1246
That issue contains repro steps and shows how to repro the example using a fork of the lsp-sample language server integrated into Emacs.
Could this repo be updated to more gracefully handle what looks to be a bogus request? That would help so that language servers don't get killed because of an issue in an LSP client.
Please let me know other info/debug output I can provide to help.

I see. That makes sense on the response side. Here I'm pretty sure this is the result of a request coming in from Emacs. Is there any info I could gather to motivate adding something in that case?

You can add an onError handler on the connection to recognize this.

Thanks @dbaeumer  I did some poking around using that. In messageReader.js right before the error the buffer is:
this.buffer.buffer.toString()
"{"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"file:///home/ry/lsptest/foo.txt","version":7},"contentChanges":[{"text":"%�"}]}}
Content-Length: 140

{"jsonrpc":"2.0","method":"textDocument/didSave","params":{"textDocument":{"uri":"file:///home/ry/lsptest/foo.txt","version":7},"text":null}}
...stuff...

Then after var msg = this.buffer.tryReadContent(this.nextMessageLength); we have:
this.buffer.buffer.toString()
"ontent-Length: 140

{"jsonrpc":"2.0","method":"textDocument/didSave","params":{"textDocument":{"uri":"file:///home/ry/lsptest/foo.txt","version":7},"text":null}}

and msg is:
msg
"{"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"file:///home/ry/lsptest/foo.txt","version":7},"contentChanges":[{"text":"%�"}]}}
C"

That trailing "C" that breaks the JSON parse.
Presumably there's something broken with that initial request?

The response has a wrong Content-Length set. This is why you end up with the additional "C". It basically belongs to the header of the next message.


GDPR compliance

Another question: what do I need to do to ensure that the site is GDPR compliant? Is that an issue, or only if I collect data from visitors to the site? And do I need to include a boilerplate privacy notice?

Here's a word doc containing the privacy statement. I was thinking that it could be linked to from the contact form page?
Privacy Statement.docx


setPermissions not available?

In the source code setPermissions.js is not exported to hooks? And the content of setPermissions.js seems just an old version of checkPermissions.

Same problem here. I think the filters branch is a little more up to date. I hope they release this soon as I am starting a new project and hoe to set this up as well.

You can always require('feathers-permissions/hooks/setPermission')

the debug statement in that file still has check-permissions btw

when are you gonna release it?

Any updates?

+1


Does tensorflow  support Xeon E5645 CPU ?

Hi,
Does tensorflow version r1.9 support Intel Xeon E5645 CPU?
Thanks !
Alan

https://ark.intel.com/products/48768/Intel-Xeon-Processor-E5645-12M-Cache-2_40-GHz-5_86-GTs-Intel-QPI
Your CPU does not have AVX instruction set.
If you install from certain community maintained packages, or build from sources, yes.
If you install from official binaries, no
Closing as a duplicate of https://github.com/tensorflow/tensorflow/issues/19584


Update Rust toolchain to edition 2021

Motivation
The most useful update edition 2021 will bring us is the Disjoint capture in closures. In edition 2018, when we use a.x in a closure, it will move the entire a into the closure. But in edition 2021, the closure will only move the field of struct x.
The second useful one is implementing IntoIterator for arrays, which I think I don't bother the explain.

@andylokandy can I take up the issue and work on this?

@burnerlee Just to mention, maybe you need to upgrade the rust-toolchain version before switching to the 2021 edition.

@andylokandy sure, I'll get started with this. Can you please assign me this issue? Also, I am new to this project. What would be a good way to get started with the project and the issue?

@burnerlee Have you started this upgrade? If not I'd like to try it.


Failed to find tool: lib.exe while targeting x86_64-pc-windows-msvc from WSL2 Ubuntu 20.04

Bevy version
0.5.0
Operating system & version
Ubuntu 20.04.2 LTS on Windows 10 x86_64 (WSL2)
What you did
Followed instructions in bevy's linux dependencies. Then tried to compile this example
What you expected to happen
To compile without problems.
What actually happened
Build failed because there's no lib.exe
Additional information
error: failed to run custom build command for `spirv-reflect v0.2.3`

Caused by:
  process didn't exit successfully: `/root/dev/rust/bevy-shaders-test/target/debug/build/spirv-reflect-95b92caf095f94c7/build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("x86_64-pc-windows-msvc")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-pc-windows-msvc = None
  CC_x86_64_pc_windows_msvc = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-pc-windows-msvc = None
  CFLAGS_x86_64_pc_windows_msvc = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "src" "-Wall" "-Wextra" "-Fo/root/dev/rust/bevy-shaders-test/target/x86_64-pc-windows-msvc/debug/build/spirv-reflect-14456f7f069d4955/out/vendor/spirv_reflect.o" "-c" "vendor/spirv_reflect.c"
  cargo:warning=vendor/spirv_reflect.c: In function ‘spvReflectChangeDescriptorBindingNumbers’:
  cargo:warning=vendor/spirv_reflect.c:4012:28: warning: comparison of integer expressions of different signedness: ‘uint32_t’ {aka ‘unsigned int’} and ‘int’ [-Wsign-compare]
  cargo:warning= 4012 |     if (new_binding_number != SPV_REFLECT_BINDING_NUMBER_DONT_CHANGE) {
  cargo:warning=      |                            ^~
  cargo:warning=vendor/spirv_reflect.c:4018:25: warning: comparison of integer expressions of different signedness: ‘uint32_t’ {aka ‘unsigned int’} and ‘int’ [-Wsign-compare]
  cargo:warning= 4018 |     if (new_set_binding != SPV_REFLECT_SET_NUMBER_DONT_CHANGE) {
  cargo:warning=      |                         ^~
  cargo:warning=vendor/spirv_reflect.c:4026:23: warning: comparison of integer expressions of different signedness: ‘uint32_t’ {aka ‘unsigned int’} and ‘int’ [-Wsign-compare]
  cargo:warning= 4026 |   if (new_set_binding != SPV_REFLECT_SET_NUMBER_DONT_CHANGE) {
  cargo:warning=      |                       ^~
  cargo:warning=vendor/spirv_reflect.c: In function ‘spvReflectChangeDescriptorSetNumber’:
  cargo:warning=vendor/spirv_reflect.c:4060:49: warning: comparison of integer expressions of different signedness: ‘uint32_t’ {aka ‘unsigned int’} and ‘int’ [-Wsign-compare]
  cargo:warning= 4060 |   if (IsNotNull(p_target_set) && new_set_number != SPV_REFLECT_SET_NUMBER_DONT_CHANGE) {
  cargo:warning=      |                                                 ^~
  exit code: 0
  AR_x86_64-pc-windows-msvc = None
  AR_x86_64_pc_windows_msvc = None
  TARGET_AR = None
  AR = None
  running: "lib.exe" "-out:/root/dev/rust/bevy-shaders-test/target/x86_64-pc-windows-msvc/debug/build/spirv-reflect-14456f7f069d4955/out/libspirv_reflect_cpp.a" "-nologo" "/root/dev/rust/bevy-shaders-test/target/x86_64-pc-windows-msvc/debug/build/spirv-reflect-14456f7f069d4955/out/vendor/spirv_reflect.o"

  --- stderr

  error occurred: Failed to find tool. Is `lib.exe` installed?

You can't target the msvc toolchain without windows as msvc only runs on windows. Targeting x86_64-pc-windows-gnu may work if you install MinGW though I am not sure if it has the required libs to link to DirectX or Vulkan.

You can't target the msvc toolchain without windows as msvc only runs on windows. Targeting x86_64-pc-windows-gnu may work if you install MinGW though I am not sure if it has the required libs to link to DirectX or Vulkan.


has some macro like #[has_permission()] @[has_role] on api handler

has some macro like #[has_permission()] @[has_role] on api handler

Since casbin configuration is model driven and is based on the subject, implementing this might require to enfore a certain model which might not work for everyone. If you have an idea for a generic impl, happy to sync together :D


Issue #12 - Shard Iterator Type Can be Taken as an input

This PR helps to take iterator type as input.Current code only attends to TRIM_HORIZON and AFTER_SEQUENCE_NUMBER.Now user can input.
Note : this was tested locally within docker.

Great idea!. Thanks.


The personal access token section no longer aligns with reality. Please update

The PAT in GH has changed to be timebound, along with other things. This page is no longer up to date. Please fix.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: bb567df0-dc83-c3d4-a65e-6a792d4197b0
Version Independent ID: f5cb30b7-f99b-c50a-d322-194f9f7db831
Content: Tutorial: Publish Azure Static Web Apps using an ARM template
Content Source: articles/static-web-apps/publish-azure-resource-manager.md
Service: static-web-apps
GitHub Login: @petender
Microsoft Alias: petender

@bfox-sugarshot
Thanks for your feedback! We will investigate and update as appropriate.

I have a pending update to address this gap.


Create Azure service bus namespace from .Net

We are using .Net framework 4.6.x and looking for a way to create an azure service bus namespace from the azure.management sdk. Having trouble implementing programmatically, Any reference or direct documentation would be helpful.

@austinarban closing this since we haven't received any further details regarding the issue. Please reopen the issue with further details if it persists.


Canwaf/pandas-upgrade

Release the pandas versions to anything greater than 1.3 but less than 2.0. Since we're not doing anything fancy and their roadmap doesn't involve breaking the very basic functions we use we should be fine until we're not.

Don't forget you'll need to go via the whole releast proceedure to get this updated in gss-utils.


Can't resolve 'react/addons'

I tried to import { DropdownInput } from 'react-dropdown-input';
and I got the error Module not found: Can't resolve 'react/addons' in '.../node_modules/react-dropdown-input'
> npm -v react
2.15.5

+1

+1

+1

+1

+1

+2

No fix found for this issue?

+1

Been looking at this problem myself and it looks like react-select might be a good alternative third party component filling the same role.

+1

Same problem here....
Nice support btw.... since Oct 17' no fix. Im gonna use something else....

+1

+1

+1

+1

+1

+1


remove the clock sample

Remove the clock sample; fix https://github.com/dart-lang/dart-pad/issues/695.

lgtm

Bouncing broken link icons are only amusing for so long...

Yeah, looks like time ran out for this sample -

Ouch, @devoncarew.


Pull Requests are created for ignored dependencies as well

We have onboarded dependabot to one of the Repos and ignored a couple of packages so not to look for updates. Still, Pull Requests are created to ignore dependencies @dependabot.
Configuration:
.github/dependabot.yml
version: 2
registries:
nuget-org:
type: nuget-feed
url: https://api.nuget.org/v3/index.json
updates:

package-ecosystem: "nuget"
directory: "/" #root
schedule:
interval: "daily"
ignore:

dependency-name: "Polly"
dependency-name: "Polly.*"
dependency-name: "Moq"
dependency-name: "FluentAssertions"

We see PRs are created for Polly, Moq, FluentAssertions and Polly.* [ Polly.Extensions.Http ].
Please provide inputs on this to fix the issue!

Please provide a public Azure DevOps repository for reproduction of this issue!

we will work for Microsoft. Due to the privacy restrictions, I can't provide the repo information. I can answer for more questions if required. we followed configuration ignore options from here. Everything is working fine but only concern is, The pull requests are created even for ignored dependencies as well.

Isn't it really strange that Microsoft, the maker/owner of Azure DevOps cannot create a repro due to privacy concerns? Does it mean you don't trust your own Azure DevOps? Generally a repro is not a clone of your current repository but instead a simplified, fairly trivial representation of the issue. In this case, just the project files with the necessary files.
Anyway, I took the liberty to see if there is a new issue that we are not seeing in our own internal usage. Just like #399 , the ignore section works, unless you are using different versions for the task and docker container or are using older task versions.
The repository: https://dev.azure.com/tingle/dependabot/_git/repro-460
The pipeline: https://dev.azure.com/tingle/dependabot/_build/results?buildId=54447
Did I miss anything?

Hi @mburumaxwell,
Thanks for the response.
The dependabot.yml is exactly same except registries section.
dependabot.yml
version: 2
registries:
nuget-org:
type: nuget-feed
url: https://api.nuget.org/v3/index.json
updates:

package-ecosystem: "nuget"
directory: "/" #root
schedule:
interval: "daily"
ignore:

dependency-name: "Polly"
dependency-name: "Polly.*"
dependency-name: "Moq"
dependency-name: "FluentAssertions"

and azure-pipelines.yml contains the below piece of code.
azure-pipelines.yml
trigger: none # Disable CI trigger
schedules:

cron: '0 2 * * MON' # Monday at 2am UTC
always: true # run even when there are no code changes
branches:
include:
- 'main'
batch: true
displayName: Daily
pool:
vmImage: 'ubuntu-latest' # requires macos or ubuntu (windows is not supported)
variables:

group: Dependabot
steps:

task: NuGetAuthenticate@1
displayName: 'NuGet Private Feed Authentication'

task: dependabot@1
inputs:
useConfigFile: true
Please comment If I need to provide any further information.

What is the NuGetAuthenticate@1 task meant to do in this particular case?
Are you using one private feed with one ore more upstream sources?
Why is the nuget feed added to the registries of the configuration file?

Yes @mburumaxwell

the nuget feed will authenticate the upstream sources
We have two registries
Added two registries. One is nuget and the other one is own upstream source

For dependabot, things work a bit differently. If you have custom/private feeds you need to add a nuget.config file to the repository in the same directory as the one in the configuration. That's where you add nuget.org if you still use it. Otherwise just add the feeds you use there. In the configuration file you will then provide credentials for each feed that is private.
Sample configuration file at .github/dependabot.yml:
version: 2
updates:
- package-ecosystem: "nuget" # See documentation for possible values
  directory: "/" # Location of package manifests
  schedule:
    interval: "daily"
    time: "02:00"
  open-pull-requests-limit: 10
registries:
  contoso:
    type: nuget-feed
    url: '<feed-url>' # e.g. an organization-wide feed URL will look like https://pkgs.dev.azure.com/<org-name>/_packaging/<feed-name>/nuget/v3/index.json
    token: ':${{DEFAULT_TOKEN}}'

Example nuget.config file at nuget.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear /> <!-- ensure only the sources defined below are used -->
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="contoso" value="https://pkgs.dev.azure.com/contoso/_packaging/Alice/nuget/v3/index.json" />
  </packageSources>
</configuration>

Example pipeline file:
steps:
- task: dependabot@1
  inputs:
    useConfigurationFile: true
    azureDevOpsAccessToken: $(DEFAULT_TOKEN)

Remember to add DEFAULT_TOKEN to the pipeline variables with a PAT that has the following permissions:

Packaging (Read)
Code (Full)
Pull Requests Threads (Read & Write)

The owner of the PAT must also be added to permissions of the specific feeds either directly or via a group and with at least reader permissions.

Note that the above are examples that need to be tuned to your specific scenario.

Yes @mburumaxwell
dependabot@1 task have the below additional properties defined.
azure-pipelines.yml

task: dependabot@1
inputs:
useConfigFile: true
azureDevOpsAccessToken: '$(System.AccessToken)'
gitHubAccessToken: '$(GitHubAccessToken)'
extraEnvironmentVariables: DEPENDABOT_EXTRA_CREDENTIALS=[{"type":"nuget_feed","token":"$(VSS_NUGET_ACCESSTOKEN)","url":""}]

and nuget.config is same as you defined.

Okay. Now at this time I can only help with a reproduction because clearly there's something else missing.
You can share it directly to me via mburumaxwell at tingle dot software

Okay @mburumaxwell  do you want me to share the files?

Access to a sample repository will help. You can strip down all the super sensitive stuff but keep what is necessary to reproduce the issue.

Is it possible to get the access on the repo you created. So, I will create my branch and add the required files to repro ( or ) need to create a new Azure DevOps account with a new repo?

Easiest and fastest way to get this done is to create a new AzureDevOps organization even with you personal account. Add the files there then share with me or invite me to collaborate.

Sure @mburumaxwell  I will try to repro and drop you the link by tomorrow EOD.

Ok

Hi @mburumaxwell  Created a new repo and committed all the required files to repro the issue. Seen an issue to run the pipelines "No hosted parallelism has been purchased or granted. To request a free parallelism grant" and dropped an request for Azure DevOps Parallelism Request Please let me know the alias to invite you to the organization.

I'm not sure I understand the parallelism issue.
Email: mburumaxwell at tingle dot software

Invited to the Repo @mburumaxwell . You can find the issue from pipelines!

I have not received any invite yet.

I dropped invite on mburumaxwell@tinglesoftware

Still haven't received. You missed a dot somewhere in the domain.

Can you please drop me exact alias to invite you in @mburumaxwell

I'm not too conversant with cross-organization invites but according to this link, you will only need the email address that I shared earlier.

invited you again @mburumaxwell with the address<EMAIL_ADDRESS>

Is it possible to connect over teams channel @mburumaxwell I will schedule a call and discuss on the issue based on your availability

Sure, you can send and invite to the same email.

Hi @mburumaxwell  Scheduled a team's meeting to discuss on the dependabot ignore issue! Please confirm or reschedule based on your availability!

I have seen the scheduled meeting. I work in a different timezone GMT+[0-3] is most ideal so I will reschedule.

Sure @mburumaxwell  Please reschedule based on your availability!

Closing, conversation moved elsewhere.


Add pull to refresh. Also add Butterknife.

@dektar
Adds pull to refresh to the MainActivity's RecyclerView.  The spinner will show A) when the user enters the activity (in onResume) and B) when the user pulls to refresh.  I removed the refresh button in the menu, let me know if you'd like me to add it back.
I also added Butterknife for convenience, so that we don't need to do a typecast + findbyId every time we want a view.  I did some minor refactoring, splitting the api listener logic in onCreateView into a helper.

@dektar Addressed your comments.
I also fixed a crash I encountered multiple times, when returning to the LocationActivity from a permissions request.  When onBackPressed() was called, I was getting an IllegalStateException because onSaveInstanceState() had already been called.  It's kind of awkward, since we're not doing FragmentTransaction, which is the normal reason this crash happens.  Instead, AppCompatActivity inherits from FragmentActivity, which accesses the fragment manager in onBackPressed(), in case there's a fragment showing.  So, the normal fixes don't work.  I looked for an elegant solution for a while and couldn't find one, so I brute forced the solution by getting rid of the optimization to call onBackPressed().  I think this is fine because, with the current UX, we wouldn't expect people to be going back and forth between the MainActivity and LocationActivity frequently.

@dektar Done.  I'm just going to make an issue for the crash, since the brute force fix was trivial.


Fix NullReferenceException in Server.Dispose

If Server.Start is not invoked or throws an exception prior to assigning ListenSocket,
a NullReferenceException would be thrown from Dispose. This may hide the root
cause exception that prevented the server instance from starting up.

Thanks.


[T3A3][W14-A3] Low Jian Sheng

For your commits, do separate what you've done if possible. Otherwise, good job!


Light Vimeo image dimesnsions?

Hi there,
I am using the light prop to return the Vimeo thumbnail which works fine. It is returning a small thumbnail rather than a high res one and was wondering if this is a question for here or for Vimeo? Is there an option to choose a different dimension?
Current Behavior
Vimeo thumbnail dimesion is tiny.
Expected Behavior
An option to choose, or naturally return a higher res image?
Thanks in advance for your help!
Jack

Hi, would this be a Vimeo issue or something I can fix within the module? Thanks

@jackkemmish Did you sort this out?

Hi @Sid-Turner-Ellis I just patched the module locally, not ideal but there was no option to change the returned image dimensions unless I dug into the code.


Decimal delimiter on utils.xlsx_to_csv

Hi there,
I was wondering if there would be the possibility to have a decimal delimiter as an optional argument in the xls_to_csv function. As per now the field containing the comma will get escaped. My requirements for the project I'm working on is that we can have a variable user chosen (toolwide) field separator as well as a decimal separator.

sheet_to_csv takes an options argument and the FS option controls the field separator.  It will correctly quote values that contain the delimiter.
If the file does not specify a number format, it uses the default Number conversion which uses the decimal point as separator.  If you want to manually override it, you can go back and edit the formatted text directly:
/* ws is the worksheet you want to convert to CSV */
var range = XLSX.utils.decode_range(ws['!ref']);
for(var R = range.s.r; R <= range.e.r; ++R) {
  for(var C = range.s.c; C <= range.e.c; ++C) {
    var cell = ws[XLSX.utils.encode_cell({r:R, c:C})];
    if(!cell || cell.t !== 'n') continue;
    /* cell.w is the formatted text */
    cell.w = cell.v.toString(10).replace(/\./, NEW_DECIMAL_SEPARATOR);
  }
}


Reproduce performance of structshot

Hi,
Thank you for publishing the source code. I have access to OntoNotes and try to reproduce the results in your paper.
In the case of 1-shot learning with NNshot, I could obtain the same average score, i.e., 33.2, although the scores of each dataset are different from yours. However, I'm struggling with Structshot. My scores on CoNLL is much lower than yours, resulting in a lower average score on the three datasets.
In the case of 5-shot learning, I couldn't obtain as high scores as yours both with NNshot and Structshot. In average, Structshot produced worse performance than NNshot.
I wonder if you use a different set of parameters than the one you publish here. I hope you can help me to shed some light here.
Thank you very much for your time!
Best regards,
Nhung

Hi Nhung,
Thanks for your interest in our work. For OntoNotes, did you use the splits noted here: https://cemantix.org/conll/2012/data.html ?
The most important parameter for StructShot is tau, which you can tune here: https://github.com/asappresearch/structshot/blob/main/structshot/run_pred.sh#L22
It is a sensitive one, so I would suggest to try values like [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1]

Hi Yi,
Thank you very much for your prompt responses.
I use the splits mentioned here: https://cemantix.org/data/ontonotes.html, which I think is probably the same as the splits that you shared. Anyway, I'll double-check them.
Thanks for your suggestions of tau value!


Has anyone ever tried to use this with Jest?

I am trying to move from using Karma to using Jest for our testing.  We have some existing specs that are using jasmine-jquery to load html files to set up the dom for testing.
I am getting an error "Fixture could not be loaded" and I was wondering if anyone has used this library with Jest or how to determine why the fixture can't be loaded.
Thanks

Man, it's totally weird, but it seems that the library hasn't been updated in ages, like suddenly people in frontend stopped writing tests altogether. I don't know what's going on, but it's unlikely that it supports jest.

Thanks for the response @ganqqwerty.
I've been digging into the jsdom code that Jest uses to see how/where Jest would serve up a static file, but still trying to get it to work.
I saw this project was not being actively maintained and I haven't found any way to load an external fixture in Jest.


How to calculate and get the result 'Gap' ?

The explain of 'Gap' on help documents is said: 'Time elapsed between the start of the previous method and entry of this method', but I can't calculate and get the result.
Help me!  Thanks!

Hi @lijunhaos
You can calculate the gap by subtracting the start time of the current method by :
a) parent's start time if the current method is the first child, or
b) previous method's start time + exec if the current method is not the first child.

ok，I got it. Thanks!


RuntimeError: Failed to add edge detection

This code:
import weatherhat
import time

sensor = weatherhat.WeatherHAT()

while True:
        sensor.update(interval = 0.5)
        time.sleep(1.0)

...produces this error:
Traceback (most recent call last):
  File "/home/pi/go.py", line 4, in <module>
    sensor = weatherhat.WeatherHAT()
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.local/lib/python3.11/site-packages/weatherhat/__init__.py", line 80, in __init__
    self._ioe.on_interrupt(self.handle_ioe_interrupt)
  File "/home/pi/.local/lib/python3.11/site-packages/ioexpander/__init__.py", line 411, in on_interrupt
    self._gpio.add_event_detect(self._interrupt_pin, self._gpio.FALLING, callback=callback, bouncetime=1)
RuntimeError: Failed to add edge detection

Fresh install of latest Rasbian (no desktop) on Pi 4.
Any help gratefully received.

This usually means the interrupt pin is in use by some other process or feature. Make sure 1-wire, for example, is not enabled in /boot/firmware/config.txt (or Raspberry Pi configuration interfaces menu) since that uses GPIO 4 by default.
Note: if you're feeling brave, it would be useful if you tried the code here: https://github.com/pimoroni/weatherhat-python/pull/29

Away at the moment but will try when I get home. Thanks so much for an excellent, supporting response!

Since the port to Pi 5 / gpiod this code is no longer present, so hopefully it works/still works for you with the new v1.0.0 version!


Incorrect removal of debugger statement as dead code

var assert = function(x) {if (!x) {alert('c');debugger;}};
assert(prompt('a') === 'b');

is compiled to (in advanced mode)
if("b"!==prompt("a")){alert("c");debugger};

While
var assert = function(x) {if (!x) {debugger;}};
assert(prompt('a') === 'b');

is compiled to
prompt("a");

The former is the desired behaviour, at least in my case.

Interesting. I would guess that we're treating debugger has having no side effects somewhere and we should probably treat as side-effect-ful even though there are no side effects visible to the rest of the program.

It is generally treated as having side-effects, but somewhere we are not.


[CLI] refacto: change cli to github.com/spf13/cobra

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


podman ps: hangs

Every so often my system gets into a state where podman ps hangs. It's in such a state right now, and I was able to strace it. I think this is the relevant portion:
openat(AT_FDCWD, "/var/run/libpod/lock/87e6d0d059aa31c85ad412680277a9e05badda21e3c96810070de1232a431eae", O_RDWR|O_CREAT, 0600) = 18
fcntl(18, F_SETFD, FD_CLOEXEC)          = 0
getrandom("\x32\xc9\x80\xa9\x0b\x3e\xda\x92\x7b\xbc\xea\x28\xbb\x69\x87\x01\xa9\x92\xc6\x22\x08\xc4\x03\xda\xd9\xbf\x77\x86\x6c\x23\xff\x04", 32, 0) = 32
munmap(0x7f4ec2fff000, 8388608)         = 0
flock(16, LOCK_UN)                      = 0
close(16)                               = 0
getpid()                                = 2017
fcntl(18, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}

^^^ this is where it's hanging

Looks like it's hanging on the lock.
Any other processes going at the same time?

Podman processes, I mean

They're all stuck on a specific container's lock. Alright. So we have a race with many ps running against each other.

10563 seems to have the lock. It looks like it's stuck in the critical section and not releasing the lock.
Can you kill 10563 and see if things start working again?

Yes, that got things unstuck.

I have a suspicion that it might be https://github.com/projectatomic/libpod/blob/master/libpod/container_api.go#L411-L416 in attach - that's one of our more unusual uses of locking. I can't see an obvious way in which that would fail, but there could be a timing condition of some sort that locks attach into that critical section.
It's one of only two places where attach grabs a lock on a container, the other being a call to container.Status(), which is about as foolproof of an API call as we have.

The obvious conclusion is that we're getting stuck somewhere on the boltdb lock, so we never hit the Unlock() in attach.go

That actually can't be true. If it was the case, ps would not even get this far - it would fail to even retrieve the container. The lock issue has to be the container lock itself.

We must be getting stuck in syncContainer(). This has three basic portions: update container from database, update container from runc, save container status back to database. Given what I said in the last comment, this can't be the DB lock, so I strongly suspect our reading status from runc is hanging somehow?

Wait, attach is showing as stopped in the ps output, and the test container says sig-proxy... @edsantiago Is there any chance you're sending a SIGSTOP to the attach process as part of that test? We can't forward SIGSTOP, it's uncatchable, but if we got one right as we were in a critical section we will never release locks.

@mheon is there any possibility that podman attach is (occasionally, somehow) stopping the container?

I'm seeing the same thing, but it is hanging on a different lock.
# strace podman ps
...
stat("/var/lib/containers/storage/vfs", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/lib/containers/storage/storage.lock", O_RDWR|O_CREAT, 0600) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
getrandom("\xcb\x49\xe7\x22\xb0\x9f\xac\xe0\x82\xd4\x93\xa5\x02\x19\xee\x04\xab\xd1\xe2\x8e\xbf\x19\xbc\x2a\x02\x2e\xde\xf0\x2f\x39\x22\x1c", 32, 0) = 32
getpid()                                = 20095
fcntl(5, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}

# lsof /var/lib/containers/storage/storage.lock
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
podman   9277 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock
podman   9449 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock
podman  19624 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock
podman  19679 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock
podman  23091 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock
podman  31179 root    3u   REG    8,1       64 537033 /var/lib/containers/storage/storage.lock

# ps auxww | grep podman
root      9277  0.0  1.1<PHONE_NUMBER>6 ?        Ssl  Apr26   0:07 /usr/bin/podman run ...
root      9449  3.9  1.0<PHONE_NUMBER>8 ?        Ssl  Apr26   8:22 /usr/bin/podman run ...
root     19624  0.0  0.9<PHONE_NUMBER>4 ?        Ssl  02:44   0:00 /usr/bin/podman run ...
root     19679  0.0  0.9<PHONE_NUMBER>6 ?        Ssl  02:44   0:00 /usr/bin/podman run ...
root     20236  0.0  0.0  11460  1020 pts/0    S+   02:56   0:00 grep --color=auto podman
root     23091  0.0  0.2 879528  5432 ?        Ssl  Apr26   0:00 /usr/bin/podman run ...
root     31179  0.0  1.0<PHONE_NUMBER>0 ?        Ssl  Apr26   0:03 /usr/bin/podman run ...

That looks like a c/storage lock, so it should be a separate issue, but let's keep it here until we can get it localized. @ipmb Can you give the commands you're using to recreate this? podman ps at the same time as another command, lots of ps commands all at once...?

I'm setting up the containers in systemd with a config mgmt system. The tasks shouldn't be happening in parallel, but in very rapid succession:

podman ps | grep {image}
podman pull {image} (if it doesn't exist)
podman rm {container}
podman run --name={container} ...
podman inspect {container}

This happens a handful of times as each container spins up. I've only seen it in testing on my mac which has a bunch of virtualization (hyperkit -> Docker -> podman on vfs), so lots of overhead. I've noticed I need to sleep for a few seconds to reliably inspect a container after the run command. On a "real" machine I can run them back-to-back with && and it always seems to work.

It seems we're blocking on c/storage's graphLock which is taken during major c/storage operations. I suspect we're seeing a race between inspect trying to grab the layer store to compute image size on disk, and some operation in podman run (on bare metal, run will probably complete before the next command begins, so no race). It can't be anything to do with image pull or container creation (the container won't be available for inspect, so you'll get a "No Such Container" error instead of a race). This means it's probably something to do with mounting the container.

I believe this is fixed now, please reopen if it still happens.

I saw this today when automating something with the podman varlink API.
$ podman info
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.0, commit: e217fdff82e0b1a6184a28c43043a4065083407f'
  Distribution:
    distribution: manjaro
    version: unknown
  MemFree: 494456832
  MemTotal:<PHONE_NUMBER>0
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree:<PHONE_NUMBER>2
  SwapTotal:<PHONE_NUMBER>4
  arch: amd64
  cpus: 8
  eventlogger: file
  hostname: REDACTED
  kernel: 4.19.69-1-MANJARO
  os: linux
  rootless: true
  uptime: 149h 0m 53.01s (Approximately 6.21 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/REDACTED/.config/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/REDACTED/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 118
  RunRoot: /run/user/1000
  VolumePath: /home/REDACTED/.local/share/containers/storage/volumes

Again, killing one of the offending processes fixed the issue:
$ ps aux | grep podman
REDACTED     1135  4.1  0.4 1329772 66520 ?       Ssl  21:33   1:01 /usr/bin/podman varlink unix:/run/user/1000/podman/io.podman
REDACTED     1152  0.0  0.3 1182052 60296 ?       Ssl  21:33   0:00 /usr/bin/podman --root /home/REDACTED/.local/share/containers/storage --runroot /run/user/1000 --log-level error --cgroup-manager cgroupfs --tmpdir /run/user/1000/libpod/tmp --runtime runc --storage-driver vfs --events-backend file container cleanup 2a5ee875bae62eed4e3c7ac10c6868f1aba50e3629cb2a5319de3a09a402d3ee
REDACTED     3179  0.0  0.0   6272  2380 pts/3    S+   21:57   0:00 grep --colour=auto podman
REDACTED    12845  0.0  0.0 119636     0 ?        S    Sep16   0:00 /usr/bin/podman
$ kill 1135
$ ps aux | grep podman
REDACTED     3269  0.0  0.0   6272  2220 pts/3    S+   21:58   0:00 grep --colour=auto podman
REDACTED    12845  0.0  0.0 119636     0 ?        S    Sep16   0:00 /usr/bin/podman

The deadlock seemed to happen when I tried to call io.podman.GetContainer immediately after calling io.podman.StartContainer. The io.podman.GetContainer was stuck reading the response from the varlink API endpoint.

I'm experiencing the podman ps hang when running a container with elevated privileges.
# podman version
Version:            1.7.0-dev
RemoteAPI Version:  1
Go Version:         go1.13.5
OS/Arch:            linux/amd64

strace podman ps
read(12, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 3536
read(12, "", 4096)                      = 0
close(12)                               = 0
munmap(0x7fbb4075c000, 131072)          = 0
flock(11, LOCK_UN)                      = 0
close(11)                               = 0
futex(0xc000060bc8, FUTEX_WAKE_PRIVATE, 1) = 1
openat(AT_FDCWD, "/var/lib/containers/storage/libpod/bolt_state.db", O_RDWR|O_CREAT|O_CLOEXEC, 0600) = 11
epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1208089800, u64=140442343701704}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 11, 0xc0001c8d9c) = -1 EPERM (Operation not permitted)
flock(11, LOCK_EX|LOCK_NB)              = 0
fstat(11, {st_mode=S_IFREG|0600, st_size=131072, ...}) = 0
pread64(11, "\0\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\355\332\f\355\2\0\0\0\0\20\0\0\0\0\0\0"..., 4096, 0) = 4096
fstat(11, {st_mode=S_IFREG|0600, st_size=131072, ...}) = 0
mmap(NULL, 131072, PROT_READ, MAP_SHARED, 11, 0) = 0x7fbb4075c000
madvise(0x7fbb4075c000, 131072, MADV_RANDOM) = 0
futex(0xc00008d2c8, FUTEX_WAKE_PRIVATE, 1) = 1
geteuid()                               = 0
munmap(0x7fbb4075c000, 131072)          = 0
flock(11, LOCK_UN)                      = 0
close(11)                               = 0
newfstatat(AT_FDCWD, "/var/run/libpod/exits/9d6c432b35e3b7fffe53c682998fe4a487e8a986d51d56fe5bf6e5c00ee1961e", 0xc0004f61d8, 0) = -1 ENOENT (No such file or directory)
futex(0x560d3cd72408, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x560d3cd72408, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x560d3cd72408, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
...repeat futex ad ifinitum...

@pgporada have a reproducer we can try?

 Perhaps this is a different issue?

@pgporada when you say kill the pid of the container, what exactly are you killing?
@mheon dont we have issues when things get killed from underneath conmon?

It's conmon getting killed before the container, the container dying before
conmon should be fine.
On Sun, Dec 22, 2019, 13:06 Brent Baude<EMAIL_ADDRESS>wrote:

@pgporada https://github.com/pgporada when you say kill the pid of the
container, what exactly are you killing?
@mheon https://github.com/mheon dont we have issues when things get
killed from underneath conmon?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/658?email_source=notifications&email_token=AB3AOCGM7HYIRJX6AYP23MLQZ6UCPA5CNFSM4E4AQJA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHPWO3Q#issuecomment-568289134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCFGDASDCEG7GT2U7OTQZ6UCPANCNFSM4E4AQJAQ
.

Conmon dying could hang us as we wait for the exit file, though.
On Sun, Dec 22, 2019, 16:29 Matthew Heon<EMAIL_ADDRESS>wrote:

It's conmon getting killed before the container, the container dying
before conmon should be fine.
On Sun, Dec 22, 2019, 13:06 Brent Baude<EMAIL_ADDRESS>wrote:

@pgporada https://github.com/pgporada when you say kill the pid of the
container, what exactly are you killing?
@mheon https://github.com/mheon dont we have issues when things get
killed from underneath conmon?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/658?email_source=notifications&email_token=AB3AOCGM7HYIRJX6AYP23MLQZ6UCPA5CNFSM4E4AQJA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHPWO3Q#issuecomment-568289134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCFGDASDCEG7GT2U7OTQZ6UCPANCNFSM4E4AQJAQ
.


it won't let me uninstall it

I downloaded playlist icons from the marketplace and when I try to uninstall it, click "reload now" and it is still there. Worst of all, there is two of them (no pun intended) now, I can uninstall the 2nd one, but after another reload it still will be there. So how to uninstall it?

This is a marketplace issue, not one caused by playlist icons.
If you open the spotify dev console, and navigate to the application tab -> localstorage, there should be a key related to the marketplace holding the info of all installed extensions. There you can remove playlist icons.
If it doesn't work, or the key is not there, I would recommend to reset the marketplace.