Vulnerability Roundup 19

Here are all the vulnerabilities from https://lwn.net/Vulnerabilities
since our last roundup.
cc: @vcunat @the-kenny @7c6f434c @globin @bachp @nlewo @fpletz @NeQuissimus. (nice goup!)
Note: The list of people CC'd on this issue participated in the last
roundup. If you participate on this roundup, I'll cc you on the next
one. If you don't participate in the next one, you won't be CC'd on
the one after that. If you would like to be CC'd on the next roundup,
add a comment to the most recent vulnerability roundup.
Permanent CC's: @joepie91, @phanimahesh, @the-kenny,
@NixOS/security-notifications
If you would like to be CC'd on all roundups (or removed from the
list), open a PR editing
https://github.com/NixOS/security/blob/master/lwnvulns/src/bin/instructions.md.
Notes on the list

The reports have been roughly grouped by the package name. This
isn't perfect, but is intended to help identify if a whole group
of reports is resolved already.
Some issues will be duplicated, because it affects multiple
packages. For example, there are sometimes problems that impact
thunderbird, and firefox. LWN might report in one vulnerability
"thunderbird firefox". These names have been split to make sure
both packages get addressed.
By each issue is a link to code search for the package name, and
a Github search by filename. These are to help, but may not return
results when we do in fact package the software. If a search
doesn't turn up, please try altering the search criteria or
looking in nixpkgs manually before asserting we don't have it.
This issue is created by https://github.com/NixOS/security

Instructions:

Triage a report: If we don't have the software or our version isn't
vulnerable, tick the box or add a comment with the report number,
stating it isn't vulnerable.
Fix the issue: If we do have the software and it is vulnerable,
either leave a comment on this issue saying so, even open a pull
request with the fix. If you open a PR, make sure to tag this
issue so we can coordinate.
When an entire section is completed, move the section to the
"Triaged and Resolved Issues" details block below.

Upon Completion ...

[ ] Run the issue through reformat one last time
[ ] Review commits since last roundup for backport candidates
[ ] Send an update e-mail to<EMAIL_ADDRESS>[ ] Update the database at https://github.com/NixOS/security

Without further ado...
Assorted (28 issues)

[x] #642649 (search, files)  DirectFB: two vulnerabilities
[x] #480386 (search, files)  cvs: remote code execution
[x] #644511 (search, files)  dcraw: denial of service
[ ] #712299 (search, files)  ed: denial of service
[x] #712370 (search, files)  firejail: three vulnerabilities
[ ] #712364 (search, files)  gd: two denial of service flaws
[ ] #712297 (search, files)  groovy: code execution
[x] #712359 (search, files)  hesiod: two vulnerabilities
[ ] #712057 (search, files)  java: multiple vulnerabilities
[ ] #712058 (search, files)  java: multiple vulnerabilities
[ ] #712059 (search, files)  java: two unspecified vulnerabilities
[ ] #712369 (search, files)  kernel: privilege escalation
[ ] #712065 (search, files)  libav: multiple vulnerabilities
[ ] #712300 (search, files)  libnl3: privilege escalation
[x] #712363 (search, files)  tiff: code execution
[x] #610398 (search, files)  lua: code execution
[ ] #712060 (search, files)  mapserver: code execution
[ ] #712067 (search, files)  mariadb: multiple unspecified vulnerabilities
[ ] #712068 (search, files)  mysql: multiple vulnerabilities
[ ] #712068 (search, files)  mysql: multiple vulnerabilities
[x] #712298 (search, files)  opus: code execution
[ ] #712155 (search, files)  php: code execution
[ ] #711946 (search, files)  php-PHPMailer: information disclosure
[ ] #640613 (search, files)  ppp: denial of service
[ ] #662269 (search, files)  squid: nonce replay vulnerability
[ ] #712371 (search, files)  t1lib: code execution
[ ] #711945 (search, files)  webkit2gtk: multiple vulnerabilities
[ ] #712304 (search, files)  xtrabackup: information disclosure

qemu (2 issues)

[ ] #712302 (search, files)  qemu: denial of service
[ ] #712301 (search, files)  qemu: denial of service

lua5_1: that's an old bug

Re: qemu: what is the current status of maybe-we-should-update-to-2.8?

firejail: old story

I never saw a PR come through trying 2.8, and I'd still like to try it. Note cvs is an old bug, but we are vulnerable :)

tiff: we already apply debian/patches/17-CVE-2017-5225.patch since the last roundup, so that should be OK.

libopus: OK since 140d135e.

ed is good on master but needs a backport to stable. It looks like all three commits to ed can be backported. what do you think, @vcunat?

It's just 1.13 -> 1.14.1 and some nitpicks along.  I'd think the update is safe (but a mass rebuild).

OK I'll hold on to that backport until later.

webkitgtk: the version on master should be OK, but we have 2.12 as the default on 16.09, so it's probably vulnerable.  I can't see any complaints since we updated the default on master, and the versions should be compatible (in this direction), so I think we should switch.

Sounds fine to me, re: webkitgtk. Not ideal, but nor is webkitgtk ;

I set up a PR to push changes to: https://github.com/NixOS/nixpkgs/pull/22131 you can both push to that branch, I think. Also that branch is building in a hydra (link in the description.)

t1lib: I believe CVE-2011-5244 was fixed by our particular patch for CVE-2010-2642.  That patch from Gentoo is certainly equal to the one we apply (taken from Arch).

Collapsed the done items. Almost half done :o

Might be good to upgrade mariadb to 10.2.x before 17.03 branch-off.

Java and kernel are fine

So is our Groovy version

👍 I collapsed the completed tasks.

As dar as I can tell squid on 16.09 is not vulnerable. It has the same version as master 3.5.23 and the issue only mentiones 3.4 and <IP_ADDRESS> through 3.5.1.

We already have a fixed version of ed (1.14.1) and not present on release (1.13) because it was introduced in 1.14.

We're safe on the second qemu issue (CVE-2016-10028) because we don't build our qemu with virtio-gpu-3d enabled.

The first qemu issue (CVE-2016-9923) is fixed in 2.8.0 and the patch is not easily applicable to our version.

Pushed fixes for libav, ppp and gd to #22131.

I think we can go ahead and upgrade qemu: https://github.com/NixOS/nixpkgs/pull/22134 but probably shouldn't do so for stable... :/ hard question. We don't really have a way to track vulnerabilities we know NixOS has but aren't fixing.

Is there any known incompatible change for Qemu 2.8 (or any regressions)? Qemu looks like software where just doing the upgrade could be safe.

squid: old stuff

@bachp technically, one of the links references a bug in 3.5.22

Just pushed new qemus everywhere. Good work, all done! Thank you! :D


Add acceptance test for syncrepl in database

Prove and fix #132

This PR also fix the syncrepl idempotent issue.

@kronos-pbrideau thanks


Add odd-width/odd-dimensioned lossy pixel tests and images

I noticed that all of the lossy reference images had even dimensions. I wasn't sure what the process was for adding new test images.
I wrote code that incorrectly handled some end pixels in rows but passed all the tests. I created these two tests to check that my function matched the current fill_single implementation. The 3x3 test is what my code failed at first since I was handling 2 pixels at a time due to the chroma plane.
https://github.com/okaneco/image-webp/commit/0f96c2efbd6037eac741b6c5a4fc4398ca11add5

The existing images are all from https://developers.google.com/speed/webp/gallery. The structure is the webp file is in tests/images/ and an equivalent PNG decoded with "-nofancy" in tests/reference/. I should really add a README to the tests directory explaining all that...
Not sure the best way to do additional images. As far as I can tell libwebp is the only decoder in widespread use. So maybe it makes sense to feed a large number of tiny images (or cropped pieces of larger ones) with random compression parameters into that encoder and use those as a test suite?

#18 added some odd-sized images and more documentation of how the tests images were obtained


GRU has non-standard def?

Hi, I was wondering if there was a reason for using different formulas for the definition of the GRU? If I'm not mistaken, the standard formulas:

and the ones used in this implementation: https://github.com/danijar/dreamerv3/blob/423291a9875bb9af43b6db7150aaa972ba889266/dreamerv3/nets.py#L131-L140
are different.

Hi, I think the version I'm using was designed by Nvidia for cuDNN. The motivation is that you compute all relevant quantities in a single matmul.

Thanks for the answer!


Fixes concurrency bug on mt_append_event function when processing the same stream.

Currently, if you have a concurrent calls from different processes appending events to the same stream id, it can lead to a mismatch in the version stored in the mt_streams table and the version stored in the mt_events table. This is due to the fact that the selection and updating of the event version on the mt_streams table is not placing any type of lock on the row after selecting out the version before updating. This creates a race condition in the mt_append_event function. Which then leads to a duplicate version key constraint error.
This change puts a FOR UPDATE lock on the event version SELECT from the mt_streams table. This puts a ROW EXCLUSIVE lock on the that given streams row, causing any other concurrent calls on the same stream id to wait on that lock, resolving the concurrency issue.

@oskardudycz @ericgreenmix So I don't know to be perfectly honest. I don't think it would change behavior, but it would definitely have an impact on performance (but that might be negligible). I like the idea of making that be an explicit opt-in feature so we don't break everyone's existing database structure.

When this situation occurs, and the version number becomes mismatched, it is a pretty catastrophic bug. We observed this two times yesterday in our production system while under heavy load.  Once the mismatch occurs, no further events can be added to the stream, effectively bringing the system down at least for that particular user using that aggregate.  Seems like more of a bug than a desired optional behavior.

I can tell you from the testing we have done, performance impact is very much so negligible. Performance impact shouldn't even really be a consideration here though, this is definitely a bug in running any type of asynchronous append requests with the event store

@ericgreenmix And no one is saying don't do this, we're just asking that the behavior be opt in.

Right, but I think to the point @jarroda made. Having this be opt in, is basically like saying, "Opt in if you do not want your system to have consistency issues that can corrupt your stream"

The opt-in is to avoid breaking SQL compatibility and all the headaches that causes for support.

I guess I just don't understand the philosophy of making, in my opinion, a major bug fix opt-in.
Requiring users to run a migration on a function to patch a bug that can break an entire event stream, doesn't seem like it should be an issue to me.

@ericgreenmix imho it should be opt-in before 4.0 then we can change that to default. As for now we didn't have exact complains about that with repro steps code (this also doesn't have the tests, yes I know that it's not easy to write them...). So I'd still consider that at that exact moment - the potential performance impact would be worse currently for the users than the bug by itself (especially that it can be fixed by changing transaction serialization if needed).
I'm working currently on making benchmarks easier to be run (through Azure Devops) after that we could easier make performance comparison.
Having that plus what I and @jeremydmiller wrote above I'd still vote for opt-in until the next major release.

@oskardudycz I would hardly call changing to using a Serializable isolation level a "fix". That would be an across the board huge performance impact. Instead of a very negligible performance impact on an exclusive row in the streams table.
What exactly do you mean by this statement?

As for now we didn't have exact complains about that with repro steps code

I also don't know how consistently reproducible tests could be written for this. It is a race condition, so it could happen in one test run but not in the other.
It seems like the impact of this bug is not being fully grasped and is being treated more like an enhancement.
During high traffic on our production environment yesterday, we had this happen on two different streams. The mt_streams version ended up being 2 version numbers behind the version number on the mt_events table. This corrupted those streams, causing System.Exception: 23505: duplicate key value violates unique constraint "pk_mt_events_stream_and_version" to be thrown on any subsequent appends to that stream. We had to manually intervene and fix the version number in the mt_streams table.
I have applied this patch already to our production environment, which has fixed this consistency issue for us and we are seeing near identical benchmarks with the update applied.
Making this bug fix opt-in, would be a big pain, based on how the SQL function is currently inserted and checked with the schema patch functions. If you both are insistent that it has to be opt-in in the 3.x version of Marten, I would rather just wait to 4.0 for this PR to be merged. Anybody who hits this bug in 3.x can see it here and manually change their function to alleviate it.
Although, I would still strongly suggest this should be put into the next 3.x version of Marten, as I believe this is a very critical bug.

@ericgreenmix Why would it be difficult to make this opt in with schema patches? It's just going to do a canonicized SQL comparison of what it should be versus what it is. I don't think making it opt in is really going to be that difficult.

@jeremydmiller I had a misconception about how that patching worked. You are right, it isn't difficult.
I went ahead and made this optional with a UseAppendEventForUpdateLock property on EventGraph . I still don't agree with this bug being "fixed" with an opt-in setting, however if this is what it takes to get it out in 3.x of Marten, to make it easy on others that hit this issue, then I am fine with it.

I've got one minor comment I added above, but it's not that important. Go for it is my call.

Great point, done.

@ericgreenmix It would help if we have some documentation around this. Can you please take a look at it?


added top position sticky to table header

Makes it easier to navigate and use quick filters while scrolling through the table

Thanks @manizm for this. Let me just check the UX then I agree this can be merged.

@manizm thanks for this contribution. I've included it in a design update we are working on. https://www.loom.com/share/9f0fad2a62334e5687eec5fc4b2deea4


[system] Allow for nested css rules in system prop

This PR adds support for nested css rules in the system css prop:
<Box css={{
  color: 'blue,
  '&:hover': {
    color: 'pink',
    '> button': {
      color: 'red'
    }
  }
}} />

I'm closing, the resolution of the problem has been deprioritzed for a long time. We had a blocker, v5. You will get this feature by using the css props of emotion. As for the system, we plan to revamp it, have a second shoot at it.


Refactor libp2p network backend

Refactored the verbose code block in the async task.
COMING SOON: Further refactoring in the mixnet branch.

Going into refactors. I think we could take over branches of those methods as well, ideally looking forward for something lie:
match x:
    foo(*args) => handle_foo(*args),
    bar(*args) => handle_bar(*args),
    ....

Where each branch has its own method.

Going into refactors. I think we could take over branches of those methods as well, ideally looking forward for something lie:
match x:
    foo(*args) => handle_foo(*args),
    bar(*args) => handle_bar(*args),
    ....

Where each branch has its own method.
Yeah, I agree. I'm going to work on it after cleaning up some stuff in the mixnet branch, since there are more things to be refactored the entire code structure could be changed by it.

I'll postpone this PR until we have a working version of testnet with mixnet that was already merged into master.

Closing this PR since we may need more fundamental refactorings


Connecting with MongoDB root role fails when not using admin database

Hi,
I have a local MongoDB instance with authentication enabled and a user that has the root role. So this user can connect to any database on that instance when running from the shell.
When running the sample then the connection fails with errors like this:
State: %{auth: [{"superadmin", "*****"}], database: "ecto_simple", opts: [backoff: 1000, port: 27017, on_connect: #PID<0.1382.0>, pool_name: Simple.Repo.Pool, pool: Ecto.Pools.Poolboy, otp_app: :simple, repo: Simple.Repo, adapter: Mongo.Ecto, hostname: 'localhost'], queue: %{}, request_id: 0, socket: #Port<0.9116>, tail: "", timeout: 5000, wire_version: nil, write_concern: [w: 1]}
Only when changing the database to admin then the connection succeeds but nothing is returned.
I have tried prefix when running the query and also have tried the schema_prefix attribute but it seems that the prefix is not working as there is always a request to the admin database and not the echo_simple database.
Thanks in advance
Dannys

Thanks for the fast reply, that solved it. Is this already documented somewhere?

I don't remember if it is documented somewhere. I will need to double check, but all of the options you pass into your repository configuration will be passed down to the underlying mongodb package. So any options that Mongo.start_link/1 function will take, this adapter will also take.


enh(asciidoc) support unconstrained syntax

Changes
resolves #2869
Checklist

[x] Added markup tests
[ ] Updated the changelog at CHANGES.md
[x] Added myself to AUTHORS.txt, under Contributors

Hey @joshgoebel thanks for your prompt reviews!

Maybe I should mark this pull request as draft for now?

I was hoping to maybe prevent you from traveling down some dead-end roads but if you just want to explore on your own and then ping me later, that's fine.  One concern I always have is that it's possible to do many things with our engine that we don't actually encourage and there is a certain line where you start "parsing" code instead of merely pattern matching it.  And we try not to cross into that territory.
I worry getting the rules perfect around a single * might be hard and that makes me wonder how badly we were doing it before. :-)
You picked a non-trivial problem for starters. :-)

I was hoping to maybe prevent you from traveling down some dead-end roads but if you just want to explore on your own and then ping me later, that's fine.
One concern I always have is that it's possible to do many things with our engine that we don't actually encourage and there is a certain line where you start "parsing" code instead of merely pattern matching it. And we try not to cross into that territory.

You are probably right.
I will keep the pull request active and explicitly request your review when there's something interesting. Are you OK with that?

I worry getting the rules perfect around a single * might be hard and that makes me wonder how badly we were doing it before. :-)

I adopted a new strategy!
I've implemented basic rules without all the subtle variations. I think the result is quite good :smile:
The regular expressions are relatively simple to read and all the tests are passing.

You picked a non-trivial problem for starters. :-)

I like challenges :1st_place_medal:

The regular expressions are relatively simple to read and all the tests are passing.

Simple is good, but I think it only works so well because you've just covered the very basic cases.  I'm not convinced yet we can solve the * problem 100%, but if we can it's going to be with something gnarly like I mentioned earlier:
/\*(([^*\n\\]|\\[^\n])+\n)+([^*\n\\]|\\[^\n])*\*/

ie matching:

CONTENT = [ content up to \n (and handling escapes)]
LINE = CONTENT [\n]

"*"LINE
LINE(possibly repeated multiple time)
CONTENT"*"

You can play around on https://regex101.com/.  It's super useful for stuff like this.

The regex I pasted earlier.

Made it a bit easier to see with named groups:

So if we just want to focus on minimum to merge I think we need to focus on:
      // inline contrained string (single line)
      {
        className: 'strong',
        // must not precede or follow a word character
        begin: /\B\*\w+\*\B/,
      },
      // inline constrained strong (multi-line)
      {
        className: 'strong',
        // must not precede or follow a word character
        begin:

We need more tests cases and those rules need to be widened so as to not fail with simple content in between the ** pair (examples above in my most recent responses).  Once that's working there are some minor consistency issues we could go over or I can maybe just clean up myself.

Add changelog entry please.

@joshgoebel Done.
Once it's merged, I will also open a follow-up pull request to apply the same unconstrained rules to emphasis and monospaced.


Applying in aemCheckout vaultCheckoutArgs before arguments

For command aemCheckout arguments vaultCheckoutArgs being applied before basic arguments like host which results in vault error
[ERROR] Unexpected --force, while processing . Type --help for more information.

OK, I know the problem will be fixed in 1.1.1 soon

I thinking to refactor this to even:
`aemVlt -Paem.vlt.command='checkout --force $contentPath'
I don't remember exact order of parameters in VLT tool (that's the problem because it is sensitive), so that I'd like to move responsibility for configuring it correctly to plugin user and let him freedom in running all possible VLT commands:
https://docs.adobe.com/docs/en/crx/2-3/how_to/how_to_use_the_vlttool.html
not only 'checkout' but also 'commit' 'import' , 'export' ...

pr created
https://github.com/Cognifide/gradle-aem-plugin/pull/21/files
@heorhii-burlachenko please check if it fits your need.
Current order will be
checkout -f D:\Projects\gradle-aem-example\content\src\main\content\META-INF\vault\filter.xml http://localhost:4502 / --force D:\Projects\gradle-aem-example\content\src\main\content

@piotr-wilczynski Looks good for me, thank you very much. Admire your work, guys.

fixed. Check out the new version 1.1.1


[bisection_without_regressed_by] Set the regression keyword when setting regressed_by field

To avoid cases like https://bugzilla.mozilla.org/show_bug.cgi?id=1782411#a2638736_575867

Similar to (part of) #1415.

Similar to (part of) #1415.
We do not need to merge tools here like in #1415. We could just simply always add the regression keyword as part of the change. Bugzilla will silently ignore that if the keyword already exists.

@marco-c do you think we should always set the regression keyword (even when we needinfo the author/assignee)?

Yes I think it makes sense!


Files are not uploaded as public even with publicFiles at true

First, thank you for this nice provider :)
I'm using a fine grained ACL bucket.
My config is:
module.exports = ({ env }) => ({
    upload: {
      config: {
        provider: '@strapi-community/strapi-provider-upload-google-cloud-storage',
        providerOptions: {
            bucketName: env('GCS_BUCKET_NAME'),
            serviceAccount: env.json('GCS_SERVICE_ACCOUNT'),
            publicFiles: true,
            uniform: false,
        },
      },
    },
});

Uploaded files are not set as public in the bucket.
I think the issue come from this line:
https://github.com/strapi-community/strapi-provider-upload-google-cloud-storage/blob/a6b593744d151c9890156182ca1f1d8c5009e7c8/lib/provider.js#L236
fileAttributes needs to be set as a parameter:
file.stream.pipe(bucketFile.createWriteStream(fileAttributes)).on('finish', () => {

You can find the documentation here if needed.

I created a PR https://github.com/strapi-community/strapi-provider-upload-google-cloud-storage/pull/142

Hello @femtozer,
Thanks for your contribution !
I will merge it as soon as possible, you just need to add your signature.
Thanks

Hello @Lith
I don't know if there was an other way but I had to create a new PR #143 to have the commit verified.

@femtozer @Lith Hi, any updates on this fix?

@femtozer @Lith Any update on this fix!!

Any idea when the #143 will be merged ?

Same problem here, any updates?

@Lith  let's merge #143

another reason why it can happen is that your all files are public unavailable. you have to make it accessible for everybody
https://cloud.google.com/storage/docs/access-control/making-data-public

Strapi V4.2.x is works fine and update Strapi V4.3.x  upload image to library, Internal Server Error and 500
I'm not sure if it's a plugin issue ?

Sorry about the delay... I would like more people manage this repository but I'm alone and I was really too busy during last month :'(
I've merge #143 and I will publish new release on NPM.
Please apologies 😢

Thanks, @Lith 🙌


Failing test case of github issue

https://github.com/doctrine/mongodb-odm/issues/1428

This has been included in #1432. Thanks!


Short solution needed: "How to uninstall Node.js from Ubuntu" (nodejs)

Please help us write most modern and shortest code solution for this issue:
How to uninstall Node.js from Ubuntu (technology: nodejs)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments explain solution.
Link to this issue in comments of pull request.

How to uninstall Node.js from Ubuntu


Added binding for APPGW_CONFIG_NAME_PREFIX in helm chart

Checklist

[x] The title of the PR is clear and informative
[x] If applicable, the changes made in the PR have proper test coverage
[x] Issues addressed by the PR are mentioned in the description followed by Fixes.

Description
When work with agic in shared mode, is a normal scenario when you have 2 or more environment behind the same agw and promote from one to the next (dev -> uat -> prod).  In this case, the namespaces and services have the same naming, so all the agw internal resources are in conflict between them in the different environments.
Agic supports the use of prefix that is added in each resource but that setting is not editable using helm.
This PR added support to manage the prefix in order to allow share the same agw between different clusters those have the same application in different environments.

Fixes
#957

Added a minor comment
Done! :)


Allow To define tolerance for a key

I want to define a tolerance for a key.
the case is the foolowing one :
I create lot of unit tests on a low level of granularity. So I get only one value (=one key) in result.
In this case, i can't define tolerance.
David

I'll not support tolerance for a key, but I could imagine a case where you define that your result-set is only cell (with keyword one-cell-query) and in this specific case the unique cell will be a value and not a key.

As a workaround at the moment I'd recommend to use and additional known field and put it as the key something as:
<system-under-test>
  <execution>
    select 'A', ... your query
  </execution>
</system-under-test>
<assert>
  <equalTo tolerance="10">
    <resultSet>
      <row><cell>A</cell><cell>100</cell></row>
    </resultSet>
  </equalTo>
</assert>

After implementing it, I don't think, it's especially smart to have to specify that the result contains one row at the query level. I'll move this to the equalTo level
<equalTo behavior="single-row">
...
</equalTo>

preview available at https://ci.appveyor.com/project/CdricLCharlier/nbi/build/1.14.0-ci.24


Update links - vegetation indices collection

Update link to product user manual
make links open in new tab

@zcernigoj lets try to merge this changes to test the deploy


Path to OSM file is hard coded

Details
Should make the input to get_osm2pgsql_command() more flexible to allow setting full path.
https://github.com/rustprooflabs/osm2pgsql-tuner/blob/0.0.4/osm2pgsql_tuner/tuner.py#L216

#22 fixed this in the backend.  The hard coded path was moved to routes.py in the webapp, so the webapp and API are still hard coded to that path.  A new issue can be opened to fix that if desired.


Small csv export improvement

Make attributes always last column
Format Map object as simple string instead of json

Here is an example
report (4).zip

You are right.

Merged, thanks.


Add Morse potential

This PR resolves Issue #6 by adding the Morse potential. Specifically, three functions have been added to energy.py:

morse
morse_pair
morse_pair_neighbor_list

which have the expected behavior. The implementation was modeled after the Lennard Jones potential, though there is a slight difference in how r_onset and r_cutoff are scaled by sigma. I am open to a discussion here...
I have tested this myself, but is there a more rigorous testing protocol in place, or anything like that?

This looks great. Can you take a look at energy_test.py and add a test or two following soft_sphere and lennard_jones? I'm not totally sure what the best way to test potentials is, but so far I've made sure that e.g. the minimum is at the right placea nd certain known values are correct. Often it is good just to make sure API changes don't break the potentials.

Ok, I added tests to energy_test.py. They all run fine except for test_morse_small_neighbor_list_energy, which is based on test_lennard_jones_small_neighbor_list_energy, but this is because there was a fix to the neighbor list for small systems that was made after I made the branch for this PR. So once this gets merged they should all work.
Should I take this test out and put it back in later, or try to merge on my end, or just leave it?

Awesome, thanks for all the changes! This looks great.


hint_text naming is conducive to development errors.

Is your feature request related to a problem? Please describe.
No, except that it is a design inconsistency. When developing, the inconsistency between hint_text and size_hint often leads to errors. It is an understandable design since "hint text" is a noun and "hint size" is not really, but still is conducive to typos.
Describe the solution you'd like
String replace "hint_text" to "text_hint", "placeholder", or "text_placeholder" for version 3. This could mimic CSS design decisions.
Describe alternatives you've considered
Creating a fork with a different name for my personal use. Creating a CSS wrapper that simply covers up the problem (this is not a good idea, but just putting it to answer the question.) Kivy is a nice project and I'd like to stick with a version of it.
Additional context
None.

Using VS Codium's project search feature, I found no variable named "placeholder", so renaming "hint_text" should theoretically cause no problems. (Even if such a variable was found, it could be dealt with.) I will go ahead and fork 2.3.0 for testing.

The code modification, build, run, and test was successful. To the Kivy Team, please consider this feature (change) request. Either way, thank you for all your work. The fork is available in my account for your review, and simply involved a "find and replace" of the sub-strings after checking for variable collisions.

Hi @skersdaniel, I agree, as I often found myself or users using placeholder, then after realizing that the feature was named hint_text.
Feel free to propose a PR, and add the migration instructions to https://github.com/kivy/kivy/blob/master/doc/sources/migration.rst

Well, the instructions are above if the community wants the change. I don't know much about git and community code management other than git clone and git reset. If it were me, everything would mirror CSS.

PS. In my fork, which I probably won't use for a while since I'm playing with Imgui now, all I did was carefully rename all instances of "hint_text" to "placeholder" in the project. I know this is overkill, but if the design changes, it might as well be complete. Of note, my fork is a quick test, and also of note, because I have black formatter installed, it made quite a few changes automatically that I never saw. Something to be aware of.

@misl6 Hello is this issue still relevant and need working on it? I could try to tackle it


looks: when there is no file, display a single bar in the quickfix

it's really a question of aesthetics. i'm often dealing with quickfix lists not mapped to any files. in that case the display is
 |  |data

I find it ugly, so this changes it in that case to just:
  |data

I don't think we can do much better. If we don't put anything then vim puts ||  anyway, which is even worse. Maybe we could have a single leading space, maybe a space after the bar or something, but this seems quite diminishing returns. And a space after the bar will be visible on many configs which highlight trailing spaces. I think after this change, the display is good enough and to my eyes much better than with the double separator.

Although maybe using quickfix without any backing file may not be quite "the right way". If you think I'm holding it wrong, feel free to reject the pr of course.

You're definitely not holding it wrong; this is a legitimate use case for quickfix. It's designed to store things like grep or LSP references results, but the original use case was to get the output of make and store the locations of error and also all the other random stdout from the process. When errorformat can't parse anything, you get a quickfix with no filenames that starts with ||.
I have a pending PR to completely redo the display (#32) that I'm going to merge soon. I added a commit to that which will do mostly the same thing as this PR. Give it a try and see if it works as expected for you.

great, thank you! i started testing the new branch, I think a trailing space is added if there is no text in the output:

i have configured neovim to display trailing spaces, so it'll a little distracting. but this is already much better!

Great, I'll continue the development on that PR branch


Release pipeline variable group cleanup

Since changing the GitHub PAT we need to ensure that

All release pipelines use the new PAT and variable group
The old Variable Group is no longer needed and removed

Old VG's are cleaned up and we're only using the new PAT.
Still need to clean up instances of the PAT being used in lieu of an environment variable in some scripts

PAT instance cleanup is not a threat (we don't log the credentials there)... The larger discussion here centers around the plan for those few pipelines that have this.


Prevent a game (Fortnite) from reading the mouse input?

This is a great project, works really well. One issue I had was preventing the mouse input from being read by Fortnite. It seems to be capturing both (even after suppressing mouse events in pynput). Do you have a solution to this, or way to prevent the mouse input from being read (but still working with pynput listening to mouse). Sorry if that didn't make sense, thanks for the project!

Hi, I don't know how to prevent Windows from sending mouse input to Fortnite. Maybe this SO can help, but it certainly sounds painful as you need to code you own specific driver apparently.
(Closing the issue as it is not related to vgamepad)


Pause on load screen instead of afterwards

Nexus inquiry from https://www.nexusmods.com/skyrimspecialedition/users/83038368

is there a way in theory to make the game pause after finished loading, while staying on the loading screen, then continue after I pressed a certain button? I really want to read those texts

Suggestion from https://www.nexusmods.com/skyrimspecialedition/users/1074586

Might it be possible to just hook the function that displays the text? Either to pause, or to pause if a button is pressed?

delta feature request

It's possible to pause just after load and not on fast travel and doors?

For investigation

Won't fix this. Too much work, lots of other things to do.


Insufficient number of observations

Can you please suggest what values should be manually assigned for resolving the following error?

[17/10/2017 11:26:33] Sorting splice-graphs by read count
[17/10/2017 11:26:33] Finished sorting splice-graphs by read count
[17/10/2017 11:26:33] Spawning 1 thread(s) for fetching alignments and 1 i/o thread
[17/10/2017 11:26:33] Estimating fragment length distribution from 4 transcripts longer than 2500 nucleotides
ERROR: Insufficient number of observations for estimating the fragment length distribution parameters. Please specify these manually using the --frag-mean and --frag-sd options.

Hi,
Thank you for reporting the problem. Bayesembler requires at least 1000 paired-end reads that align to single-isoform genes for estimating the length distribution of the sequencing fragments. You can try any of the following:

If you are only running on a subset of your data, you should try running on the full set
If you already have an idea about the mean and standard deviation of the fragment length distribution you can provide these as input using the options suggested in the error message
If you have multiple samples that you think should have the same fragment length distribution, you could try to pool the data across samples and do a prerun of Bayesembler on the pooled data to get the estimates from the Bayesembler output and then provide these as input to runs on individual samples as described above.

I hope this helps! Best regards,
Lasse


feat(client/android): propagate structured errors from Go to TypeScript

Added detailed error propagation from Golang to TypeScript for Android Outline Client.
Current (Left) v.s. New (Right)

Tested for TCP&UDP: Go connectivity check result: TCPAndUDPConnectivityResult{TCPError:null,UDPError:null,}
Tested for TCP Only: Go connectivity check result: TCPAndUDPConnectivityResult{TCPError:null,UDPError:PlatformError{Code:ERR_PROXY_SERVER_UDP_NOT_SUPPORTED,Message:UDP connectivity check timed out,Cause:null,},}


Adding a method to generate VDB deck link

VDB links are made in a way that using card's id and count can generate a decklist.
This method uses this possibility to generate a deck link

Can you maybe add a test for this?


Work on the Excericise Manual for Module Reuse

@SimonRPSM Actually rather than manuals it might make sens to point them and make them use the developer guide? We could just point to what part of the developer guide they need.

Closing this for now. Let's discuss what the future DevGuide improvements will need to be.


Evaluating Claude / PaLM / etc.

Hi,
any reason this doesn't also evaluate other models like Claude, Palm, Llama 2 (via Replicate), etc.?
Happy to make a PR for the relevant code changes if that's a blocker

I've evaluated other LLMs like Claude, PaLM, etc. In evaluation, I used the following code to convert the multi-round message used for ChatGPT into a single round prompt:
prompt = ""
for turn in messages:
    if turn['role'] == 'system':
        prompt += f"{turn['content']}\n\n"
    elif turn['role'] == 'user':
        prompt += f"{turn['content']}\n\n"
prompt += "The ranking results of the 20 passages (only identifiers) is:"

I used the models to re-rank the top-20 BM25 passages on TREC DL-19, here are the results I got for reference:

Method
NDCG@1
NDCG@5
NDCG@10

OpenAI text-davinci-003
70.54
61.90
57.24

OpenAI gpt-3.5-turbo-0301
75.58
66.19
60.89

OpenAI gpt-4-0314
79.46
71.65
65.68

Cohere rerank-english-v2.0
79.46
71.56
64.78

Antropic claude-v1
74.81
65.47
60.25

Antropic claude-instant-v1
69.38
63.12
57.93

Google chat-bison-001
63.95
59.59
54.51

Google text-bison-001
69.77
64.46
58.67

Google bard-2023.06.07
68.60
61.62
56.27

Google FLAN-T5-XXL (11B)
52.71
51.63
50.26

Tsinghua ChatGLM-6B
54.26
52.77
50.58

LMSYS Vicuna-13B-v1.1
54.26
51.55
49.08

I haven't yet evaluated the latest LLMs like LLaMA-2, Claude-2, etc.
Since the APIs of the different LLMs vary significantly, I've conducted individual evaluations for each model. It would be great if the relevant code could be uploaded to help the use of LLMs other than ChatGPT :)


(bugsbash) remove redundant %s

Signed-off-by: aSquare14<EMAIL_ADDRESS>Pull Request template
Why is this PR required? What issue does it fix?:
the argument is already a string, there's no need to use fmt.Sprintf
What this PR does?:
Does this PR require any upgrade changes?:
If the changes in this PR are manually verified, list down the scenarios covered::
Any additional information for your reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs
Checklist:

[ ] Fixes #
[ ] PR Title follows the convention of  <type>(<scope>): <subject>
[ ] Has the change log section been updated?
[ ] Commit has unit tests
[ ] Commit has integration tests
[ ] (Optional) Are upgrade changes included in this PR? If not, mention the issue/PR to track:
[ ] (Optional) If documentation changes are required, which issue on https://github.com/openebs/openebs-docs is used to track them:

PLEASE REMOVE BELOW INFORMATION BEFORE SUBMITTING
The PR title message must follow convention:
<type>(<scope>): <subject>.
Where:
Most common types are:
* feat        - for new features, not a new feature for build script
* fix         - for bug fixes or improvements, not a fix for build script
* chore       - changes not related to production code
* docs        - changes related to documentation
* style       - formatting, missing semi colons, linting fix etc; no significant production code changes
* test        - adding missing tests, refactoring tests; no production code change
* refactor    - refactoring production code, eg. renaming a variable or function name, there should not be any significant production code changes
* cherry-pick - if PR is merged in master branch and raised to release branch(like v0.4.x)
IMPORTANT: Please review the CONTRIBUTING.md file for detailed contributing guidelines.

Codecov Report

Merging #54 (705516e) into develop (718fa08) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop      #54   +/-   ##
========================================
  Coverage    34.60%   34.60%
========================================
  Files           27       27
  Lines         2124     2124
========================================
  Hits           735      735
  Misses        1352     1352
  Partials        37       37

Impacted Files
Coverage Δ

cmd/provisioner-localpv/app/start.go
0.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 718fa08...705516e. Read the comment docs.

@niladrih Thanks for the review ! Done :)


[Document Translation] Fix Document Translation README.md links

[Document Translation] Fix Document Translation README.md links.  Two links are corrected.

API change check
API changes are not detected in this pull request.


[QST] GPU Memory is completely consumed in AWS-EMR

I am trying to run Spark-Rapids with AWS-EMR. I am facing a problem that the GPU memory is completely consumed when the processing is initiated. And there is no space left in the GPU memory to perform processing.
Attached is an image, showing this phenomenon.
The AWS-EMR version is 7.1.0.
I want to ask that how can I solve this problem ?

The issue was due to configuring the following parameter
"spark.plugins":"com.nvidia.spark.SQLPlugin"
Removing it, resolved the issue.

@akmalmasud96 just making sure I understand, do you want to run the the spark-rapids plugin? If so the spark.plugins flag is required. Thanks!

The RAPIDS Accelerator consumes almost all of the GPU memory by default.  It does not expect to share the GPU with another process.  You can configure the RAPIDS Accelerator to consume much less GPU memory, although that often has detrimental effects on performance due to extra spilling to fit within the smaller amount of GPU memory.  See the spark.rapids.memory.gpu.maxAllocFraction or spark.rapids.memory.gpu.reserve configs to limit the amount of memory used, either by setting a maximum fraction or the amount of memory to leave reserved, respectively.

Thanks for describing. I had figured it out. I want to ask that can we use Raft in spark-rapids for vector operations ?


[FLINK-28491][table-planner] Introduce APPROX_COUNT_DISTINCT aggregate function for batch sql

What is the purpose of the change
Introduce APPROX_COUNT_DISTINCT aggregate function for batch sql, see https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386481 for more details
Brief change log

Introduce HyperLogLogPlusPlus
Refactor AggFunctionTestBase: Allows its input type and its output type to be different
Introduce APPROX_COUNT_DISTINCT aggregate function for batch sql

Verifying this change
This change added tests and can be verified as follows:

Added HyperLogLogPlusPlusTest to verify the logic of HyperLogLogPlusPlus
Added ApproxCountDistinctAggFunctionTest to verify the logic of  ApproxCountDistinctAggFunctions
Extended SortAggregateTest  to verify the plan for APPROX_COUNT_DISTINCT queries
Extended SortAggITCase  to verify the result for APPROX_COUNT_DISTINCT queries

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Thanks for the review, I will rebase the master and squash the commit. I will merge the pr once it turns green

@flinkbot run azure


Support for unminified-webpack-plugin

How about compatability/support for unminified-webpack-plugin?

Hi! You mean to be able to generate unminified files in parallel?

@w0rse
Hi, yes sure, but with having webpack being run twice, just in one go.

unminified-webpack-plugin checks for the presence of the UglifyJsPlugin in the configuration with an instanceof test.
I worked around this by doing the following:
const UglifyJsParallelPlugin = require('webpack-uglify-parallel');
UglifyJsParallelPlugin.prototype = Object.assign(new webpack.optimize.UglifyJsPlugin(), UglifyJsParallelPlugin.prototype);

Looks a bit hacky :) We should work with unminified-webpack-plugin author to remove the strong check for UglifyJsPlugin.

Conceptually, extending UglifyJsPlugin makes sense. The difficulty is that its not designed/executed in a friendly way to support this. If they broke up the monolithic apply() into smaller pieces, you could easily reuse what they already offer without duplication. My workaround effectively overrides the bulk of the code of the original, but makes the type checking happy.


add strikethrough

Hi,
Could you add the strikethrough mode, which is \e[9m ?
Thanks

Sure

There you go! It's available in version 0.7.5.

Works like a charm.
Thanks a lot !


Add the Boffin

New character just dropped: https://wiki.bloodontheclocktower.com/Boffin
We're excited to play with it ASAP! I'd like to add an override so we don't have to wait for Pocket Grimoire or BotC Online to support it, similar to what you did for Lord of Typhon in https://github.com/tchajed/botc-tools/commit/021ccdb8bedde4a066c3db31a4c3007ba70f5719.

I was just about to do this work and then saw the pull request - thanks!


Custom fetch method

Adds support for a custom fetch method. This could be useful if you must communicate with the backend server in a different way than this link tool does by default.

Hey,
Any idea when this pull request is going to be merged?
I need to access a web server with authentication and structured response data wrapping the expected response.
would be really helpful!

+1

+1
Please merge it, cause we have the same approach for ImageTool also =)

This is a nice addition! Thanks

+1

+1

+1

Any news?


added LogWrapper to bridge indicatif and the log crate

this is a much nicer solution than https://github.com/console-rs/indicatif/issues/566 and https://github.com/console-rs/indicatif/pull/567 in my opinion.
this is behind the "log" feature for now.
we could consider making this a default feature, it barely adds any code, does add the log crate as dependency though.

To be honest, I'm not sure this belongs in indicatif. I think I'd rather see it as a separate crate, e.g. like the other loggers: https://docs.rs/log/latest/log/#available-logging-implementations. I don't have strong feelings though, so I could be convinced otherwise.

I agree. Would be happy to link to it from the README/documentation, though.

yeah i agree, did it here:
https://crates.io/crates/indicatif-log-bridge
https://github.com/djugei/indicatif-log-bridge

yeah i agree, did it here:
https://crates.io/crates/indicatif-log-bridge https://github.com/djugei/indicatif-log-bridge

Awesome, thanks! Feel free to submit a PR that edits the README to mention the crate.

allright i will, but first i gotta polish/add a bit of documentation to it


Rename execute to spawn

Brought up in #645 and #639. This should be considered as part of the 0.2 changes.

I don't disagree... but I don't think that there are any plans for a futures 0.2 release anytime soon.


Wrong behaviour of [Unreachable code]

Environment data

Language Server version: v2022.9.10
OS and version: Ubuntu 20.04
Python version (& distribution if applicable, e.g. Anaconda): 3.8.10

numpy version: 1.22.4

Issue description
Send np.ndarray type parameters to the np.cross() function can cause the unreachable code after the np.cross() function.
Code Snippet
import numpy as np

# Send np.ndarray type to the np.cross() function can cause the [unreachable code] after the np.cross() function.
x: np.ndarray = np.array([1, 2, 3])
y: np.ndarray = np.array([4, 5, 6])

# Correct behaviour after np.cross function.
# x = [1, 2, 3]
# y = [4, 5, 6]

res = np.cross(x, y)
# The code below is all unreachable!

print("Some code should be executed.")
print("Some code should be executed.")
print("Some code should be executed.")
print("Some code should be executed.")
print("Some code should be executed.")

Repro Steps
Just paste code to vscode.
Expected behavior

The code after np.cross() function should not be gray colored and get [unreachable code] warning.
Actual behavior
The code block after np.cross() function can be reached when execution. However, Pylance color them with [unreachable code color]

Logs

[Info  - 12:08:27 PM] (109463) Pylance language server 2022.9.10 (pyright aba41864) starting
[Info  - 12:08:27 PM] (109463) Server root directory: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist
[Info  - 12:08:27 PM] (109463) Starting service instance "scripts"
[Info  - 12:08:27 PM] (109463) Notebook support: LSP
[Info  - 12:08:27 PM] (109463) Interactive window support: LSP
[Info  - 12:08:27 PM] (109463) virtual workspace: untitled:Untitled-1.ipynb?jupyter-notebook
[Info  - 12:08:27 PM] (109463) Starting service instance "Untitled-1.ipynb"
[Info  - 12:08:27 PM] (109463) No pyproject.toml file found.
[Info  - 12:08:27 PM] (109463) Setting pythonPath for service "Untitled-1.ipynb": "/bin/python3.8"
[Warn  - 12:08:27 PM] (109463) stubPath typings is not a valid directory.
[Info  - 12:08:27 PM] (109463) Assuming Python version 3.8
[Info  - 12:08:27 PM] (109463) Assuming Python platform Linux
[Info  - 12:08:27 PM] (109463) Search paths for <default>
[Info  - 12:08:27 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib
[Info  - 12:08:27 PM] (109463)   typings
[Info  - 12:08:27 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stubs/...
[Info  - 12:08:27 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/bundled/stubs
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3.8
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3.8/lib-dynload
[Info  - 12:08:27 PM] (109463)   /home/cookteam/.local/lib/python3.8/site-packages
[Info  - 12:08:27 PM] (109463)   /usr/local/lib/python3.8/dist-packages
[Info  - 12:08:27 PM] (109463)   /usr/local/lib/python3.8/dist-packages/vrep
[Info  - 12:08:27 PM] (109463)   /usr/local/lib/python3.8/dist-packages/cookteam
[Info  - 12:08:27 PM] (109463)   /usr/local/lib/python3.8/dist-packages/cookteam/Calibration
[Info  - 12:08:27 PM] (109463)   /home/cookteam/Workspace/cookteam/idl
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3/dist-packages
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist/utils
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist/RTM_IDL
[Info  - 12:08:27 PM] (109463)   /usr/lib/python3/dist-packages/omniORB/COS
[Info  - 12:08:27 PM] (109463) Searching for source files
[Info  - 12:08:27 PM] (109463) No source files found.
[Info  - 12:08:27 PM] (109463) No configuration file found.
[Info  - 12:08:27 PM] (109463) No pyproject.toml file found.
[Info  - 12:08:27 PM] (109463) Setting pythonPath for service "scripts": "/bin/python3.8"
[Warn  - 12:08:27 PM] (109463) stubPath /home/cookteam/Workspace/cookteam/scripts/typings is not a valid directory.
[Info  - 12:08:27 PM] (109463) Assuming Python version 3.8
[Info  - 12:08:27 PM] (109463) Assuming Python platform Linux
[Info  - 12:08:27 PM] (109463) Search paths for /home/cookteam/Workspace/cookteam/scripts
[Info  - 12:08:28 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib
[Info  - 12:08:28 PM] (109463)   /home/cookteam/Workspace/cookteam/scripts
[Info  - 12:08:28 PM] (109463)   /home/cookteam/Workspace/cookteam/scripts/typings
[Info  - 12:08:28 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stubs/...
[Info  - 12:08:28 PM] (109463)   /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/bundled/stubs
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3.8
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3.8/lib-dynload
[Info  - 12:08:28 PM] (109463)   /home/cookteam/.local/lib/python3.8/site-packages
[Info  - 12:08:28 PM] (109463)   /usr/local/lib/python3.8/dist-packages
[Info  - 12:08:28 PM] (109463)   /usr/local/lib/python3.8/dist-packages/vrep
[Info  - 12:08:28 PM] (109463)   /home/cookteam/Workspace/share/scripts/EnvSetup
[Info  - 12:08:28 PM] (109463)   /home/cookteam/Workspace/share/scripts/EnvSetup/Calibration
[Info  - 12:08:28 PM] (109463)   /home/cookteam/Workspace/cookteam/idl
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3/dist-packages
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist/utils
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3/dist-packages/OpenRTM_aist/RTM_IDL
[Info  - 12:08:28 PM] (109463)   /usr/lib/python3/dist-packages/omniORB/COS
[Info  - 12:08:28 PM] (109463) Adding fs watcher for library directories:
 /usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/home/cookteam/.local/lib/python3.8/site-packages
/usr/local/lib/python3.8/dist-packages
/usr/local/lib/python3.8/dist-packages/vrep
/usr/local/lib/python3.8/dist-packages/cookteam
/usr/local/lib/python3.8/dist-packages/cookteam/Calibration
/home/cookteam/Workspace/cookteam/idl
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/OpenRTM_aist
/usr/lib/python3/dist-packages/OpenRTM_aist/utils
/usr/lib/python3/dist-packages/OpenRTM_aist/RTM_IDL
/usr/lib/python3/dist-packages/omniORB/COS
[Info  - 12:08:28 PM] (109463) Adding fs watcher for directories:
 /home/cookteam/Workspace/cookteam/scripts
[Info  - 12:08:28 PM] (109463) Searching for source files
[Info  - 12:08:28 PM] (109463) Found 47 source files
(109463) [IDX(FG)] index libraries /home/cookteam/Workspace/cookteam/scripts (index) ...
(109463) [IDX(FG)]   read stdlib indices (28ms)
(109463) [IDX(FG)] index libraries /home/cookteam/Workspace/cookteam/scripts (index) [succeed] (28ms)
(109463) [FG] analyzing: Untitled-1.ipynb:pylance-notebook-cell:W0sdW50aXRsZWQ=.py ...
(109463) [FG]   parsing: Untitled-1.ipynb:pylance-notebook-cell:W0sdW50aXRsZWQ=.py (23ms)
(109463) [FG]   parsing: /usr/local/lib/python3.8/dist-packages/IPython/display.py [fs read 0ms] (10ms)
(109463) [FG]   parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 3ms] (126ms)
(109463) [FG]   binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi (52ms)
(109463) [FG]   binding: /usr/local/lib/python3.8/dist-packages/IPython/display.py ...
(109463) [FG]     parsing: /usr/local/lib/python3.8/dist-packages/IPython/core/display.py [fs read 2ms] (99ms)
(109463) [FG]     binding: /usr/local/lib/python3.8/dist-packages/IPython/core/display.py (8ms)
(109463) [FG]     parsing: /usr/local/lib/python3.8/dist-packages/IPython/lib/display.py [fs read 0ms] (74ms)
(109463) [FG]     binding: /usr/local/lib/python3.8/dist-packages/IPython/lib/display.py (8ms)
(109463) [FG]   binding: /usr/local/lib/python3.8/dist-packages/IPython/display.py (192ms)
(109463) [FG]   binding: Untitled-1.ipynb:pylance-notebook-cell:W0sdW50aXRsZWQ=.py (0ms)
(109463) [FG]   checking: Untitled-1.ipynb:pylance-notebook-cell:W0sdW50aXRsZWQ=.py (1ms)
(109463) [FG] analyzing: Untitled-1.ipynb:pylance-notebook-cell:W0sdW50aXRsZWQ=.py (405ms)
[Info  - 12:08:28 PM] (109463) Background analysis(1) root directory: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist
[Info  - 12:08:28 PM] (109463) Background analysis(1) started
(109463) Background analysis message: setConfigOptions
(109463) Background analysis message: setImportResolver
(109463) Background analysis message: ensurePartialStubPackages
(109463) Background analysis message: setTrackedFiles
(109463) Background analysis message: markAllFilesDirty
(109463) Background analysis message: setFileOpened
(109463) Background analysis message: analyze
(109463) [BG(1)] analyzing: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py ...
(109463) [BG(1)]   parsing: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (27ms)
(109463) [BG(1)]   parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 2ms] (109ms)
(109463) [BG(1)]   binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi (29ms)
(109463) [BG(1)]   binding: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (1ms)
(109463) [BG(1)]   checking: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py ...
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/typing.pyi [fs read 1ms] (33ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/typing.pyi (10ms)
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/typing_extensions.pyi [fs read 0ms] (3ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/typing_extensions.pyi (1ms)
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 1ms] (24ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (2ms)
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/abc.pyi [fs read 1ms] (1ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/abc.pyi (1ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/__init__.pyi [fs read 0ms] (141ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/__init__.pyi (26ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/core/multiarray.pyi [fs read 0ms] (14ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/core/multiarray.pyi (3ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/__init__.py [fs read 0ms] (9ms)
[Info  - 12:08:28 PM] (109463) Could not resolve source for '._ufunc' in file '/home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/__init__.py'
[Info  - 12:08:28 PM] (109463)   Attempting to resolve relative import
[Info  - 12:08:28 PM] (109463)   Attempting to resolve using root path '/home/cookteam/.local/lib/python3.8/site-packages/numpy/typing'
[Info  - 12:08:28 PM] (109463)   Resolved import with file '/home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_ufunc.pyi'
[Info  - 12:08:28 PM] (109463)   Attempting to resolve using root path '/home/cookteam/.local/lib/python3.8/site-packages/numpy/typing'
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/__init__.py (2ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_dtype_like.py [fs read 0ms] (1ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_dtype_like.py (1ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_generic_alias.py [fs read 0ms] (4ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_generic_alias.py (2ms)
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/collections/abc.pyi [fs read 0ms] (0ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/collections/abc.pyi ...
(109463) [BG(1)]       parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/_collections_abc.pyi [fs read 0ms] (1ms)
(109463) [BG(1)]       binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/_collections_abc.pyi (1ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/collections/abc.pyi (2ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_shape.py [fs read 0ms] (0ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_shape.py (0ms)
(109463) [BG(1)]     parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/enum.pyi [fs read 0ms] (4ms)
(109463) [BG(1)]     binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/enum.pyi (7ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_array_like.py [fs read 0ms] (1ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_array_like.py (1ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_nested_sequence.py [fs read 0ms] (2ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/typing/_nested_sequence.py (0ms)
(109463) [BG(1)]     parsing: /home/cookteam/.local/lib/python3.8/site-packages/numpy/core/numeric.pyi [fs read 1ms] (5ms)
(109463) [BG(1)]     binding: /home/cookteam/.local/lib/python3.8/site-packages/numpy/core/numeric.pyi (2ms)
(109463) [BG(1)]   checking: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (400ms)
(109463) [BG(1)] analyzing: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (570ms)
(109463) Background analysis message: getSemanticTokens full
(109463) [BG(1)] getSemanticTokens full at /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (3ms)
(109463) Background analysis message: getSemanticTokens range
(109463) [BG(1)] getSemanticTokens range 0:0 - 18:0 at /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (2ms)
(109463) Background analysis message: resumeAnalysis
(109463) [FG] parsing: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (14ms)
(109463) [FG] parsing: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 1ms] (81ms)
(109463) [FG] binding: /home/cookteam/.vscode/extensions/ms-python.vscode-pylance-2022.9.10/dist/typeshed-fallback/stdlib/builtins.pyi (19ms)
(109463) [FG] binding: /home/cookteam/Workspace/cookteam/scripts/Research/Exp2/bug_report.py (13ms)

Duplicate of https://github.com/microsoft/pylance-release/issues/3277


Implement incrcov

Incremental covariance. See @stdlib/math/statistics/incr/variance.

Implemented in https://github.com/stdlib-js/stdlib/commit/10840405dbae34dab2ef245e614d0aefe3248b4c. Closing.


Disable styles empty related link URLs

Disable styles empty related link URLs.
Removals

Related links that referenced www addresses.

Changes

Moved link_list__disabled class to global context.
Related Links now disable styles links with empty URLs.
Re-generates CSS.

Testing

Related links section in /budget/ or the /the-bureau/leadership-calendar/

Review

@jimmynotjim
@sebworks
@KimberlyMunoz

Preview

Notes

Spun off from DF-2099, which includes an empty related link.

:+1:


C_beta_vector: altlocs

Describe the bug
Running
fig = add_vector_to_plot(g, fig, "c_beta_vector")
For some PDBs (i.e. 6rew, 7w9w) yields an error message because apparently some vectors are only 2D:
IndexError                                Traceback (most recent call last)
Input In [31], in <cell line: 10>()
      8 fig = plotly_protein_structure_graph(g, node_size_multiplier=1)
      9 fig = add_vector_to_plot(g, fig, "sidechain_vector", colour="red", scale=1.5)
---> 10 fig = add_vector_to_plot(g, fig, "c_beta_vector", colour="blue", scale=1.5)
     11 fig = add_vector_to_plot(g, fig, "sequence_neighbour_vector_n_to_c", colour="green", scale=1.5)
     12 fig

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/graphein/protein/visualisation.py:447, in add_vector_to_plot(g, fig, vector, scale, colour, width)
    440     x_edges.extend(
    441         [d["coords"][0], d["coords"][0] + d[vector][0] * scale, None]
    442     )
    443     y_edges.extend(
    444         [d["coords"][1], d["coords"][1] + d[vector][1] * scale, None]
    445     )
    446     z_edges.extend(
--> 447         [d["coords"][2], d["coords"][2] + d[vector][2] * scale, None]
    448     )
    449     edge_text.extend([None, f"{vector}", None])
    451 edge_trace = go.Scatter3d(
    452     x=x_edges,
    453     y=y_edges,
   (...)
    458     hoverinfo="text",
    459 )

IndexError: index 2 is out of bounds for axis 0 with size 2

To Reproduce
Steps to reproduce the behavior:
from graphein.protein.config import ProteinGraphConfig
from graphein.protein.graphs import construct_graph
from graphein.protein.features.nodes.geometry import add_sidechain_vector, add_beta_carbon_vector, add_sequence_neighbour_vector

config = ProteinGraphConfig()
g = construct_graph(config=config, pdb_code="6rew")

add_sidechain_vector(g) #  works
add_beta_carbon_vector(g) # works
add_sequence_neighbour_vector(g) # works

fig = plotly_protein_structure_graph(g, node_size_multiplier=1)
fig = add_vector_to_plot(g, fig, "sidechain_vector", colour="red", scale=1.5)  # works
fig = add_vector_to_plot(g, fig, "c_beta_vector", colour="blue", scale=1.5)  # does not work
fig = add_vector_to_plot(g, fig, "sequence_neighbour_vector_n_to_c", colour="green", scale=1.5) # works
fig

This yields the error message above.
Expected behavior
All PDBs and amino acids should be vectorizable (in 3D) and all vectors plottable.
I would expect all vectors to be 3D, with missing dimensions just having 0 in their corresponding index, e.g. (2, 3) --> (2,3,0)
Desktop (please complete the following information):

OS: Linux
Python Version: 3.8
Graphein Version  & how it was installed: 1.2.1, installed via Pip

Hi @avivko thanks for raising this issue. It looks like the vector computation functions aren't accounting for possible altlocs and therefore compute a vector for each possible CB position. I'll try to figure out an elegant solution for this. In the meantime, you can use something like pdbtools to clean up the structure and remove undesired altlocs.

We try to remove them in the graph construction. The trouble was that the cbeta function relied on the raw_pdb_df which we keep as metadata for traceability. I forgot we also compute a clean rgroup dataframe in construction so it makes much more sense to use that instead.
I just opened a PR and added some tests for your examples. Will get it merged in once tests pass :)

Now fixed in 1.3.0 :)


Feature/efgs push keys hangfire job

Added back the HF jobs with unit tests

I think 12 unit tests are failing.
Here you can investigate which ones:
https://github.com/folkehelseinstituttet/Fhi.Smittestopp.Backend/runs/1625815436?check_suite_focus=true
Put it for a review after you fix them.


Respond with total numbers of pages when do a response for pagination

In order to make a better search for frontend development the api response call for those calls which allows for pagination

for example an api call /api/units?numberPerPage=10&currentPage=5 should return

{
    "data": [ ... ]
    "totalNumberOfPages": 20
}

Done in the latest Pull Request


Add a warning about plugins and the agent in docker

This adds a really prominent warning about using the agent in docker and our plugins.
The issue at https://github.com/buildkite-plugins/docker-buildkite-plugin/issues/61 provides some more context.

@toolmantim I took some of your suggestions and included an example. I also shuffled things around a bit so that the caveats are around accessing docker within a build vs specifically plugins.

Those changes look awesome @lox!
I think the issues justify the amount of words/attention we're giving it, and that people running big buildkite/agent Docker setups (especially on platforms like k8s and things) need to be aware of the path-of-least-pain when setting it up… otherwise it'll be developers discovering the pain points months later when they try to use one of our Docker plugins.
It's also a big motivator to have a better answer! 😁

Yup, I agree @toolmantim. Anything else before we ship it?

Not from me, only @harrietgrace’s edits I think!

I updated a couple of little words, nothing major. Happy for it to go out! Awesome additions @lox 👏🏻 🚢🚀


Prevent MUI from optimizing class names in prod

I would like to prevent MUI from optimising classNames:

development: .MuiAppBar-root-12

production: `.c12``
documentation: https://material-ui-next.com/customization/css-in-js/#class-names

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior
With NODE_ENV=production MUI optimizes the names, which is cool, but I am using MUI with https://github.com/kriasoft/react-starter-kit and (it may be configuration noobness), but I am seeing the following code being generated. Class names in HTML optimised, but styles not optimised:

I've been through the documentation trying to figure out how to disable the class name optimisation but can't find anything. Should there be a configuration flag, or am I missing something? As a side note I'm doing this side project to cut some learning curve, so I may be missing something obvious.
Your Environment
current dependencies, is mostly the stuff from here https://github.com/kriasoft/react-starter-kit, except with an added material ui:
"dependencies": {
    "babel-polyfill": "7.0.0-beta.3",
    "bluebird": "^3.5.1",
    "body-parser": "^1.18.2",
    "classnames": "^2.2.5",
    "cookie-parser": "^1.4.3",
    "core-js": "^2.5.1",
    "express": "^4.16.2",
    "express-graphql": "^0.6.11",
    "express-jwt": "^5.3.0",
    "graphql": "^0.11.7",
    "history": "^4.7.2",
    "install": "^0.10.1",
    "isomorphic-style-loader": "^4.0.0",
    "jsonwebtoken": "^8.1.0",
    "material-ui": "^1.0.0-beta.18",
    "node-fetch": "^1.7.3",
    "normalize.css": "^7.0.0",
    "passport": "^0.4.0",
    "passport-facebook": "^2.1.1",
    "pretty-error": "^2.1.1",
    "prop-types": "^15.6.0",
    "query-string": "^5.0.1",
    "ramda": "^0.25.0",
    "react": "^16.0.0",
    "react-dom": "^16.0.0",
    "rss-parser": "^2.10.6",
    "sequelize": "^4.17.2",
    "serialize-javascript": "^1.3.0",
    "source-map-support": "^0.5.0",
    "universal-router": "^4.3.0",
    "whatwg-fetch": "^2.0.3",
    "yarn": "^1.2.1"
  }

Any help or direction would be great. And thanks for the hand.

Class names in HTML optimised, but styles not optimised

Wow, it's very weird! It's hard to tell what's going on without a reproduction repository.
Without, I can't help much. I would encourage looking why the NODE_ENV change.

how to disable the class name optimisation but can't find anything. Should there be a configuration flag, or am I missing something?

People can provide their own class name generator with the JssProvider component and createGenerateClassName  option of JSS.

@oliviertassinari ive got the opposite issue currently in my production build (also using the kriasoft starter kit) where the stylesheets have been optimized correctly, but the development classes are on the dom elements (see here). ive also followed the guide in the css-in-js docs almost to a t. any ideas?

in case anyone finds this in the future and you're experiencing something similar, see this issue in webpack

@mlg87 In the case of SSR, it means your server isn't configured for production, only the client is.


Compatibility update

Dropped:

Django 1.11, Python 2.7, 3.5 support.

Added:

Django 3.0, Python 3.8 support.

Blocked by https://github.com/jazzband/sorl-thumbnail/issues/601


impl test for ghash, polyval

implement ghash and polyval, relying on impl of mulx in #48.
Closes #42
Closes #43

pausing for tonight. Status:
tasks: implement ghash

do: impl ghash using mulx

[x] test bytereverse
[x] impl bytereverse
[x] tests for ghash
impl ghash
clean

do: polyval

tests for polyval
impl polyval
clean

comms
back

reduce repo unused codepaths - test/{gfmulint,ghash}
bug: ghash 256 bits returned

🤯

Updating the remaining scope of this PR:

incorporate kai comments from 48 into 49
merge main
impl tests for polyval
impl tests for ghash/polyval in terms of one another, as spceified in ietf spec

next:

how should we impl galois field mul in circom <- the biggest focus for thor right now


Cloud init fixes + retentionPolicy deprecation

Fixes cloud init lock related issues. The automated tests would regularly fail due to az CLI installation failure. The CLI failed due to DPKG lock being held by some other process when cloud init was running. Fixed by updated apt.conf in cloudinit to wait for lock.
retentionPolicyhave been deprecated on diagnostic settings. This was causing deployment failures. Removed those.

approved


Switch coverage badge from codeclimate to coveralls

Coverage remained the same at 100.0% when pulling e9575ed4cafc1f49023b4dbfc5e575c67ae5243d on fix/coveralls_badge into d15669fae9287ad34170247287a4e1b252f1a354 on master.

Coverage remained the same at 100.0% when pulling e9575ed4cafc1f49023b4dbfc5e575c67ae5243d on fix/coveralls_badge into d15669fae9287ad34170247287a4e1b252f1a354 on master.


pull request for e2e test at 12/30/2016 20:26:11

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


unable to use erase on mask layer for inpaint

whatever white has touched when drawing a mask stay's as a mask even after erasing.
to reproduce:
create mask layer
use white to create mask
remove some of white (with eraser)
https://imgur.com/a/EYEA8H0
hope this is clear (used black baground to show instead of origional image)
run inpaint
everything that white has touched is rendered

Oh yeah, I ran into that too but forgot about it :p


Update for 0.7.1 release

This fix gets rid of the error message coming up in 0.7.1 release.

I would love to see this shipped.

Thanks!

This is published 0.3.2


nixos/jellyfin: Fix NVENC, NVDEC acceleration

Fix NVENC, NVDEC usage when transcoding with hardware capabilities
Fixes https://github.com/NixOS/nixpkgs/issues/152008

Description of changes

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

@minijackson @purcell @jojosch @nyanloutre, can you review the changes please?

I tested it, it doesn't work, you need either
PrivateDevices = false;

or
DeviceAllow = pkgs.lib.mkForce [ "char-drm rw" "char-nvidia-frontend rw" "char-nvidia-uvm rw" ];

DeviceAllow is safer, but this whitelist might not be enough to make the other hardware accelerator work (OpenMAX OMX, AMD AMF, etc). I tested it with NVDEC/NVENC though, and it works perfectly.

I don't have a GPU in my jellyfin server to verify/test but the value of RestrictAddressFamilies and the removal of ProtectClock = true; now match the upstream service file.

This version still has PrivateDevice = true, unlike the upstream service file

Thanks for your feedback @D4Delta!
Strangely, this PR works fine for me. I would proceed with removing PrivateDevices so in this way we'll not have to whitelist devices as we go.

Thanks for your feedback @D4Delta!
Strangely, this PR works fine for me. I would proceed with removing PrivateDevices so in this way we'll not have to whitelist devices as we go.

Weird, how are you testing it? I personally have set NVENC the following way

[X] H264
[X] HEVC
[X] MPEG2
[X] MPEG4
[ ] VP8
[X] VP9
[X] Enable 10-bit hardware decoding for HEVC
[ ] Enable 10-bit hardware decoding for VP9
[ ] Enable enhanced NVDEC decoder
[ ] Enable hardware encoding
[ ] Enable Tone mapping

And then I try to open a random video, and when it doesn't work I get a "Playback Error" in jellyfin


Fix roll issue when a condition is checked & Implement ListOne feature. - [merged]

In GitLab by @s.paquay1 on Nov 22, 2023, 06:03
Merges master -> master
When the sheet has a list with condition = true and at least one condition is checked, the roll fail. I've fixed this issue.
I've also fixed issue #90, a deepClone method call.
I've re-added ListOne feature in this MR too and reviewed it.
I've also added basic tooltip on ListOne and ListMany options.
There's still an pre-existing issue with ListMany and ListOne:
When you delete one option of a ListMany or ListOne TOML entry, it doesn't update properly.
Let's say you have a ListMany with option1, option2 and option3, you remove option1, the sheet will display option2, option3, option3.
This is caused by the updateDocuments method call, this method receive what is updated, so if there was 3 options and you remove one, you'll get 2 options, and the updateDocuments method will only update the first 2 elements of the list and won't delete the remaining element.  (As it's not part of the updateDocuments, it doesn't replace the array of options)
I didn't find a easy way to fix it, but as it's not a new bug related to ListOne (only a pre-existing one), I've submitted the ListOne feature with this known issue.

In GitLab by @s.paquay1 on Nov 22, 2023, 07:00
added 1 commit
f7b1d934 - #90: Fix deepClone method wrongly named.
Compare with previous version

In GitLab by @s.paquay1 on Nov 22, 2023, 06:14
added 1 commit
ca7d0eb1 - #53: Implement ListOne TOML feature.
Compare with previous version

In GitLab by @s.paquay1 on Nov 22, 2023, 06:26
added 1 commit
799c38da - #67: Add basic tooltip on ListOne and ListMany options.
Compare with previous version

In GitLab by @mclemente on Nov 22, 2023, 12:21
marked this merge request as ready

In GitLab by @mclemente on Nov 22, 2023, 12:25
mentioned in commit 4d57a04ab55deee3ebb27a787af835cb5cee2a53


Made probability weighting in gibbs sampler dependent on parents prob…

…abilities instead of the children's
To be honest, I'm not 100% sure, but I think it makes more sense to use the parent nodes' probabilities for weighting the result as they values are used to look up the probability for the current node in the probability table.
Please correct me if I am wrong as I use this modified version for a university project.

Why would this be the right thing to do?  Also, it breaks the bayesian network unit tests and causes the Gibbs sampler to produce the wrong outputs.

As I said, I am not 100% sure about this. But why do we multiply it with the probability of the children when we want to create a probability for the node's state which depends on its parents' states?
What tool did you use to calculate the results you compare the the output to in the unit test? I'd like to give it a try.

The calculation depends on the parents as well as the children, the why is complicated.  You should google bayesian networks and read about d-separation.  I calculated the correct value with the junction tree algorithm.

Yes, as written in the "Edit" section, the source I used for the algorithm was updated an the error was fixed. The PDF I linked there also describes it like you implemented it.


Pause Audio if card is removed, Resume if swipe again.

I would like to pause the audio if the card is removed. If it's removed for longer than 1 Minute I would like to stop the playback. If the card is placed again on the reader, the playback should resume or if it was stopped it should restart.
I think this could be done by sending artificial card ids in the reader.py...
what do you think?

I did some hacking. I changed Reader.py (using exprimental as base, since I run a RM6300).
I added a counter to count the number of empty reads (reads without a card), of course I added a else: to the if read_byte == b'\x02':
There I check if the counter is equals 10, and set a card_id "stop". While below 10 I set a card_id "pause" and add one to the counter.
this causes that the card eval receives a stop or pause card - which i did configure in the rfid_trigger_play.conf.
It works as expected, but it's still a hack. What would be better way to implement such feature?

Hi @jebos
could you create a pull request agains the develop branch, so I can take a look at the code? Possibly the changes can be merged even though it's a edge case - and then be documented in the wiki?

I will check if I find a better way to implement it (less hacky) nevertheless you find my changes in my fork (Feature branch)

Hey, my Kids actually played around with it and decided they do not like this. We are now at a "Controll Card" Mechanism which is kind of changed to the Master Behavior in a way that they can keep the card on the reader and have dedicated pause  / stop cards, but pause wont toggle the playback. they seem happy, I close this request.


Code coverage

Add code coverage generation for E2E tests
Add code coverage generation for UI tests
Moved Cypress from client/src to client

Security Hotspots to be fixed


C++ infer

hi @geneing ! I used the C++ code for infer! but the speed is so slowly
mels shape: (80, 500)
take  times:53.759968996047974
Seven seconds of audio takes about 53 seconds！
my hparams

rnn_dims=256,
fc_dims=128,
sparsity_target=0.90,
How to optimize the params! thanks!

@maozhiqiang That's unexpectedly slow. On my computer (6 year old laptop) with the same hparams it runs a little slower than real time. Let's check a few things:

Did you run training long enough to prune the weights? Current code should print how pruning is progressing.
I'm assuming you ran convert_model to create a weight file.
Have you compiled the library with optimization? Without at least "-O2" optimization level the code will run very slowly. Surprisingly, "-O3" optimization produces a library that runs ~30% slower than with O2. I get best results with "-O2 -ffast-math"

Sorry about the lack of detailed instructions. I'll get it done...

hi @geneing ! Thank you for your reply. the training log is
epoch:1801, running loss:211.44811499118805, average loss:1.5547655514057945, current lr:0.0002390525520373693, num_pruned:530824 (0.9%)
and used convert_model to convert pth to .bin model
I used make for complied, how to used -O2 for complie
thank your

Run ccmake or cmake-gui . Switch to advanced mode ("t" in ccmake / a checkbox in cmake-gui). Find CMAKE_BUILD_TYPE entry and type RelWithDebInfo. Find CMAKE_CXX_FLAGS_RELWITHDEBINFO and edit to include -ffast-math flag.
You can also set build type from cmake command line: https://cmake.org/pipermail/cmake/2008-March/020347.html

I just add
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -Wall -O2")
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -Wall -O2")
to CmakeList.txt  and recomplied
and now speed
mels shape: (80, 500)
5.093661546707153
Seven seconds of audio takes about 5 seconds！
Is this correct? thank you ！

Sounds right. "Eigen3" library that I use, employs every templating trick to get best performance. When optimized, it's performance is excellent. In debug mode it is super inefficient.
A few more flags to play with "-ffast-math" "-march=native".

thank you @geneing

Make sure that input mel is correct - e.g. use one of the training set inputs.
It may be easier to debug using python library

sys.path.insert(0,'lib/build-src-RelDebInfo')  #use correct path to the shared library WaveRNNVocoder...so
import WaveRNNVocoder
mel = np.load(fname).T  #check that the first dimension is 80. Plot to check that it looks like correct mel
wav = vocoder.melToWav(mel) #plot to see what wav looks like

thank you!

I used synthsis.py the outputs is normal,but using the test_wavernnvocoder.py has not worked!
sample :
test_0_mel.npy_orig.wav.zip

@maozhiqiang Could you, please, attach the mel data you are using as an input. Then I can try to reproduce your problem.

@geneing thank you! test mels as follows
test_0_mel.npy.zip

@maozhiqiang Works for me with b2f5fc106.

test.zip
Commands:
import numpy as np
import librosa
import sys
sys.path.insert(0,'../WaveRNN-Pytorch/lib/build-src-RelDebInfo')
import WaveRNNVocoder
mel=np.load('eval/test_0_mel.npy')
vocoder=WaveRNNVocoder.Vocoder()
vocoder.loadWeights('../WaveRNN-Pytorch/model_outputs/model.bin')
wav=vocoder.melToWav(mel)
plot(wav)
librosa.output.write_wav('test.wav', wav, 16000)
The speech is a bit noisy and quiet. Cantonese?

@geneing thank you! my result is also noise, I saw the same code！ I don't know why!

I'm not sure how to help you. Here's the weight file I'm using.
model.bin.zip

thank you! I will try!

Hi! Thank you for this awesome work!
I successfully trained the Pytorch WaveRNN model:
input_type='bits',
bits=10,
rnn_dims=800,
fc_dims=256,
pad=2,
upsample_factors=(4, 4, 16),
compute_dims=128,
res_out_dims=64*2,
res_blocks=10

Now I am trying to run inference in CPU using the C++ library. I compiled the library and run the convert_model.py, but when I try to run inference I get Aborted (core dumped).
If I use the model weights you shared in the above comment it runs perfectly fine.
Anything I might've missed here?
Thanks for your help :)

@alexdemartos Would it be possible for you to obtain the stack trace when this error happens. You may have to recompile in debug mode and either run with gdb or open the core file. It will make it a lot easier to find the cause.

Hi @geneing ,
thanks for your fast response. Sorry, I am not very experienced with C++ code debugging. I compiled the library in debug mode but I don't really know how to get any detailed info. This is the error from the .so library:
MemoryError: std::bad_alloc
I tried to debug with gdb and the vocoder binary, but it crashes when loading the mel (npy) file (even with your model):
(gdb) run -w model.bin -m mels.npy
Starting program: /media/alex1/home/ubuntu/git/WaveRNN-Pytorch/library/debug/vocoder -w model-geneing.bin -m mels.npy
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

Nevertheless, I noticed loading your model gives the following details:
Loading: model.bin
Loading:Conv1d(80, 64, kernel_size=(5,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_runn
Loading:Conv1d(64, 64, kernel_size=(1,), stride=(1,))
Loading:Stretch2d()
Loading:Stretch2d()
Loading:Conv2d(1, 1, kernel_size=(1, 9), stride=(1, 1), padding=(0, 4),
Loading:Stretch2d()
Loading:Conv2d(1, 1, kernel_size=(1, 11), stride=(1, 1), padding=(0, 5),
Loading:Stretch2d()
Loading:Conv2d(1, 1, kernel_size=(1, 21), stride=(1, 1), padding=(0, 10)
Loading:Linear(in_features=112, out_features=128, bias=True)
Loading:GRU(128, 128, batch_first=True)
Loading:Linear(in_features=160, out_features=128, bias=True)
Loading:Linear(in_features=128, out_features=512, bias=True)

While loading mine, the last part of the model is not there:
Loading: checkpoints/model.bin
Loading:Conv1d(80, 128, kernel_size=(5,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
Loading:Conv1d(128, 128, kernel_size=(1,), stride=(1,), bias=False)
Loading:BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_runA
```

@geneing I tested out the model you have in model_outputs/model.bin.
It's taking me 9.5 seconds to generate 6 seconds of audio. Also the audio output quality is pretty poor
sample.wav

What should I expect the performance to be if the model was converted to use cuda?
How can I improve audio quality?

@acrosson The network in this repo is designed for best performance on CPU - low op count, branching and memory access optimized for pipelined processors. For best performance on GPU you would use something like WaveGlow - no branching, massive op count amortized over thousands of simple compute cores.
On my dinky old laptop, I can synthesize 9.1 sec of audio in 8.1 seconds on a single cpu core. There are some opportunities for further optimization still. Either your computer is even slower than mine, or there is something suboptimal with code optimization (correct -O and -march flags are important).

@geneing hi, may i ask how to use the C++interface，thank you

@maozhiqiang  how to export to C++ inference,thank you

Hi all,
When I run "python test_wavernnvocoder.py", I got this error:
ImportError: /WaveRNN-Pytorch/library/build/WaveRNNVocoder.so: undefined symbol: PyThread_tss_get
Is there anyone who can tell me how to fix it? Thank you very much!

@alexdemartos
hello,i met the same problem, i wanna ask how did you save that? thank you.

@geneing. Thanks for the excellent work.  But when i change the parameters in the hparams.py and run the C++ code . i met a erro
python3.7: /usr/include/eigen3/Eigen/src/Core/Product.h:97: Eigen::Product<Lhs, Rhs, Option>::Product(const Lhs&, const Rhs&) [with _Lhs = Eigen::Matrix<float, -1, -1, 1>; _Rhs = Eigen::Matrix<float, -1, -1, 1>; int Option = 0; Eigen::Product<Lhs, Rhs, Option>::Lhs = Eigen::Matrix<float, -1, -1, 1>; Eigen::Product<Lhs, Rhs, Option>::Rhs = Eigen::Matrix<float, -1, -1, 1>]: Assertion `lhs.cols() == rhs.rows() && "invalid matrix product" && "if you wanted a coeff-wise or a dot product use the respective explicit functions"' failed.
Aborted (core dumped)
here is the parameters, Is it the reason that parameters are too large, thank you
rnn_dims=600,
fc_dims=512,
pad=2,
upsample_factors=(4, 4, 16),
compute_dims=128,
res_out_dims=128,
res_blocks=10,

@geneing hello, i use a hparams like belows,i can get a good result, but the inference time which i take is about 8s, is there any method can speed up inference, thank you.
model parameters
rnn_dims=400,
fc_dims=256,
pad=2,
upsample_factors=(4, 5, 10),
compute_dims=64,
res_out_dims=32*2,
res_blocks=3,


Search fails any request that starts with "display: ___"

Works
https://caniuse.com/#search=display%3A
Incorrect results
https://caniuse.com/#search=display%3A f (Expected: display: flow-root)
https://caniuse.com/#search=display%3A c (Expected: display: contents)
https://caniuse.com/#search=display%3A r (Expected: display: run-in)
Including a colon breaks the search for unknown reasons. Can the search not decode more than one encoded literal from a URL?

Thanks for pointing this out, fixed now!


Add overview and description of the Frequenz Microgrid API

Typo found here:
wil -> will
https://github.com/frequenz-floss/frequenz-api-microgrid/blob/0922342b01afc0be3b8c02b1c6feef526725f493/proto/frequenz/api/microgrid/microgrid.proto#L137C54-L137C54

Typo found here:
wil -> will

Fixed

Rebased on latest upstream head.