Find a bug in SlicedByteBuf in 4.0.36.Final netty

When I invoked
public ByteBuf getBytes(int index, ByteBuffer dst)
It throws index out of bounds exception. There is a wrong logic to invoke its checkIndex0 method. I have to roll back the version to 4.0.23 in my pom.xml, then the issue gone.
Could you consider to adopt another version if 4.0.36 is not required? Thanks.
At the end, appreciate you for contributing this project to open source.

Hi Patrick,
LittleProxy-mitm don't require a specific Netty version. I have copied
most of the pom.xml from LittleProxy since it was intended to be
included or to be a module.
Whithin my application I use Netty 4.1.0.Final. It should be easy to add
a Netty dependency in your pom.xml too. Please try to reproduce the
problem with more recent versions up to 4.0.41.Final and post the results.
I'll have a closer look at this.
Regards Frank
Am 23.09.2016 um 10:37 schrieb patrickseptem:

When I invoked
|public ByteBuf getBytes(int index, ByteBuffer dst)|
It throws index out of bounds exception. There is a wrong logic to
invoke its checkIndex0 method. I have to roll back the version to 4.0.23
in my pom.xml, then the issue gone.
Could you consider to adopt another version if 4.0.36 is not required?
Thanks.
At the end, appreciate you for contributing this project to open source.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ganskef/LittleProxy-mitm/issues/22, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AKIGhsn1J-mOCAtxK1USxZN9VhxdhKiLks5qs4_YgaJpZM4KEvHm.

Hi Frank,
Just tried with other versions 4.0.37 & above, the bug is not fixed. Filed
an issue in https://github.com/netty/netty/issues/5864 also. Here I guess
you can add a known issue and suggest user downgrade the netty if they met
the same issue.
Thanks & Regards,
Patrick
On Fri, Sep 23, 2016 at 6:33 PM Frank Ganske<EMAIL_ADDRESS>wrote:

Hi Patrick,
LittleProxy-mitm don't require a specific Netty version. I have copied
most of the pom.xml from LittleProxy since it was intended to be
included or to be a module.
Whithin my application I use Netty 4.1.0.Final. It should be easy to add
a Netty dependency in your pom.xml too. Please try to reproduce the
problem with more recent versions up to 4.0.41.Final and post the results.
I'll have a closer look at this.
Regards Frank
Am 23.09.2016 um 10:37 schrieb patrickseptem:

When I invoked
|public ByteBuf getBytes(int index, ByteBuffer dst)|
It throws index out of bounds exception. There is a wrong logic to
invoke its checkIndex0 method. I have to roll back the version to 4.0.23
in my pom.xml, then the issue gone.
Could you consider to adopt another version if 4.0.36 is not required?
Thanks.
At the end, appreciate you for contributing this project to open source.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ganskef/LittleProxy-mitm/issues/22, or mute the
thread
<
https://github.com/notifications/unsubscribe-auth/AKIGhsn1J-mOCAtxK1USxZN9VhxdhKiLks5qs4_YgaJpZM4KEvHm
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ganskef/LittleProxy-mitm/issues/22#issuecomment-249157927,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADJplvMdfkPuPv6d0Ay3aUpTCiTqhsrjks5qs6rqgaJpZM4KEvHm
.

The issue is accepted by netty team as a defect.

Hi Patrick, thank you for pointing this out! So we're waiting for Netty 4.0.42.Final ...
@jekh please note, we should plan a LittleProxy/LittleProxy-mitm release after integrating and testing this fix, I think.

It is possible to use LittleProxy / LittleProxy-mitm with a more recent Netty 4.x version.


docs: show communication.png

Description
Fix README.md to show the image inline instead of a link.
Checklist

[ ] My code follows the style guidelines of the project
[ ] I have performed a self-review of my own code
[X] I have made corresponding changes to the documentation

:tada: This PR is included in version 1.7.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


1 feature 1 user input system

Doesn't look like this is ready for review, there are merge conflicts and the gradle build fails

/area ui

/close


feat: support dynamic import in script loader

close #18
and @crimx can you add me to the maintainer of this repo so I can push to this repo directly?

Thanks! I am thinking about transferring this repo to a organization. How about we name it awesome-webextension?

Thanks! I am thinking about transferring this repo to a organization. How about we name it awesome-webextension?

Cool

Do you receive the invitation?

Do you receive the invitation?

yes thanks!

Added.


Flip topology.pi-base.org to point to Cloudflare pages

Once all the other open items in the prerelease milestone are closed.

[x] Confirm issues here are resolved
[ ] Do a final UAT pass

[ ] Run Cypress tests against topology.pages.dev

[ ] Follow Cloudflare's directions for configuring DNS

🔬 Testing Notes

[ ] Ensure topology.pi-base.org has updated
[ ] Ensure topology.jdabbs.com legacy deep links (e.g. topology.jdabbs.com/spaces?text=plank) redirect as expected

I will do an initial preview for the /data maintainer community on June 2, and have them report any issues here if they find any.

We ended up not previewing it together live, but I've started a conversation at https://github.com/orgs/pi-base/discussions/321 for async feedback.

✅


Where is the server list ?

My Setup

ubuntu - gnome-shell (not the std ubuntu desktop)
snap

[x] I have tested with the latest version
[x] I can simulate the issue easily

Description
I'd like to connect my rocketchat client to an additionnal server. From the doc, i read that I should look for the server list, and click on add. That seems quite obvious, but I can't find the server list.
I check every menu I could find no luck.
Current Behavior
No server list visible.
Expected Behavior
Server list visible.

Please enable it using the menu View > Server list.

hi @h4wkmoon, please see how to on this print:
https://d.hullen.me/EZHdUV


Need an example of mentioning my docker host in a task.

@zeromsi here's one example:
https://github.com/tektoncd/pipeline/blob/50212455bfaa22165a887104e3f6f6c862a4213c/examples/v1beta1/taskruns/dind-sidecar.yaml#L1-L65
/kind question

hey @jerop thank you!

@jerop
i tried the above example but i am not able to mount a workspace folder as volume for the docker image. can you please help me on this.
exmplae:
docker run -v "$(workspaces.source.path):/workspace" --rm use-case-configuration-updater:latest
template -i /workspace/overlays/distributor-template -o /workspace/overlays -d /workspace/example-distributor.yaml
above command is failling


[Update Package] Microsoft/DotNet/SDK/3_1/3.1.422 new CDN urls

[Update Package] Microsoft/DotNet/SDK/3_1/3.1.422 new CDN urls
( #202037 )
Microsoft Reviewers: Open in CodeFlow

@wingetbot run


Could We consider using eslint-plugin-jsx-a11y?

It can be very helpful for enforcing people for writing accessible markup.
From README file:
Ryan Florence built out this awesome runtime-analysis tool called react-a11y. It is super useful. However, since you're probably already using linting in your project, this plugin comes for free and closer to the actual development process. Pairing this plugin with an editor lint plugin, you can bake accessibility standards into your application in real-time.
Since react-a11y it's deprecated in favor of react-axe, could we consider also, using react-axe?
Is there a preliminar discussion about it?

More from a11y topic, this week I ran a git grep aria-label just to see wrong usages and I created this gist: https://gist.github.com/diegonvs/9fb0a1deef30cfacb0583817f24a048b#file-just-aria-label-txt-L22
Is there a way to enforce devs for avoid using hard coded strings on aria-* attrs?

Could We consider using eslint-plugin-jsx-a11y?

I think you already answered your own question on that front:

react-a11y it's deprecated in favor of react-axe

So, yeah, we'd need to find an alternative. I am not aware of any existing integration of react-axe via eslint, but all I did was search for "eslint" in their repo and found nothing beyond a few internal usages.

Is there a preliminar discussion about it?

Couldn't find anything in the liferay-frontend-guidelines or liferay-npm-tools repo, or in this one.
It might be appropriate to create a ticket in the guidelines repo and discuss it there because the solution may or may not end up involving ESLint. Up to you though; the main thing is starting the conversation and you've already done that.

Closing in favor of liferay/liferay-frontend-guidelines#174


The default of buf size in "ble_gap_adv_set_fields" func too short to use (IDFGH-11798)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.2-beta1-263-ge49823f10c
Espressif SoC revision.
ESP
Operating System used.
Linux
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
None
Development Kit.
ESP
Power Supply used.
USB
What is the expected behavior?
I want to add  Manufacturer info in adv ,when i use mfg_data .it failed for short buf size
What is the actual behavior?
,when i change the buf size ,it lead to BLE_HCI_MAX_ADV_DATA_LEN ,but when i change it my project can not work ,always reboot.
Steps to reproduce.

Step
Step
Step
...

Debug Logs.
const char *mfgstr = "xxxxxx"
fields.mfg_data = (uint8_t *)mfgstr;
fields.mfg_data_len = strlen(mfgstr);
   rc = ble_gap_adv_set_fields(&fields);
    if (rc != 0) {
        MODLOG_DFLT(ERROR, "error setting advertisement data; rc=%d\n", rc);
        return;
    }
log: rc=4

More Information.
so how to add Manufacturer info in adv ?

E (1494) NimBLE: error setting advertisement data; rc=4
when it turn to maniufacturer set
   /*** 0xff - Manufacturer specific data. */     if ((adv_fields->mfg_data != NULL) && (adv_fields->mfg_data_len >= 2)) {         rc = ble_hs_adv_set_flat_mbuf(BLE_HS_ADV_TYPE_MFG_DATA,                                       adv_fields->mfg_data_len,                                       adv_fields->mfg_data,                                       dst, &dst_len_local, max_len, om);         if (rc != 0) {             return rc;         }     }
the left size  empty,so it went to BLE_HS_EMSGSIZE
   if (*dst_len + 2 + data_len > max_len) {         return BLE_HS_EMSGSIZE;     }

sry for close the issues

Hi @hibfh ,

I want to add Manufacturer info in adv ,when i use mfg_data .it failed for short buf size

Per spec, the maximum size of data that can be sent in legacy adv report is fixed. Existing example already sends details ( like name / flags ). You need to modify the complete adv data so that the final size fits as per spec recommendation.

ok i‘ll reduce some details


Fully transparent backgrounds have aliasing on text

Fully transparent:

Partial transparency (literally 1/255 alpha):

Looks like an edge case specifically for 0 alpha

Okay now I'm confused. I remember reading something about RGB information leaking into alpha channels or something somewhere. Think that's what's happening here. Text shows u8 background colors:


Support lists component

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Summary 💡
https://mui.com/material-ui/react-list/
As this will have sub-components. We should follow the same approach as we'll do in Cards component.
Examples 🌈

https://marmelab.com/ra-enterprise-demo/#/
Also check DX of retool, appsmith.
Motivation 🔦
No response

@prakhargupta1 so you want to be able to rebuild same design that is shown in example?

avatar
main text
helper text
numbers

Should each list item be constructed manually or should the list be built from array where I specify only how single list item looks like and the rest is automatically rendered?

Yes, this template should be fine. Instead of Avatar we can also show image. This way no new component need to added. So Image+Typography can be there by default.
Retool has Image+Rating component as the default sub-components and deleting image from 1 deletes from other entries as well. This experience looks good to me.
It should be like an array where defining one automatically renders others. By default, we can show 3 items.

I'll start working on this one if no one's picked it up already - will ask if I have any questions.

There's an interesting component in retool called "List View"
It allows you to set a number of items, and allows you to drag in other components, as a template which is repeated the same number of items.
I wonder if we could improve on it:

We allow you to template one item
Instead of setting a number, we allow you to assign an array, with data that can be used in the template

There's an interesting component in retool called "List View"
It allows you to set a number of items, and allows you to drag in other components, as a template which is repeated the same number of items.
I wonder if we could improve on it:

We allow you to template one item
Instead of setting a number, we allow you to assign an array, with data that can be used in the template

I was probably thinking of something like that, just passing in an array of objects with all the same keys, and build the list view from that data.
Also will probably try to make our component consistent with the API for the MUI List component as much as possible.

I agree, I think the user flow should be similar to data grid - where one connects data source and then can somehow (ideally via drag and drop) define "template" which will be rendered


App behavior when outside of a region is broken

From a user:

Here's a fun one. When I activated the app in Shanghai, it first showed me a map, alerted that I was out of range, and after choosing my "region" went back to what I think it intended as the map screen. Only it was entirely black.

Fixed as of #1019


Flow: Unsteady Flow Crash

On OSX, I experience a crash with the following sequence:
Dataset: DUKU
/glade/p/cisl/vast/vapor/data/VDC/vapor-3.0.0/WRF/DUKU/DUKU-3.0.0.nc

Create a Flow Renderer
Go to timestep 3
Enable unsteady flow
apply the Uniform/Non-Random rake
Change the Z dimensionality of the uniform rake

Note on 9/10/2019:
This crash seems to be related to the crash that's reported in #1848 . Will investigate later.


Routed back to /login even with a token

I'm finding myself in the situation that if I restart the server, then go back to <IP_ADDRESS>:9000 , I always get routed back to the login page, even if I have a token. Looking into the issue, it appears that the lines to blame are in client/app/modules/core/controllers/router.ctrl.js, more precisely,
    ApiService.checkConnection()
      .then(function() {
        console.log('ApiService.checkConnection success');
        if (!AppAuth.currentUser) {
          $location.path('/login');
        } else {
          $location.path('/app');
        }
      })

For some reason, at this point, AppAuth.currentUser has not been set, so we go into the first part of the if statement. Is there any particular reason why not to use LoopBackAuth here instead of AppAuth? If I change the condition to
if (!LoopBackAuth.currentUserId)

I get the desired effect; routed to /app if I have a token, otherwise, taken to /login .

This issue should be fixed. If not, please reopen with the steps to reproduce.


Release a stable v1.0

I think once we've had a few weeks of people using 0.4.0 and we haven't had any major issues, we can promote that up to 1.0.0.
@obj63mc, what do you think? You're largely already using it, could you upgrade your sites to use 0.4.0 without any forks and see if it works for you? Same goes for anyone else too - it'd be great to get multiple real-world tests of 0.4.0 before we make something stable, especially given 1.0.0 will be the first properly semantic version release and we can't as easily making breaking API changes after that.

Can do. Will pull that down in the morning to a few of our sites and let you know of any issues. Right now I believe everything is exactly the same as my fork so we should be good.

 If all is well, shall I release a v1.0?

No issues for us. Using in production for 3 sites 2 newly launched this month and another launching next and all is working well. Feel free to make it 1.0

Fantastic, thanks! So this will transition the module into being semantically versioned, so from now any changes we make need to not break backwards compatibility (unless they're made on a new version as v2 etc).
Will create the new tag now.
Thanks @obj63mc for your testing!

Version 1.0.0 has been released 🎉


CLI: allow user to add multiple nodes to deploy workers on

Is your feature request related to a problem? Please describe
grid cli allow the user to enter only one node to deploy workers on
Describe the solution you'd like
user should be able to add node id for every worker

added k8s support of working on multiple nodes:

nodes given are assigned to workers
if no. of nodes is less than workers, the remaining workers will be assigned randomly
Extra nodes are skipped

Tested on :

deploying the cluster with multiple workers with no nodes
deploying the cluster with multiple workers with number of nodes equal to the number of workers
deploying the cluster with multiple workers with number of nodes that's less than the number of workers
deploying the cluster with multiple workers with number of nodes that's more than the number of workers
deploying the cluster with no workers
deploying the cluster with multiple workers on same node
deploying the cluster with workers on same node as master node


Torch installation in a self-contained folder for windows with msvc

Hi,
This PR is to get torch installed in a self-contained folder for windows with msvc. The procedure is like the counter party for *nix. Instructions are in README
It works relying on several patches  that I think are not best solutions, so I did not send PRs for them.

cunn, cudnn patches are to load THC.dll explicitly instead of using ffi.C directly for windows due to different ffi.load behaviors for different platforms. I think a better solution could be having cutorch.C defined in cutorch to load THC.dll for windows and to refer to ffi.C for *.nix.
cudnn is using LongTensor for size_t which does not work for windows since long is 32bit on windows. The patch does not contain LongTensor replacement, so cudnn can only partially work. I think a better solution is to improve torch integer types so that LongTensor uses int64 for all platforms.

Limitations for msvc users:

Latest torch git sources are not guaranteed to not break msvc compilation since windows is not officially supported. Have to stick with distro stable version.
Type long is used extensively in torch with the assumption that it is 64 bit while long is 32 bit on windows. Be careful with LongTensor.
Only the luajit version can work, lua + luaffifb currently does not work since msvc luaffifb does not work for float argument functions.
Some torch packages in torch luarocks repository can not be installed for windows with 'luarocks install' command, though many requires trivial changes to work on windows.

Followups:

To cleanup the other version of windows installation files.

Great thanks to @howard0su 's review and evaluation.
Thanks,

thanks!


Extend DSL metadata

Currently, the /v1/capability endpoint returns the following:
{
  "dsls": [
    {
      "output": "true",
      "input": "true",
      "validationSchema": "/v1/capabilities/Integration/schema",
      "deployable": "true",
      "name": "Integration",
      "description": "An Integration defines a workflow of actions and steps.",
      "step-kinds": "[CAMEL-CONNECTOR, EIP, EIP-BRANCH]"
    }
  ]
}

Full response
{
  "dsls": [
    {
      "output": "true",
      "input": "true",
      "validationSchema": "/v1/capabilities/Integration/schema",
      "deployable": "true",
      "name": "Integration",
      "description": "An Integration defines a workflow of actions and steps.",
      "step-kinds": "[CAMEL-CONNECTOR, EIP, EIP-BRANCH]"
    },
    {
      "output": "true",
      "input": "true",
      "validationSchema": "/v1/capabilities/Camel Route/schema",
      "deployable": "false",
      "name": "Camel Route",
      "description": "A camel route is a non deployable in cluster workflow of actions and steps.",
      "step-kinds": "[CAMEL-CONNECTOR, EIP, EIP-BRANCH]"
    },
    {
      "output": "true",
      "input": "true",
      "validationSchema": "/v1/capabilities/Kamelet/schema",
      "deployable": "true",
      "name": "Kamelet",
      "description": "A Kamelet is a snippet of a route. It defines meta building blocks or steps that can be reused on integrations.",
      "step-kinds": "[CAMEL-CONNECTOR, EIP, EIP-BRANCH]"
    },
    {
      "output": "true",
      "input": "true",
      "default": "true",
      "validationSchema": "/v1/capabilities/KameletBinding/schema",
      "deployable": "true",
      "name": "KameletBinding",
      "description": "Kamelet Bindings are used to create simple integrations that link a start step to an end step with optional intermediate action steps.",
      "step-kinds": "[KAMELET, KNATIVE]"
    }
  ]
}

The goal for this issue is to enhance the existing model to provide new properties:

Property
Type
Description

supportsMultipleFlows
boolean
Signals whether the current DSL supports creating multiple flows in the same canvas or not

stepsName
string
A name to be used in the catalog

[:nail_care:][:nail_care:][:nail_care:]  In addition to that, it would be useful to avoid using dash in the property names if possible, otherwise this gets slightly inconvenient while using them in the UI :smile:

Adding the change step-kinds to stepKinds too.


Brightness profiles dropping to zero are spoiling the stacks produced by XGA

This may be a bug, or a consequence of the data. Either way I have to explore and explain what is going on here (for instance):

A first step to mitigate this would be to add a flag to the stacking function that allows it to throw away any profiles that do go to zero - but this doesn't seem like a particularly nice way of doing it in the long term.
I will also be looking at clusters where this happens, and trying to figure out why.

To be clear, I don't think these are the only reason the stacks are not very good, Kathy and I both think there might be some more fundamental problems with the method, but this is a good place to start.

I suspect that this has something to do with background values I'm measuring, Paul also agrees. I may not be removing edge pixels in the brightness profile generation, and for the R_200 profiles the background values are gonna be right on the edge of the field of view.

I've merged the branch dealing with this into master, and am closing this thread. I have no doubt that this will still happen, but its probably got a lot to do with a) expecting too much from the data I have, and b) my unsophisticated background techniques. Neither of which I am going to be able to solve in the scope of this issue


The check for access token completed on the UI front

What type of PR is this?

/kind bug

What does this PR do / why we need it:
Git-host access token requires that the commit-status-tracker to be true, however on the UI side we get an error if the commit-status-tracker option is no. Hence the fix.

Codecov Report

Merging #46 into master will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #46      +/-   ##
==========================================
+ Coverage   61.26%   61.36%   +0.09%
==========================================
  Files          52       52
  Lines        2476     2464      -12
==========================================
- Hits         1517     1512       -5
+ Misses        825      818       -7
  Partials      134      134

Flag
Coverage Δ

#unittests
61.36% <100.00%> (+0.09%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/cmd/bootstrap.go
31.57% <100.00%> (-0.35%)
:arrow_down:

pkg/cmd/ui/validate.go
42.10% <0.00%> (-1.14%)
:arrow_down:

pkg/cmd/service/add.go
70.00% <0.00%> (ø)

pkg/cmd/webhook/list.go
30.43% <0.00%> (ø)

pkg/cmd/webhook/delete.go
33.33% <0.00%> (ø)

pkg/cmd/service/service.go
0.00% <0.00%> (ø)

pkg/pipelines/secrets/secrets.go
47.56% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87d4093...51d1186. Read the comment docs.

/lgtm

/lgtm

/approve


position issue

I got the player postion through mc.player.getTilePos() ,then I press the F3 to check the postion, I found they are different, Why does this happen ? I suppose they should be the same. Pls help on this. Thanks a lot.

This is by design as locations are relative to the spawn point, this is to emulate the original behaviour of the API created for the Raspberry Pi.
You can change this to use absolute positions in the config.yml of the RaspberryJuice plugin. https://github.com/zhuowei/RaspberryJuice#config


Create lab instructions to use PodMan

...using the k3s VM. Preference would be doing this on their laptop.

brew install podman
podman machine init
podman machine start
sudo mkdir /usr/local/bin
sudo ln -s /opt/homebrew/bin/podman /usr/local/bin/docker

This get you further but still results in an error:
time="2022-08-11T11:36:27-05:00" level=warning msg="SHELL is not supported for OCI image format, [/bin/bash -o pipefail -c] will be ignored. Must use `docker` format"
--> f36d5bdd179
[8/14] STEP 6/6: COPY --link --chown=101:0 LICENSE /licenses/
[7/14] STEP 1/4: FROM c56f65de680b0be71b86217b4ec069d1ec755714a5bc82e1d809d15f53c36513 AS debian-plus-nap
[7/14] STEP 2/4: ARG NGINX_PLUS_VERSION
Error: server probably quit: unexpected EOF
make: *** [debian-image-nap-dos-plus] Error 125

Added the podman helper:
sudo /opt/homebrew/Cellar/podman/4.2.0/bin/podman-mac-helper install
	podman machine stop; podman machine start

Dan also recommended changing the make file to resolve this error:
make debian-image-nap-dos-plus PREFIX=ghcr.io/$GITHUB_USER/nginx-plus-ingress TARGET=container DOCKER_BUILD_OPTIONS="--platform linux/amd64 --format docker"

Which made it further but resulted in the following error:
--> 0ec75b54b2d
[8/14] STEP 6/6: COPY --link --chown=101:0 LICENSE /licenses/
[7/14] STEP 1/4: FROM ad6ba6d8b24278ffecf0977e9ad1b33fbc6c8cb92877af4057b403a6d5620f9b AS debian-plus-nap
[7/14] STEP 2/4: ARG NGINX_PLUS_VERSION
Error: server probably quit: unexpected EOF
make: *** [debian-image-nap-dos-plus] Error 125

The right approach (to avoid hacks) is that the ingress controller's build scripts need to support Podman natively. This has been requested in this issue however, this may not happen anytime soon. May revisit when that occurs, but will mark as wontfix for now.


PasswordCompareAsync prevents timeouts on resource constraint devices

Closes #982
Changed to faster brcypt implementation (brcyptjs) for better performance and made the compare call async so it does not timeout anymore. The check still takes ages on resource constraint devices as on the older models of the raspberry pi but it is usable now, you just need a little patience ;)
Performance comparison of the different bcrypt implementations can be found here: https://github.com/thelounge/lounge/pull/983

FYI I'll give this one more try before merging.


Add MIBI and mibi_deepcell

Adds a definition for Multiplex Ion Beam Imaging to assay_types, plus mibi_deepcell, a derived assay.

The string MIBI (in uppercase) should be covered by the "main name", which is the string at the top of the YAML block.

@jswelling can you please sync the new changes from test-release branch into your branch to fix the Travis build error? Your changes didn’t cause the build error.

@yuanzhou I merged the HEAD of test-release, but the tests are still failing.

@jswelling the error from the Travis CI build is different from what I had addressed. It looks like the portal transformation complained: https://app.travis-ci.com/github/hubmapconsortium/search-api/builds/245458219

@yuanzhou the new problem was caused by tabs masquerading as spaces in a yaml file- I've fixed it, and the tests passed.


Bump appveyor versions for numpy, pandas, python

[ ] Tests added / passed
[ ] Passes flake8 dask

I'm surprised by the appveyor failure here.  Does anyone see what I'm doing wrong?

Tried without luck
On Fri, Oct 12, 2018 at 11:15 AM Tom Augspurger<EMAIL_ADDRESS>wrote:

@TomAugspurger commented on this pull request.
In appveyor.yml
https://github.com/dask/dask/pull/4083#discussion_r224819786:

   ARCH: "64"

 NUMPY: "1.12.1"

 PANDAS: "0.19.2"

 NUMPY: "1.15"

Maybe try exact versions like 1.15.2 and 0.23.4?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/4083#pullrequestreview-164288074, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszOfSKwvZNt0nEzeu922nXXcqfd3nks5ukLICgaJpZM4XYXy5
.

There are not compatible numpy and pandas versions.
PackageNotFoundError: Package not found: '' Packages missing in current win-64 channels:
  - numpy 1.15
  - pandas 0.23

ref: https://ci.appveyor.com/project/daskdev/dask/builds/19449313#L146

Yeah, I saw the log.  I was confused though, because there are clearly files available:
https://anaconda.org/anaconda/numpy/files
There are clearly files there.  I'll try specifying explicit versions again and push that so we can test both ways.

PackageNotFoundError: Package not found: '' Packages missing in current win-64 channels:
  - numpy 1.15.2
  - pandas 0.23.4

That's strange, since they're both there in https://repo.continuum.io/pkgs/main/win-64/

Yup.
On Fri, Oct 12, 2018 at 2:01 PM Tom Augspurger<EMAIL_ADDRESS>wrote:

That's strange, since they're both there in
https://repo.continuum.io/pkgs/main/win-64/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/4083#issuecomment-429410262, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszBc-cFAzfC3iHnGLnQJ7J165legDks5ukNkUgaJpZM4XYXy5
.


Add checks for encrypted keys and pulumi creds to setup.debug

What does this PR do?
Add more checks to inv setup.debug

Ensures pulumi is logged in
if privateKeyPath is configured, ensures it is not encrypted or a proper privateKeyPassword is provided

Fail inv.setup if pulumi cannot be executed, e.g. not on PATH
Which scenarios this will impact?
Motivation
Additional Notes

/merge


Upload plugin

Hello. First of all, very good job with this project. 👍
I'm trying to make use of upload plugin. For some reason it does not work in my case. The problem is I am not getting any response so have no idea what went wrong. If I use default options I am not even getting an 'error' message. If I use imgur as in your example it works. I get 'error' only if I do not echo anything from serverPath.
I believe this upload plugin is very important and deserves some documentation.

Hi, I just used it and managed to make it work.
First you need to manually copy the file using $_FILES global and after you processed it you need to output a json response like echo json_encode($data) where $data is an array with a success variable and another variable called file with the path to the file.
If you want you can use different names but you need to change these values in the js of the plugin.
( you can find them in the defaultOptions ).
Hope this helps you :)

Duplicate of #205


Input comments contains spelling mistake "Deafault" (cosmetic)

Input comments include multiple references to "Deafault" that should be "Default".

Thank you! It will be fixed.

Fixed in 2.28.


Use local date format for pickup dates

Currently it uses the German format: 20.01.2016
That's not very localized.
Three options:

Should detect the local format of the user (via the browser)
should use the same as the angular-translate locale
just use ISO8601 (2016-01-20) for simplicity

I'm against using ISO8601. My favorite would be defining it in our angular translate files [I think your number 2]

That may be possible, but I don't know how...

resolved with #300


SqlAlchemy POC

This is a very early work in progress, and not intended for review. The PR is intended to show the direction that the SQLAlchemy refactor is going in, and hopefully get some early feedback from those that are interested.
The main idea is that we pull out our clunky object model for all of the objects in our registry, and use the SQLAlchemy library to do the heavy work for us.
What works

Rough outline model of Datasets, ReferenceSets and References.
Polymorphism within the DB for HtslibReferenceSets and SimulatedReferenceSets. We can store either in the DB, and we instantiate the correct object at run time when we need to access information for a specific information. This is the key requirement for more flexible storage backends.
Search and GET requests for references, referencesets and datasets. The paging algorithm has been reworked to use SQL natively, and is much cleaner.
Native support for multiple databases for the registry. I've tested this on Sqlite and Postgres and it works great on both.
Updated the repo manager CLI to use the new interface for the subset of data types supported. The repo manager now takes a DB URL in the form of a SQLAlchemy connection string.
Online updates to the DB. Updates to the registry from the CLI are now reflected in the server without needing a server restart.

TODO

A fair bit of refactoring and shuffling around of code.
Complete the rest of the data model (ReadGroupSets, VariantSets, etc).
Fix test cases.
Add some way of testing the postgres support using Travis.

Questions

Do we really need to worry about ID collisions between tables? I.e., does it actually matter if we have a  Dataset and a VariantSet with the same ID? I don't think we need uniqueness to fulfill the requirements of the server, and it adds a fair bit of complexity ensuring that they are unique.
How should we model things like ReferenceSet.source_accesssions? At the moment we embed some JSON into the table and don't worry about it. However, the right thing to do here would seem to be to model it with a table. It would surely be more forward-compatible to do this, if we ever need to get clever about accession numbers.
More generally, do we go for a full-on SQL model of all objects (like Experiment, VariantSetMetadata, etc), or do we continue using embedded JSON stored in a table?

Reactions:

If we are going to store data as well have registry database,
would a document store make more sense?  Then again, postgress
has a document store facility:
- https://blog.andyet.com/2016/02/04/postgres-9.5-document-store-hybrid/

If we are storing GA4GH record in blobs, it would be good to
research storing the binary protobuf.  This would reduce the
size and might be faster to decode into memory.

We need to have a design that handles being a REST front-end
to the UCSC Browser database.  The UCSC Browser database will
not map well to SqlAlchemy, as it mostly highly denormalized
and doesn't use foreign keys.  It's closer to htslib in SQL
than a true relational database.  It's doubtful that we can
afford to do an ETL of the subset of the data that can be
represented in GA4GH.   SqlAlchemy can't be the abstraction
layer if we are going to support this.

We need to know the goals before we decided on the
implementation.
Jerome Kelleher<EMAIL_ADDRESS>writes:

This is a very early work in progress, and not intended for review. The PR is
intended to show the direction that the SQLAlchemy refactor is going in, and
hopefully get some early feedback from those that are interested.
The main idea is that we pull out our clunky object model for all of the
objects in our registry, and use the SQLAlchemy library to do the heavy work
for us.
What works
• Rough outline model of Datasets, ReferenceSets and References.
• Polymorphism within the DB for HtslibReferenceSets and
SimulatedReferenceSets. We can store either in the DB, and we instantiate
the correct object at run time when we need to access information for a
specific information. This is the key requirement for more flexible storage
backends.
• Search and GET requests for references, referencesets and datasets. The
paging algorithm has been reworked to use SQL natively, and is much
cleaner.
• Native support for multiple databases for the registry. I've tested this on
Sqlite and Postgres and it works great on both.
• Updated the repo manager CLI to use the new interface for the subset of
data types supported. The repo manager now takes a DB URL in the form of a
SQLAlchemy connection string.
• Online updates to the DB. Updates to the registry from the CLI are now
reflected in the server without needing a server restart.
TODO
• A fair bit of refactoring and shuffling around of code.
• Complete the rest of the data model (ReadGroupSets, VariantSets, etc).
• Fix test cases.
• Add some way of testing the postgres support using Travis.
Questions
• Do we really need to worry about ID collisions between tables? I.e., does
it actually matter if we have a Dataset and a VariantSet with the same ID?
I don't think we need uniqueness to fulfill the requirements of the server,
and it adds a fair bit of complexity ensuring that they are unique.
• How should we model things like ReferenceSet.source_accesssions? At the
moment we embed some JSON into the table and don't worry about it. However,
the right thing to do here would seem to be to model it with a table. It
would surely be more forward-compatible to do this, if we ever need to get
clever about accession numbers.
• More generally, do we go for a full-on SQL model of all objects (like
Experiment, VariantSetMetadata, etc), or do we continue using embedded JSON
stored in a table?
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
You can view, comment on, or merge this pull request online at:
https://github.com/ga4gh/server/pull/1259
Commit Summary
• Simple initial modelling work.
• Simple initial modelling work.
• Basic implementaion of references.
• Rough working queries for references API.
• Verified works on postgres.
• Modified ID generation.
• Added constraints and checks.
• Merge branch 'sql-alchemy-poc' of github.com:jeromekelleher/server into
sql-alchemy-poc
File Changes
• M ga4gh/backend.py (123)
• M ga4gh/cli.py (141)
• M ga4gh/datamodel/init.py (12)
• M ga4gh/datamodel/datasets.py (25)
• M ga4gh/datamodel/references.py (209)
• M ga4gh/datarepo.py (163)
• M ga4gh/frontend.py (40)
• A sqlalchemy-demo.py (25)
Patch Links:
• https://github.com/ga4gh/server/pull/1259.patch
• https://github.com/ga4gh/server/pull/1259.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.*

One thing that is worth thinking about is that Postgres (and SQLAlchemy) support JSON fields so you can dump python dicts into the field and when you pull them out they are proper dictionaries. Plus you can index into them which is handy.
I think the big trade-off is that the datamodel is no longer explicit so if you are joining on them, you lose a lot of the performance that relational dbs offer in query planning. It's probably not a huge deal for references directly since I doubt references will ever number more than a few hundred/thousand (though is could be a concern if query via the accession for other models). Also lost is the ability to know what attributes define the object and which are required and which are optional. Thus, the code working with this data needs to be robust.
So I would say favor tables for object with defined set of keys and where the values would be used in queries, but in cases where it is strictly metadata storage dump it into a json field.
Overall, I'm a big fan of this, but don't know enough about the UCSC Browser schema that @diekhans mentioned to know what the overhead of the transform would be.

Thanks for the feedback @diekhans and @andrewjesaitis, this is very helpful.
@andrewjesaitis, I tend to agree with you. I think we should lean towards using tables where possible in the interest of forward compatibility as well. We may not have queries on parts of the data model at the moment, but hopefully we will have them in the future. It'll be a lot easier to accommodate this if we already have the information we need in tables.
@diekhans One thing that I should clear up here is that SqlAlchemy is only for the registry. The actual implementations of the data sources for the particular objects (e.g. VariantSet) is separate. This is an important separation of concerns for me; we a well defined SQL based approach to manage the high-level data model, as this is what it excels at. For bulk data storage, there's any number of other things we can use. I don't think we should try to do both (metadata/data model management and bulk data storage) at the same time.
In this new approach, we can have any number of different types of (e.g.) VariantSet. Right now we have SimulatedVariantSet, and HtslibVariantSet, and the polymorphism between these is taken care of by SqlAlchemy. At query time, we create an instance of the correct type of object, which has all the information it needs to be able to run the query. So, for example, we might have UcscBrowserVariantSet which has a URL to access the information, and whatever IDs are necessary to map into the remote data model. Paging will be done at the level of the data source, so if there is a mapping  between GA4GH objects and the data source we should be able to accommodate it.
So, this change is not about changing how we access bulk data within a set, it's about changing how we store and access information about the sets themselves. It's also about creating a framework to allow us to be vastly more flexible about where and how we obtain the data that's within these sets. Does this make sense?

Oh, and on the id.  From the API perspective, I have never been
happy that id is server-local (really more like a pointer) and
that the true object data is is this ill-defined name think.
It's probably the right thing to do, but feels like that API
punted on a very important bioinformatics issue.
It makes me nervous to no have id unique within a server and
require context.  Maybe I am just being a purist.
How about explain the id trades off a bit more, maybe we can
come up with a good solution.

Yes, this makes a lot of sense...
I haven't looked at the change, so the object boundaries were
not clear.  This makes a lot sense for containers.
Jerome Kelleher<EMAIL_ADDRESS>writes:

Thanks for the feedback @diekhans and @andrewjesaitis, this is very helpful.
@andrewjesaitis, I tend to agree with you. I think we should lean towards using
tables where possible in the interest of forward compatibility as well. We may
not have queries on parts of the data model at the moment, but hopefully we
will have them in the future. It'll be a lot easier to accommodate this if we
already have the information we need in tables.
@diekhans One thing that I should clear up here is that SqlAlchemy is only for
the registry. The actual implementations of the data sources for the particular
objects (e.g. VariantSet) is separate. This is an important separation of
concerns for me; we a well defined SQL based approach to manage the high-level
data model, as this is what it excels at. For bulk data storage, there's any
number of other things we can use. I don't think we should try to do both
(metadata/data model management and bulk data storage) at the same time.
In this new approach, we can have any number of different types of (e.g.)
VariantSet. Right now we have SimulatedVariantSet, and HtslibVariantSet, and
the polymorphism between these is taken care of by SqlAlchemy. At query time,
we create an instance of the correct type of object, which has all the
information it needs to be able to run the query. So, for example, we might
have UcscBrowserVariantSet which has a URL to access the information, and
whatever IDs are necessary to map into the remote data model. Paging will be
done at the level of the data source, so if there is a mapping between GA4GH
objects and the data source we should be able to accommodate it.
So, this change is not about changing how we access bulk data within a set,
it's about changing how we store and access information about the sets
themselves. It's also about creating a framework to allow us to be vastly more
flexible about where and how we obtain the data that's within these sets. Does
this make sense?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

Excellent, glad to hear it @diekhans!
Yeah, I hear you on the ID front. As I see it, there are two approaches we can take for ensuring repository-wide unique IDs:

Create a global ID table for all the externally visible objects in the registry, and rejection sample until we get a unique one when we're creating a new object. Pros: easy; cons: an extra table, and maintenance overhead (we have to remember to delete the IDs from this table).

Create a unique prefix for each externally visible object type in the registry. Since we'd like to use integers for IDs if at all possible (within the registry, at least), we could use the top 4 bits of the 32bit value for this. Pros: no extra overhead, and easy to maintain; Cons: mildly tricky to implement.

I'm assuming at the moment that individual IDs should be generated randomly to avoid any potential information side channels in the future. Since we're going to be storing sensitive data I think we should take this seriously.

Although approach is to use UUIDs.  Although it does generate
128 bit ids, I am not sure how big this impact actually is. .
They are not guaranteed to be unique, but I think we can live
with the probably of the server group being destroyed by a meteor
being hight than a collision.
The high-bit approach sits well with me.  It might be cleaner to
do it as two columns that form the key.
Jerome Kelleher<EMAIL_ADDRESS>writes:

Excellent, glad to hear it @diekhans!
Yeah, I hear you on the ID front. As I see it, there are two approaches we can
take for ensuring repository-wide unique IDs:

Create a global ID table for all the externally visible objects in the
registry, and rejection sample until we get a unique one when we're
creating a new object. Pros: easy; cons: an extra table, and maintenance
overhead (we have to remember to delete the IDs from this table).

Create a unique prefix for each externally visible object type in the
registry. Since we'd like to use integers for IDs if at all possible
(within the registry, at least), we could use the top 4 bits of the 32bit
value for this. Pros: no extra overhead, and easy to maintain; Cons: mildly
tricky to implement.

I'm assuming at the moment that individual IDs should be generated randomly to
avoid any potential information side channels in the future. Since we're going
to be storing sensitive data I think we should take this seriously.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

Am I right to understand that the compound ID scheme will go away with this PR, at least for objects where we haven't required it?

Some progress made:

Decent support for metadata across references, variants and reads.
Good support for variants with vastly simpler paging algorithm.
Partial support for reads using same paging algorithm. (I haven't fully convinced myself this is correct yet though...)

Here's the current state of the registry DB schema. I've tried to follow the letter of the protocol here for forwards compatibility, except for one or two minor cases. It would be great if SQLers could have a look at this to make sure it all makes sense!

@david4096 Yes, CompoundId almost entirely goes away now. We still need it for variants and reads, but we'll be able to simplify the code a lot. Maybe 128 UUIDs are the way to go, but it just feels a bit wasteful to be joining on such large IDs. I can definitely be convinced here though.

I hope any require compound id support still gets handle through
polymorphism vs having to separate id paths.
Given all the advantages of UUIDs, I would like to see how much
impact they have for rejecting them based on worry.  If stored
at a number rather than a string, they are represented as two words
on a 64-bit computer.
I normally didn't worry about going from an one to two word
representation on a 16-bit computer which were orders of
magnitude slower.  Does it really matter here?
Jerome Kelleher<EMAIL_ADDRESS>writes:

@david4096 Yes, CompoundId almost entirely goes away now. We still need it for
variants and reads, but we'll be able to simplify the code a lot. Maybe 128
UUIDs are the way to go, but it just feels a bit wasteful to be joining on such
large IDs. I can definitely be convinced here though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

I hope any require compound id support still gets handle through
polymorphism vs having to separate id paths.
Not sure I follow here @diekhans, can you elaborate a bit? Basically we have VariantCompoundId and ReadCompoundId which contain enough information to uniquely identify a variant/read within a respository. These are used in two places: (1) for GET variant; and (2) in the new paging algorithm for reads/variants.

Given all the advantages of UUIDs, I would like to see how much
impact they have for rejecting them based on worry.  If stored
at a number rather than a string, they are represented as two words
on a 64-bit computer.
It's not quite as simple as that though, as SqlAlchemy doesn't have a native 128 bit number type, so we'll have to do some fiddling around to make it work across DBs. Not necessarily a big deal, but there should be some real benefit before taking on this complexity. Being able to treat IDs directly as ints in Python code rather than bytes is handy as well. What are the benefits of UUIDs, other than making the probability of collision small enough that it can be disregarded (which is probably also true of a uniformly generated 64 bit number)?

Anyway, it'll be easy to change this once we have the code working. All external ID columns are allocated by a single function, so it'll be simple to change it. Right now we're just generating a random 31 bit value and not worrying about collisions. We should definitely look at this properly when the rest of the code is up to scratch.

point #1 is just to keep an object for an id, as opposed to
baking any assumptions into the code
point #2 the benefit of UUIDs is they can be used cross table,
database, servers with some centralized authority to allocate
them.  I have never seen an random number algorithm that can do
this with 64-bit numbers.  Although I discussion of 64-bit host
ids generated with timestamps, I would like to see an academic
paper on an algorithm before trusting it.  Also, UUIDs are the
defacto standard why to generate unique ids.
although #1 hides it all anyway.
Python can handled both 128-bit ints and has a native UUID type.
Postgress has a UUID type, sqllite and mysql can have int(128).
It probably breaks on Sql Server  ;-)
See
http://docs.sqlalchemy.org/en/latest/core/custom_types.html#backend-agnostic-guid-type
on how to do a custom UUID type.
The disadvantage of UUIDs is they you don't have any information
to identify where the object is (table, file line, URL, etc)
Having id has a tuple (CompoundId) does this.
I guess this comes down to the properties we want id to have.
How about explicitly documenting (i.e. a comment) the CompoundId
class with the motivation??
Jerome Kelleher<EMAIL_ADDRESS>writes:

I hope any require compound id support still gets handle through
polymorphism vs having to separate id paths.
Not sure I follow here @diekhans, can you elaborate a bit? Basically we
have VariantCompoundId and ReadCompoundId which contain enough information
to uniquely identify a variant/read within a respository. These are used in
two places: (1) for GET variant; and (2) in the new paging algorithm for
reads/variants.

Given all the advantages of UUIDs, I would like to see how much
impact they have for rejecting them based on worry. If stored
at a number rather than a string, they are represented as two words
on a 64-bit computer.
It's not quite as simple as that though, as SqlAlchemy doesn't have a
native 128 bit number type, so we'll have to do some fiddling around to
make it work across DBs. Not necessarily a big deal, but there should be
some real benefit before taking on this complexity. Being able to treat IDs
directly as ints in Python code rather than bytes is handy as well. What
are the benefits of UUIDs, other than making the probability of collision
small enough that it can be disregarded (which is probably also true of a
uniformly generated 64 bit number)?

Anyway, it'll be easy to change this once we have the code working. All
external ID columns are allocated by a single function, so it'll be simple to
change it. Right now we're just generating a random 31 bit value and not
worrying about collisions. We should definitely look at this properly when the
rest of the code is up to scratch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

@diekhans Would this publication be something along the lines of what you might be looking for?
T. Nishimura, Tables of 64-bit Mersenne Twisters ACM Transactions on Modeling and Computer Simulation 10. (2000) 348--357.
The publication is available here for download:
http://dl.acm.org/citation.cfm?id=369540
The 64-bit implementation of the code is available here for download:
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt64.html

OK, UUIDs are good, but there are some issues:

We don't need globally unique IDs,  just unique IDs within a repository. I don't see any point in generating a globally unique ID for say, a Dataset. What good does it do? This tells us nothing about its contents, and has no meaning across different repositories. You can say the same for ReadGroupSets and VariantSets, etc. Does it matter if there are ID clashes between repositories? The probability of this is really small, and unless ID formats are mandated by the schema, there are no guarantees anyway.
Which specific UUID form are you in favour of? If it's a UUID1 or UUID3, then we're leaking a lot of information which might or might not turn out to be an information side channel. (I think we should keep this in mind since we'll be dealing with sensitive data.)  It it's UUID4 then we're basically just generating a 122 bit uniform random number, most likely by drawing from /dev/urandom. The collision probabilities are just based on this source being uniform. There's no special  special sauce here, we can use exactly the same code to generate our own random numbers. A 64 bit random number will effectively be the same thing as a UUID4. The probability of a collision within a repository is about the same as dropping a ball bearing from a jet randomly and hitting, say, an egg sandwich (to paraphrase Douglas Adams). And we can make sure it doesn't happen pretty easily by using the top byte as a table specific value.

So, yes UUIDs solve some problems, but they don't solve everything. As you pointed out, we won't be able to tell what type of object an ID came from if we use UUID.  Also, I'm not sure that the problem they do solve (global uniqueness) is a problem that we actually need to solve.
Anyway, I think we should put this discussion off until after the code is ready for use and we can try out the various options. It'll be very easy to change, as I've deliberately isolated all the columns that generate externally visible IDs. The GUID SqlAlchemy type looks great, and would be easy to plug in.
In more progress in the code itself, I've just fixed the long standing issue with search over multiple read groups (within a read group set). It turned out to be pretty straightforward under the new organisation.

Agree we don't have a federated API and no need to support global
uniqueness.
My reason for suggesting UUIDs is that is the standard way to
generate uniques ids..  The probability is much higher that a
meteor will destroy your data center than a UUID collides.  The
logical is do it the common way unless the common way doesn't
work.
However, if you want the id to have actual meaning other than a
strong yet independent uniqueness property, it's the wrong tool.
I am trying to simulate discussion; not pushing anything.
Nice on fixing the read group stuff!!
Jerome Kelleher<EMAIL_ADDRESS>writes:

OK, UUIDs are good, but there are some issues:

We don't need globally unique IDs, just unique IDs within a repository. I
don't see any point in generating a globally unique ID for say, a Dataset.
What good does it do? This tells us nothing about its contents, and has no
meaning across different repositories. You can say the same for
ReadGroupSets and VariantSets, etc. Does it matter if there are ID clashes
between repositories? The probability of this is really small, and unless
ID formats are mandated by the schema, there are no guarantees anyway.
Which specific UUID form are you in favour of? If it's a UUID1 or UUID3,
then we're leaking a lot of information which might or might not turn out
to be an information side channel. (I think we should keep this in mind
since we'll be dealing with sensitive data.) It it's UUID4 then we're
basically just generating a 122 bit uniform random number, most likely by
drawing from /dev/urandom. The collision probabilities are just based on
this source being uniform. There's no special special sauce here, we can
use exactly the same code to generate our own random numbers. A 64 bit
random number will effectively be the same thing as a UUID4. The
probability of a collision within a repository is about the same as
dropping a ball bearing from a jet randomly and hitting, say, an egg
sandwich (to paraphrase Douglas Adams). And we can make sure it doesn't
happen pretty easily by using the top byte as a table specific value.

So, yes UUIDs solve some problems, but they don't solve everything. As you
pointed out, we won't be able to tell what type of object an ID came from if we
use UUID. Also, I'm not sure that the problem they do solve (global uniqueness)
is a problem that we actually need to solve.
Anyway, I think we should put this discussion off until after the code is ready
for use and we can try out the various options. It'll be very easy to change,
as I've deliberately isolated all the columns that generate externally visible
IDs. The GUID SqlAlchemy type looks great, and would be easy to plug in.
In more progress in the code itself, I've just fixed the long standing issue
with search over multiple read groups (within a read group set). It turned out
to be pretty straightforward under the new organisation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

OK, this is starting to come together, so it would be really helpful to get some early review on the code as it is.
Main changes:

Registry DB locations are now specified with a SQLAlchemy connection string, e.g., sqlite:///file.db or postgres://server.com/database.
The entire ga4gh.datamodel package will be removed, and is being refactored into the new ga4gh.datasource package. The datasource package currently contains two modules simulator and htslib. The idea is to localise all the code that accesses (or generates) the data in a similar way so that it can share functionality.
The paging functionality for reads and variants has been pushed down into the datasource layer. This has greatly simplified these code paths, and allowed us to simply implement multi-readgroup searches.
Top level metadata is now entirely modelled in SQL using the SqlAlchemy ORM layer. All queries should be now done on-line against a fully updateable DB.
To fit in with the way SqlAlchemy is best used, there has been a substantial style change. All attributes on objects in the registry are now accessed via public attributes. Also all new code is using snake_case where possible. Test code still uses mostly camelCase in an effort to keep the diff getting too massive.

TODO

Update the CompoundId structure, and greatly simplify it.
Port ontologies
Port Variant Annotations code.
Port Sequence annotations code.
Improve documentation and general code layout.
Come up with a way of unit testing Postgres support (particularly for repo manager interface).

It would be good to get some feedback on this @dcolligan --- would you mind having a quick look through?

Well this is a massive change, and a work in progress, but I see the motivation and the benefits.  Having a bunch of SQL strings (that needed to be updated with every field change) was getting cumbersome.  On-line updates that don't require a server restart is a win.  Multiple DB support is a win.  Greater extensibility is a win.  Using well-tested third party libraries over custom code is almost always a good idea.
As far as answers to the questions you posed at the outset, I would favor stuffing fields in a JSON blob until there is a proven motivation for needing to normalize them in some kind of relational schema.  IDs: registry-unique would be nice (if strictly unnecessary), globally-unique seems unnecessary at this point.  64 bits should be sufficient.

I don't know if it was resolved it the POC, however the GA4GH API specification does
require that ids be unique across the server
from https://github.com/ga4gh/schemas/blob/master/doc/source/api/apidesign_intro.rst

ids are unique within the scope of the server instance

I've kicked this can down the road a bit @diekhans. This will be easy to sort this out once the code is merged and stable. For now, we're just generating random 32 bit numbers for every externally visible object. For practical purposes this means that IDs are unique within a server instance until we sort this out properly.

Has performance testing been done using the SQL offset/limit
mechanism for paging?
I am concern that it will be slow to get progressively slower
subsequent page request, especially for queries that are not
returning all rows in physical order.  Such as a range search
for variants.

Not at scale, no. I would have thought that this would be fine once the query is using an index? All queries from the registry DB should use an index, as a matter of principle.
More specialised queries like range searches over variants are outside the scope of this PR.

Even if it's index, it has to play back the query and run it up
to the desired offset.  This slows down the larger the index
gets.  See
https://developer.wordpress.com/2014/02/14/an-efficient-alternative-to-paging-with-sql-offsets/
http://sqlperformance.com/2015/01/t-sql-queries/pagination-with-offset-fetch
This will should be fine if the data is small, as with most
containers.  A couple of random mysql trials showed significant
slowdown with an offset of 40000 vs 1.
your mileage may vary
Jerome Kelleher<EMAIL_ADDRESS>writes:

Not at scale, no. I would have thought that this would be fine once the query
is using an index? All queries from the registry DB should use an index, as a
matter of principle.
More specialised queries like range searches over variants are outside the
scope of this PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

Hmm, that is interesting! OK, this is something we'll have to consider. If we're searching on primary keys in table order then we're fine, as we can use the ID as the page token and resume from there. If not, then we'll have to think of something else.
Thanks for the heads-up @diekhans, I had not considered this issue...

Closing for now.  We can open another branch if we decide to pursue this.


Ignore - Test PR do debug nightlies at CI

We found that the latest version of copy-webpack-plugin is also incompatible with webpack 4.x:
https://github.com/webpack-contrib/copy-webpack-plugin/issues/575

We found that the latest version of copy-webpack-plugin is also incompatible with webpack 4.x:
https://github.com/webpack-contrib/copy-webpack-plugin/issues/575


Podman remote not working with RHCOS on Windows/Hyper-V

Please see: https://github.com/code-ready/crc/issues/1083 and related debugging info in https://github.com/code-ready/crc/issues/961#issuecomment-595661000. We have tried to use podman-remote with the RHCOS VM (using CRC) on Windows+Hyper-V but no matter how it is configured it does not work as expected, while the VM works on macOS. Also, targetting a VM running F30 on Hyper-V works.

tested with @baude and got an updated client which looks good:
PS C:\Users\gbraad\.crc\bin> $Env:PODMAN_USER = "core"
PS C:\Users\gbraad\.crc\bin> $Env:PODMAN_HOST = "<IP_ADDRESS>"
PS C:\Users\gbraad\.crc\bin> $Env:PODMAN_IDENTITY_FILE = "C:\Users\gbraad\.crc\machines\crc\id_rsa"
PS C:\Users\gbraad\.crc\bin> $Env:PODMAN_IGNORE_HOSTS = "1"
PS C:\Users\gbraad\.crc\bin> .\podman-remote-windows.exe --identity-file $env:PODMAN_IDENTITY_FILE --username $env:PODMAN_USER --remote-host $env:PODMAN_HOST info
client:
  Connection: ssh -p 22 -T -i C:\Users\gbraad\.crc\machines\crc\id_rsa -q -o StrictHostKeyChecking=no
    -o UserKnownHostsFile=/dev/null<EMAIL_ADDRESS>-- varlink -A 'podman --log-level=debug
    varlink $VARLINK_ADDRESS' bridge
  Connection Type: BridgeConnection
  OS Arch: windows/amd64
  Podman Version: 1.8.1-dev
  RemoteAPI Version: 1
host:
  arch: amd64
  buildah_version: 1.12.0-dev
  cpus: 4
  distribution:
    distribution: '"rhcos"'
    version: "4.3"
  eventlogger: journald
  hostname: crc-jccc5-master-0
  kernel: 4.18.0-147.3.1.el8_1.x86_64
  mem_free:<PHONE_NUMBER>2
  mem_total:<PHONE_NUMBER>2
  os: linux
  swap_free: 0
  swap_total: 0
  uptime: 7m 10.28s
registries:
  blocked: null
  insecure: null
  search: null
store:
  containers: 0
  graph_driver_name: overlay
  graph_driver_options: |-
    map[overlay.mount_program:map[Executable:/usr/bin/fuse-overlayfs Package:fuse-overlayfs-0.4.1-1.module+el8.1.0+4081+b29780af.x86_64 Version:fuse-overlayfs: version 0.4.1
    FUSE library version 3.2.1
    using FUSE kernel interface version 7.26]]
  graph_root: /var/home/core/.local/share/containers/storage
  graph_status:
    backing_filesystem: xfs
    native_overlay_diff: "false"
    supports_d_type: "true"
  images: 0
  run_root: /run/user/1000

https://github.com/code-ready/crc/issues/1083#issuecomment-598182506

Fixed with https://github.com/containers/libpod/releases/tag/v1.8.2


driver/fakesensor: fix bugs.

1.fix crash casued by closing the handle during the reading.
2.fix set_interval fail sometime:When fakesensor was activated, csv file is not opened timely, so
the sensor->interval that set_interval() seted will be coverd by
interval form csv file.
Signed-off-by: buyuer<EMAIL_ADDRESS>Summary
Impact
Testing

hi @buyuer why did you decide to close it? Did you manage to fix the issue other way?

@acassis here is the fix https://github.com/apache/incubator-nuttx/pull/4007

thank you @xiaoxiang781216


Update heatmap fixtures to restore tests (#8615)

Update heatmap fixtures to restore tests
Add hint to check the fixture age on fail

Backport #8651

Backport #8615


documentation examples, wasatch svg and background

Added escape processing for faster svg testing. Included wasatch photonics graphics examples and command line documentation.

Faster testing is already possible using the '-e' option. Additionally, those company branded examples are not necessary in my opinion.


tortoise beacon: prevent bad msgs being gossiped and add tests

Motivation

Closes #2817

Changes

for messages that fail verification, propagate errors and don't gossip about them
add test for various error conditions when handling gossip messages
rewrite hander_test.go to separate the test cases

Test Plan

unit tests
DevOps Notes

[x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
[x] This PR does not affect public APIs
[x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
[x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors merge

Build failed:

ci-stage2

an old issue https://spacemesh.slack.com/archives/C01BBB2U64U/p1630336484017700
quoting @AmitShaul in the slack channel
it’s not a timezone issue, it seems that the nodes took too long to start and the preset genesis delta already expired

bors merge


cfn-lint silently ignores unrecognized arguments

CloudFormation Lint Version
cfn-lint 0.78.1
What operating system are you using?
Ubuntu 20.04
Describe the bug
Today I got really confused about why cfn-lint was ignoring aws-sso-util's own validation spec.
Why are the outputs different?

$ cfn-lint --template Macro-Test.yaml --region eu-central-1 --spec-override cfn-lint-spec.json
E3001 Invalid or unsupported Type SSOUtil::SSO::AssignmentGroup for resource AssignmentGroup in eu-central-1
Macro-Test.yaml:25:5

$ cfn-lint --template Macro-Test.yaml --region eu-central-1 --override-spec cfn-lint-spec.json

It should have been the second one. I had mixed up the name of the parameter!
The CLI just seems to accept and silently ignore whatever it doesn't understand.
$ cfn-lint --template Macro-Test.yaml --region eu-central-1 --fdsfdsafdsafdsafdsafsa
E3001 Invalid or unsupported Type SSOUtil::SSO::AssignmentGroup for resource AssignmentGroup in eu-central-1
Macro-Test.yaml:25:5

Expected behavior
Fail on the unknown argument like this:
$ cfn-lint --template Macro-Test.yaml --region eu-central-1 --spec-override cfn-lint-spec.json
fatal: unrecognized argument: --spec-override

(Inspired by git version 2.41.0's handling of unknown arguments. Try git log --names-only.)
or even better:
cfn-lint: '--spec-override' is not an argument. See 'cfn-lint --help'.

The most similar arguments are:
        --override-spec

(Inspired by git version 2.41.0's handling of unknown subcommands. Try git lag.)
Reproduction template
Template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS-SSO-Util-2020-11-08

Resources:
  AssignmentGroup:
    Type: SSOUtil::SSO::AssignmentGroup
    Properties:
      Name: ReadOnly
      InstanceArn: arn:aws:sso:::instance/ssoins-ffffffffffffffff
      Principal:
        - Type: USER
          Id: ffffffff-ffff-ffff-ffff-ffffffffffffffff
      PermissionSet:
        - arn:aws:sso:::permissionSet/ssoins-ffffffffffffffff/ps-ffffffffffffffff
      Target:
        - Type: AWS_OU
          Id: ou-zzzz-zzzzzzzz

Spec:
{
    "ResourceTypes": {
        "AWS::SSO::PermissionSet": {
            "Properties": {
                "InstanceArn": {
                    "Required": false
                },
                "InlinePolicy": {
                    "PrimitiveType": "Json"
                }
            }
        },
        "SSOUtil::SSO::AssignmentGroup": {
            "Properties": {
                "InstanceArn": {
                    "PrimitiveType": "String",
                    "Required": false,
                    "UpdateType": "Immutable"
                },
                "Name": {
                    "PrimitiveType": "String",
                    "Required": false,
                    "UpdateType": "Immutable"
                },
                "Principal": {
                    "Required": true,
                    "UpdateType": "Mutable"
                },
                "PermissionSet": {
                    "Required": true,
                    "UpdateType": "Mutable"
                },
                "Target": {
                    "Required": true,
                    "UpdateType": "Mutable"
                }
            }
        }
    }
}

This is tagged good first issue and indeed it's potentially a one-line change, but is there a reason it uses ArgumentParser.parse_known_args() instead of parse_args()?
https://github.com/aws-cloudformation/cfn-lint/blob/86f0370bd43b400ed4c485180dbc2697f73367b2/src/cfnlint/config.py#L315

I've been thinking about this one for a while. I don't think so... except maybe just a miss.  But good point I would want to see this through the ringer a few times to make sure I'm not missing something.

Hi,
I am new to the open source community and would love to contribute to this project. Have raised a PR for the same.
Thanks.

This is tagged good first issue and indeed it's potentially a one-line change, but is there a reason it uses ArgumentParser.parse_known_args() instead of parse_args()?
https://github.com/aws-cloudformation/cfn-lint/blob/86f0370bd43b400ed4c485180dbc2697f73367b2/src/cfnlint/config.py#L315

I don't believe so.  I was debating rolling this change into v1 just in case we are to break something as a result of this change.


ERROR:aw_browser_terminator.cc(123)

E/chromium: [ERROR:aw_browser_terminator.cc(123)] Renderer process (32341) crash detected (code 5).
A/chromium: [FATAL:crashpad_client_linux.cc(667)] Render process (32341)'s crash wasn't handled by all associated  webviews, triggering application crash.
A/libc: Fatal signal 5 (SIGTRAP), code -6 (SI_TKILL) in tid 32300 (m.boolv.recycle), pid 32300

Any luck resolving this?


BCH transaction error/return

Hi,
A few hours ago I thought I had sent 1 BCH to legacy address 18cC12roA4MHZQjULXYgkZphu17rgi4iCC. This was a BCH Blender deposit.
However the output address (as far as I can tell) on my wallet transaction details is 1AetzhW7kj4L9TkmQKeqCeUi7uH4Ks6zh8
The transaction page on BCH Blender says 'waiting for deposit'. I thought the transaction was finished though, it has 25 confirmation.
All I did was copy and paste the legacy address.
Have I sent it to the wrong address?
Can I get the payment returned to me?
Any help would be really appreciated.

duplicate of #680


Are there available indoor datasets or pinhole camera datasets?

It would be great if you could provide datasets and configuration files.
thanks.

I fixed some problems and run indoor successfully,you can look up to https://github.com/skyrim835/LVI-SAM-modified

I will try it, thanks.


adds alt builds for 1804 and next salt

removed fastly from snsalt build, kicking

Now failing on our custom kong Salt state, which needs a Python 3 upgrade?
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:           ID: add-plugin-acl-for-annual_reports
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:     Function: kong.post_plugin
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:         Name: acl
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:       Result: False
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:      Comment: An exception occurred in this state: Traceback (most recent call last):
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                 File "/usr/lib/python3/dist-packages/salt/state.py", line 1919, in call
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                   **cdata['kwargs'])
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                 File "/usr/lib/python3/dist-packages/salt/loader.py", line 1918, in wrapper
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                   return f(*args, **kwargs)
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                 File "/var/cache/salt/minion/extmods/states/kong.py", line 28, in post_plugin
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                   existing_plugin = _plugin(admin_api, api, name)
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                 File "/var/cache/salt/minion/extmods/states/kong.py", line 45, in _plugin
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                   matching = [plugin for plugin in json.loads(response.content)['data'] if plugin['name'] == name]
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                 File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:                   s.__class__.__name__))
[2020-03-03T07:10:40.900Z] [<IP_ADDRESS>] out:               TypeError: the JSON object must be str, not 'bytes'

Now failing on our custom kong Salt state, which needs a Python 3 upgrade?

that's been fixed (see diff and fresh), but base will continue to fail. I'm trying to get snsalt build to pass but I'm getting permissions errors from Vault. I shouldn't be, I disabled fastly for this project+alt-config so something is holding on to that config.
Will give it another kick while I inspect state of Alfred

I think the problem is this PR was built with one set of config and has had that config changed from out beneath it, leaving it with context files that are incorrect. For example, this:
/srv/builder/.cfn/contexts/api-gateway--pr-15-fresh-snsalt.json

contains references to fastly and it's this context that is used to build the Cloudformation and Terraform templates.
closing PR and opening a duplicate one


Bottom red progress bar

How to hide bottom red progress bar when video play.

change the progressbar in every state

I am using cn.jzvd.JZVideoPlayerStandard in recycleview  so how can do that. How to change the progressbar in every state

GL


fix typos in comment.

What have you changed? (mandatory)
fix typos in comment and output log.
How has this PR been tested? (mandatory)
just comment and output log, no test needed.
Does this PR affect documentation (docs) update? (mandatory)
No
Does this PR affect tidb-ansible update? (mandatory)
No

@BusyJay  could you please take a look, thanks.


pythonPackages.signedjson: 1.0.0 -> 1.1.0

Motivation for this change
noticed it was broken reviewing #80447
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

since #80447 is on pause, going to merge this as it's still an improvement, other branch and just rebase onto master again.

@GrahamcOfBorg build pythonPackages.signedjson
@GrahamcOfBorg build python3Packages.signedjson
@GrahamcOfBorg build python38Packages.signedjson

unfortunately this breaks matrix-synapse, which is the only consumer:
error: build of '/nix/store/sh6yaln159has3rk0x90kmkdn2gjip28-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/80449
1 package failed to build:
matrix-synapse

3 package built:
python27Packages.signedjson python37Packages.signedjson python38Packages.signedjson


[Usage] CLI stuck at "User:"

Describe the issue
Issue:
Can't run this command cause stuck at "User:"
Command:
I run the example demo CLI.

Log:
PASTE THE LOGS HERE.

Screenshots:
You may attach screenshots if it better explains the issue.

Solved


Craig primate recon update

Draft of code to do an mle reconstruction of the hca

When this gets merged it may be worth making the first release of gonomics so in the paper you can refer people to gonomics v0.1 instead of gonomics commit fb2bf53


Corrected mattermost config get command example

Documentation for: https://github.com/mattermost/docs/issues/6855

Thanks, @coltoneshaw! This PR is ready for your final review.


python3Packages.nbclient: 0.5.8 -> 0.5.9, fix build

This also fixes the build failure introduced in https://github.com/NixOS/nixpkgs/pull/146151, as for some reason the bin directory is empty when doCheck = true and then the postFixup failed.
0.5.9 contains https://github.com/jupyter/nbclient/pull/173 so no need for the postFixup anymore.
This should also close https://github.com/NixOS/nixpkgs/issues/146277.
The original issue was https://github.com/NixOS/nixpkgs/issues/145968.
Run
nix build --impure --expr "let pkgs = import ./. {}; in pkgs.python3.withPackages (ps: [ ps.jupyterlab ])"

for testing.
Motivation for this change
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

@cole-h
@ofborg eval

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
Result of nixpkgs-review pr 146813 run on x86_64-linux 1

4 packages marked as broken and skipped:

cq-editor
python38Packages.skein
python38Packages.spyder
python39Packages.skein

30 packages failed to build and already failed to build on hydra master:

python38Packages.atomman: log was empty
python38Packages.ipydatawidgets: log was empty
python38Packages.ipympl: log was empty
python38Packages.potentials: log was empty
python38Packages.qiskit: log was empty
python38Packages.qiskit-aer: log was empty
python38Packages.qiskit-aqua: log was empty
python38Packages.qiskit-finance: log was empty
python38Packages.qiskit-ibmq-provider: log was empty
python38Packages.qiskit-ignis: log was empty
python38Packages.qiskit-machine-learning: log was empty
python38Packages.qiskit-nature: log was empty
python38Packages.qiskit-optimization: log was empty
python38Packages.qiskit-terra: log was empty
python38Packages.robotframework-tools: log was empty
python39Packages.atomman: log was empty
python39Packages.ipydatawidgets: log was empty
python39Packages.ipympl: log was empty
python39Packages.potentials: log was empty
python39Packages.qiskit: log was empty
python39Packages.qiskit-aer: log was empty
python39Packages.qiskit-aqua: log was empty
python39Packages.qiskit-finance: log was empty
python39Packages.qiskit-ibmq-provider: log was empty
python39Packages.qiskit-ignis: log was empty
python39Packages.qiskit-machine-learning: log was empty
python39Packages.qiskit-nature: log was empty
python39Packages.qiskit-optimization: log was empty
python39Packages.qiskit-terra: log was empty
python39Packages.robotframework-tools: log was empty

1 package failed to build and are new build failure:

python38Packages.datashader: log was empty

102 packages built:

ihaskell
jupyter
opsdroid
python38Packages.Nikola
python38Packages.Pweave
python38Packages.ansible-kernel
python38Packages.batchspawner
python38Packages.cufflinks
python38Packages.dockerspawner
python38Packages.einops
python38Packages.holoviews
python38Packages.hvplot
python38Packages.intake
python38Packages.ipyvue
python38Packages.ipyvuetify
python38Packages.ipywidgets
python38Packages.jupyter
python38Packages.jupyter-server-mathjax
python38Packages.jupyter-sphinx
python38Packages.jupyter_server
python38Packages.jupyterhub
python38Packages.jupyterhub-ldapauthenticator
python38Packages.jupyterhub-systemdspawner
python38Packages.jupyterhub-tmpauthenticator
python38Packages.jupyterlab
python38Packages.jupyterlab-git
python38Packages.jupyterlab_launcher
python38Packages.jupyterlab_server
python38Packages.jupytext
python38Packages.kmapper
python38Packages.lightparam
python38Packages.livelossplot
python38Packages.modeled
python38Packages.moretools
python38Packages.nbclassic
python38Packages.nbclient
python38Packages.nbconflux
python38Packages.nbconvert
python38Packages.nbdime
python38Packages.nbsmoke
python38Packages.nbsphinx
python38Packages.nix-kernel
python38Packages.notebook
python38Packages.notedown
python38Packages.oauthenticator
python38Packages.papermill
python38Packages.phik
python38Packages.pydeck
python38Packages.stytra
python38Packages.vega
python38Packages.widgetsnbextension
python38Packages.zetup
python39Packages.Nikola
python39Packages.Pweave
python39Packages.ansible-kernel
python39Packages.batchspawner
python39Packages.cufflinks
python39Packages.datashader
python39Packages.dockerspawner
python39Packages.einops
python39Packages.holoviews
python39Packages.hvplot
python39Packages.intake
python39Packages.ipyvue
python39Packages.ipyvuetify
python39Packages.ipywidgets
python39Packages.jupyter
python39Packages.jupyter-server-mathjax
python39Packages.jupyter-sphinx
python39Packages.jupyter_server
python39Packages.jupyterhub
python39Packages.jupyterhub-ldapauthenticator
python39Packages.jupyterhub-systemdspawner
python39Packages.jupyterhub-tmpauthenticator
python39Packages.jupyterlab
python39Packages.jupyterlab-git
python39Packages.jupyterlab_launcher
python39Packages.jupyterlab_server
python39Packages.jupytext
python39Packages.kmapper
python39Packages.lightparam
python39Packages.livelossplot
python39Packages.modeled
python39Packages.moretools
python39Packages.nbclassic
python39Packages.nbclient
python39Packages.nbconflux
python39Packages.nbconvert
python39Packages.nbdime
python39Packages.nbsmoke
python39Packages.nbsphinx
python39Packages.nix-kernel
python39Packages.notebook
python39Packages.notedown
python39Packages.oauthenticator
python39Packages.papermill
python39Packages.phik
python39Packages.pydeck
python39Packages.stytra
python39Packages.vega
python39Packages.widgetsnbextension
python39Packages.zetup

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

ihaskell:

warning: unused-argument
Unused argument: paths.
Near pkgs/build-support/trivial-builders.nix:372:12:
    |
372 |          , paths
    |            ^


Is there a way to use RunspacePools such that module loading uses more than one processor?

I'm experimenting with running PowerShell unit tests in parallel. It seems like one open runspace per thread is needed to achieve speedup.  I'm seeing good speedup after the runspaces are open (see #6965(comment).  I haven't, however, found a reliable way to parallelize module importing that occurs when opening the runspaces.  (Runspaces.OpenAsync() looks promising, but it seems to suffer from #7034.)
  The result is that despite that there might be many processors available and one runspace for each, only one processor is utilized to perform module importing for all of the runspaces.
Steps to reproduce
$processorCount = [System.Environment]::ProcessorCount

Write-Host "Processor Count: $processorCount"

$moduleContent = {
    function fibonacci {
        param([int]$n)
        [bigint]$a=0
        [bigint]$b=1
        foreach ($x in 0..$n)
        {
            $a,$b = $b,($a+$b)
        }
        $b
    }
    0..0 | % { fibonacci 100000 }
}

$modulePath = "$([System.IO.Path]::GetTempPath())slowLoading.psm1"
$moduleContent | Set-Content $modulePath

$t_import = Measure-Command {
    Import-Module $modulePath -Force
}

$initialSessionState = [initialsessionstate]::CreateDefault()
$initialSessionState.ImportPSModule($modulePath)

$rsp = [runspacefactory]::CreateRunspacePool($initialSessionState)
$rsp.SetMaxRunspaces($processorCount) | Out-Null
$rsp.Open()

$ps = 1..$processorCount | % { [powershell]::Create().AddScript({'done'}) }
$ps | % { $_.RunspacePool = $rsp }

$t_begin = Measure-Command {
    $invocation = $ps.BeginInvoke()
}

$t_wait = Measure-Command {
    while ( $invocation.IsCompleted -contains $false )
    {
        sleep 0.1
    }
}

[pscustomobject]@{
    'name'     = 'Import-Module slowLoading.psm1'
    'time(ms)' = [int]$t_import.TotalMilliseconds
}

[pscustomobject]@{
    'name'     = 'BeginInvoke()'
    'time(ms)' = [int]$t_begin.TotalMilliseconds
}

[pscustomobject]@{
    'name'     = 'Wait'
    'time(ms)' = [int]$t_wait.TotalMilliseconds
}

Expected behavior
Processor Count: 16

name                           time(ms)
----                           --------
Import-Module slowLoading.psm1     5122
BeginInvoke()                       100  (approximately)
Wait                               6000  (approximately)

Actual behavior
Processor Count: 16

name                           time(ms)
----                           --------
Import-Module slowLoading.psm1     5122
BeginInvoke()                     19452
Wait                                 13

Here is the CPU utilization graph for the above test run:

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.1.0-preview.691
PSEdition                      Core
GitCommitId                    v6.1.0-preview.691
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

It looks like the following might be relevant to answering this question:

The time spent on script module compilation affects parallelization differently from the time spent executing a script module's scriptblock.  (I think importing Pester is mostly spent compiling, while importing slowLoading.psm1 is mostly spent executing the scriptblock.)
Importing a module in one runspace slows the execution of code in another runspace that uses that module.

Below are the results of some tests related to these aspects.
.OpenAsync() with slowLoading.psm1
It looks like .OpenAsync() parallelizes importing of slowLoading.psm1 nicely.  parallelLoadModule.ps1 is a demonstration.  Here is the output of two test runs on a 16-core processor:
Baseline w/ $p overridden to 1

ProcessorCount: 1
54ms : OpenAsync()
68ms : WaitAll()
4018ms : done

ProcessorCount: 16
44ms : OpenAsync()
219ms : WaitAll()
7060ms : done

Opening 16x runspaces took 1.75x as long.  The CPU utilization reflects that all processors are used in parallel:

This seems like good news for parallelized importing of script modules.  However, slowLoading.psm1 isn't a very good stand-in for real-world modules.
.OpenAsync() w/ Pester
Pester is a real-world script module that takes some time to import.
Changing the call in parallelModuleLoad.ps1 to Import-Module $modulePath and .ImportPSModule($modulePath) to Import-Module Pester and .ImportPSModule('Pester'), respectively, causes Pester to be imported into each of the runspaces when .OpenAsync() is called.   Here are the corresponding tests runs:
Baseline w/ $p overridden to 1

ProcessorCount: 1
46ms : OpenAsync()
59ms : WaitAll()
991ms : done

ProcessorCount: 16
59ms : OpenAsync()
234ms : WaitAll()
7627ms : done

Opening 16x runspaces that have Pester imported took 7.69x as long.  The CPU utilization reflects some parallelization occuring:

.OpenAsync() w/ Pester while Running Pester
It's likely that code in a script module will be executed while another runspace is importing that module.
This test profiles importing Pester into a number of runspaces while the first runspace performs CPU-bound execution using Pester.  The results from a typical test run are as follows:
ProcessorCount: 16
59ms : OpenAsync()
7229ms : importing done
throughput while importing: 143 iterations/s
throughput while solo: 246 iterations/s

The time taken to import Pester into the 16 runspaces took about the same time as when Pester code was not simultaneously being executed (7229ms vs. 7627ms).  On the other hand, the single-threaded throughput of Pester code while other runspaces were importing Pester was 0.58x the throughput of that code when executed solo.
This seems to point to the following:

There is some sort of performance interaction between runspaces using the same module.
Executing code using a module in one runspace does not slow loading of that module in another runspace.
Importing a module into one runspace does slow code executing using that module in another runspace.

Per #7153, sharing an AuthorizationManager object amongst Runspaces being opened causes contention.  I wonder if RunspacePool uses a single AuthorizationManager when opening runspaces.