questions about surface Reconstruction

Hi，
Recently, I have read the pytorch code of your work,but I canot fully understand how to reconstrcut surface. According to code,It doesnot use any information in GT_mesh and even the query_point is random generated. How to make sure that our generated mesh can correspond to GT_mesh one by one?
Could you please answer my question?Thank you so much !

Hi，
Please refer to Equation 1 and Equation 2 in the paper for details of how this work without GT mesh, and to the code for the query point sampling rules.
https://arxiv.org/pdf/2011.13495.pdf
https://github.com/mabaorui/NeuralPull/blob/9932b3a18f79db52fd1455f8cd220cf1c7b378a9/sample_query_point.py#L143
https://github.com/bearprin/neuralpull-pytorch/blame/6757bd26d5d006b4168952158ec67a299f8766ba/dataset/train_dataset.py#L65

HI，
I read the paper again .the method is so novel and your impressive work inspires me a lot .Unfortunately，for I am a newhand on 3D reconstruction，I am still confused that I can understand how it works at training step but inference.
【  sample query locations around each point pj of the ground truth point cloud P .】So it needs point P to sample query_point? I don‘t whether I understand it correctly .
At inference step , code
class ValDataset(data.Dataset):
def init(self, bd=0.55, resolution=128):
super(ValDataset, self).init()
shape = (resolution, resolution, resolution)
vxs = torch.arange(-bd, bd, bd * 2 / resolution)
vys = torch.arange(-bd, bd, bd * 2 / resolution)
vzs = torch.arange(-bd, bd, bd * 2 / resolution)
pxs = vxs.view(-1, 1, 1).expand(*shape).contiguous().view(resolution ** 3)
pys = vys.view(1, -1, 1).expand(*shape).contiguous().view(resolution ** 3)
pzs = vzs.view(1, 1, -1).expand(*shape).contiguous().view(resolution ** 3)
self.p = torch.stack([pxs, pys, pzs], dim=1).reshape(resolution, resolution ** 2, 3)  seems doesnot allow the sampling rules.
I noticed that
[Details. We employ Neural-Pull to reconstruct 3D surfaces
from point clouds. Given a point cloud P , we do not lever-
age any condition c in Fig. 1 and overfit the neural network
to the shape by minimizing the loss in Eq. 2, where we re-
move the network for extracting the feature of the condition.] on 4.1. But ValDatastes seems doesnot load any mesh,which conflicts with 【Given a point cloud P】.
If you could give me some help .I'll be veay grateful

Hi,
If you are only confused about the inference phase, please refer to the Marching cubes algorithm. We learned the implicit function field through the sampling query points mentioned above in the training phase. During testing, we simply apply the Marching cubes algorithm to extract the mesh. The code of ValDataset is a discretization of the implicit function field learned in our training phase into the input of Marching cubes.
https://en.wikipedia.org/wiki/Marching_cubes

Hi，
I saw a iuuse at pytorch code repo，on which the writer ever suggested 【 suggest fitting one shape by one network though the Implementation support fitting several shapes by condition code(feat). This will reduce the difficulty of convergence. Just put only one .npy in the 'npy' folder.】
Is it because 【we do not lever-age any condition c in Fig. 1 and overfit the neural networkto the shape by minimizing the loss in Eq. 2, 】？
I haven't learned tensor-flow ,so I am not sure whether  there's difference with tensorflow code. I'm looking forward to your earliest reply. Thanks a lot for the time you take ！

Hi,
It should not be intrinsically linked to the condition c, but multiple models learned simultaneously will make the network more difficult to learn and take longer time to converge. We may release the official implementation of NeuralPull later, but we've been busy lately. I think the quality of implementation bearprin is high, even though it hasn't been fully validated.

Hi，
Thanks alot for your answer.So our code can learned multiple shapes right?I alos think the quality of pytorch coide is so high,It realized mplementation of NeuralPull with only a little amount of code lines and is easily to be read. I have also read some other traditional methods，like DeepSDF. For example ,in DeepSDF, it splits datasets as train and test,and need both pont(x,y,z) and latent vector(latant code ) to get sdf(x). What makes me confused is that, how can our repo  distinguish different shapes while evaluating  without any input ，such as mesh1_horse mesh2_rabbit  ...?

Hi,
Yes, NeuralPull can learned multiple shapes. We use a onehot vector to represent which model to reconstruct.
https://github.com/bearprin/neuralpull-pytorch/blame/6757bd26d5d006b4168952158ec67a299f8766ba/train.py#L106

Hi,
Your answer is really helpful to me!! Now I understand how NeuralPull works .
from what I understand,the next code line 107 【feat[0, :, shape_ind] = 1】mekes the onehot vactor works，and in fact ，the number of query_point of ValDateset is 128^3,which equals to MarchingCube resolution so that ensures contain full pointcloud P.
Have I understood correctly?

Hi,
You are right. If you find our code or paper useful, please consider giving the repository a star. Thanks.

Hi，
I really appreciate your help.I have already star this repo


Impliment Alt+F4 on desktop to show Shutdown options

Describe the solution you'd like
To accommodate more keyboard centric users we should Impliment the Alt+F4 key command as found on the Windows Explorer Desktop. This would of course show the Cairo Shutdown options instead of the Windows one.

Looks like I forgot to close this one after merging.


Auto correct feature in CC Search

As the CC Search has just got out of beta and is now public, there should be an autocorrect option so that if a user has by chance entered the spelling of a word wrong, an option to correct it, could be displayed.
For now, a dictionary can be made(js or any other language) and the text can be checked for errors before sending the API request.
The output of the auto correct could be something like this -

Well, we can keep separate dictionaries for different languages, then using the user's requested language provide suggestions.

@aldenstpage if this would be a good API endpoint, could you please define the ticket better for what the endpoint should look like and then open it up to the community?

You're correct that this is best suited as an API endpoint. I've opened up a more detailed ticket here: https://github.com/creativecommons/cccatalog-api/issues/429


Minor Text Updates to DPCreatorText.csv

Updated a few places in the csv.  See the green colored changes in the screenshots here: https://www.icloud.com/sharedalbum/#B0lJ058xyJ8FYcn.  Red colored ones need to be updated by @ekraffmiller seperately.

Closing, these edits have been included in an already merged PR


Update cookiecutter, adopt pyproject.toml

Pull Request Checklist:

[ ] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #xyz

[x] (If applicable) Documentation has been added / updated (for bug fixes / features).
[ ] (If applicable) Tests have been added.
[x] This PR does not seem to break the templates.
[x] HISTORY.rst has been updated (with summary of main changes).

[x] Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

Updates the cookiecutter to the latest changes, adopting PEPs 517, 518, and 621.
Now using bump-my-version instead of bump2version.
Linting is performed with ruff for the majority of checks, with minimal flake8-based checks (until such a time that these checks are ported to ruff).
Documentation has been modified from the cookiecutter to maintain setuptools-relevant information.
All non-babel configurations have migrated to .flake8 and pyproject.toml.
Many low-effort lint-related fixes applied.
tox builds are now more pure Python build friendly.
pre-commit now has hooks for TOML files, JSON schemas, and ruff.

Does this PR introduce a breaking change?
There are no behavioural changes for users or developers, but the build system configurations are now more centralized into pyproject.toml. bump2version has been replaced with bump-my-version (maintained; pyproject.toml support).
setup.py has been mostly hollowed out. The only function remaining is that which generate the translations. This approach is the recommended way to maintain setup.py functionality in the new build system.
The contents of the wheel are set within pyproject.toml while those of the source distribution are still set via MANIFEST.in, similar to how it was configured before.
Other information:
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
https://docs.pypi.org/trusted-publishers/using-a-publisher/
What's missing?

~The PR summary of changes needs to be fully described~
~Linting checks need to be reconfigured for ruff~
~Trusted Publishing needs to be configured~

We'll need to modify PULL_REQUEST_TEMPLATE.md to refer to CHANGES.rst, instead of HISTORY.rst


Fix labels unmatch issue during chart upgrade.

Fix chart upgrade/rollback issue:
root@mcubt:~/istio/install/kubernetes/helm# helm upgrade istio ./istio --tls
Error: UPGRADE FAILED: Deployment.apps "istio-telemetry" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"version":"1.1.0", "istio-mixer-type":"telemetry", "app":"mixer", "istio":"mixer", "chart":"mixer", "heritage":"Tiller", "release":"istio"}: `selector` does not match template `labels` && Deployment.apps "istio-policy" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"istio-mixer-type":"policy", "chart":"mixer", "heritage":"Tiller", "release":"istio", "version":"1.1.0", "app":"mixer", "istio":"mixer"}: `selector` does not match template `labels

After apply this patch, istio chart can be upgraded from v1.0.2 to master and also rollback is smooth:
root@xaubt1:~/istio/install/kubernetes/helm# helm history istio --tls
REVISION	UPDATED                 	STATUS    	CHART      	DESCRIPTION
1       	Tue Oct 16 03:39:09 2018	SUPERSEDED	istio-1.0.1	Install complete
2       	Tue Oct 16 03:40:49 2018	SUPERSEDED	istio-1.1.0	Upgrade complete
3       	Tue Oct 16 03:43:26 2018	DEPLOYED  	istio-1.0.1	Rollback to 1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@morvencao: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
4d92facd409bc573bfa57c9db4966f949b8c911d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
563ae5f7dae752cf18317305ba232028cd2a573e
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@gyliu513 Thanks for reminder, have updated YAML for kiali and certmanager.

@morvencao should not we update all deployment YAML templates for the helm chart, like galley, gateway etc?

@gyliu513 Based on my local tests, some deployment(mixer, kiali, certmanager and prometheus) get inconsistent selector labels during upgrading, so we just need the inconsistent ones, not all.

@morvencao based on the discussion, I'd prefer we do not change the app labels, can you update the PR accordingly?

@morvencao Done

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: morvencao
If they are not already assigned, you can assign the PR to them by writing /assign @morvencao in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@morvencao: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-multicluster-e2e.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-multicluster-e2e.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test istio-pilot-multicluster-e2e

prow/e2e-mixer-no_auth.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: morvencao
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS [morvencao]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-multicluster-e2e.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test istio-pilot-multicluster-e2e

prow/e2e-mixer-no_auth.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth

prow/e2e-simpleTests-cni.sh
ff434273128cdc17561a1af7fbb4db0b78039970
link
/test e2e-simpleTests-cni

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth-mcp

prow/e2e-mixer-no_auth.sh
73bb549a34202d539172e480eecb8248409867cd
link
/test e2e-mixer-no_auth

prow/e2e-simpleTests-cni.sh
ff434273128cdc17561a1af7fbb4db0b78039970
link
/test e2e-simpleTests-cni

prow/istio-pilot-multicluster-e2e.sh
ff434273128cdc17561a1af7fbb4db0b78039970
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This has been fixed in https://github.com/istio/istio/pull/10186

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-simpleTests-cni.sh
ff434273128cdc17561a1af7fbb4db0b78039970
link
/test e2e-simpleTests-cni

prow/istio-pilot-multicluster-e2e.sh
ff434273128cdc17561a1af7fbb4db0b78039970
link
/test istio-pilot-multicluster-e2e

prow/e2e-dashboard.sh
23dc985951dcb1b9aeca17d475b960e63544e747
link
/test e2e-dashboard

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-dashboard.sh
23dc985951dcb1b9aeca17d475b960e63544e747
link
/test e2e-dashboard

prow/istio-pilot-multicluster-e2e.sh
23dc985951dcb1b9aeca17d475b960e63544e747
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


[Release][3.1.4] add 3.1.4 release doc

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report

Merging #13621 (eb5085f) into dev (6516240) will decrease coverage by 0.09%.
The diff coverage is n/a.

:exclamation: Current head eb5085f differs from pull request most recent head 1196188. Consider uploading reports for the commit 1196188 to get more accurate results

@@             Coverage Diff              @@
##                dev   #13621      +/-   ##
============================================
- Coverage     39.05%   38.97%   -0.09%
+ Complexity     4419     4398      -21
============================================
  Files          1127     1127
  Lines         42096    42062      -34
  Branches       4775     4755      -20
============================================
- Hits          16440    16393      -47
- Misses        23842    23855      +13
  Partials       1814     1814

Impacted Files
Coverage Δ

...r/plugin/task/sqoop/parameter/SqoopParameters.java
1.19% <0.00%> (-35.72%)
:arrow_down:

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...erver/master/processor/queue/TaskEventService.java
75.00% <0.00%> (-5.36%)
:arrow_down:

...hinscheduler/plugin/alert/script/ScriptSender.java
70.58% <0.00%> (-1.64%)
:arrow_down:

...cheduler/plugin/alert/dingtalk/DingTalkSender.java
34.13% <0.00%> (-0.78%)
:arrow_down:

...rver/master/runner/task/BlockingTaskProcessor.java
75.86% <0.00%> (-0.55%)
:arrow_down:

.../dolphinscheduler/plugin/task/spark/SparkTask.java
68.69% <0.00%> (-0.54%)
:arrow_down:

...inscheduler/plugin/registry/etcd/EtcdRegistry.java
50.69% <0.00%> (-0.35%)
:arrow_down:

.../dolphinscheduler/plugin/task/datax/DataxTask.java
36.62% <0.00%> (-0.26%)
:arrow_down:

...eduler/api/dto/workflow/WorkflowCreateRequest.java
100.00% <0.00%> (ø)

... and 18 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


maybe lock the icons (reload/back/icao information) on the left side, instead of repeating top/bottom.

will try and test a different layout for the icons, locked on the left side of the screen.

feature added on 1.1.3: https://github.com/jflores82/simchecklists/commit/f533cfa2f9e764a1c5204cdef89dbb5b92970771
icons change position based on the screen resolution.


Faster and better location data

Checklist

[x] I have read the Contributor Guide
[x] I have read and agree to the Code of Conduct
[x] I have added a description of my changes and why I'd like them included in the section below

Summary
Before:

After:

Description of Changes
This PR:

Improve the function names for trait calls
Adds crate name to the file path
Adds line number to the location
Optimize profile_function

On my M1 MacBook Pro, profile_function goes from 86ns->58ns, about 50% faster. profile_scope goes from 60ns to 58ns, which is within noise levels.
The optimization is done by caching the results of the location gathering functions. This also allows us to make them more expensive (since they are now only run once), and therefor more powerful.
So where puffin used to output the location mod.rs, it now outputs cratename/…/modulename/mod.rs:124.
I also improved the function name for when calling a trait method on a type. Previously you'd only see TraitName>::function_name but now you'll see <ConreteTypeName as TraitName>::function_name.
The downside is more data being sent, but I think the improves usefulness of the location is worth it!
This is not a breaking change, so this can be released in a patch-release of puffin

Nice! I have a branch here also with some other optimizations I want to finish up ASAP https://github.com/EmbarkStudios/puffin/tree/perf-improvement-test

I'll have a look at removing unsafe access and maybe just going for the stream-info-once solution as proposed in #167


feature: Implement voting function in Hot section

note: We have a function to make this work. This placeholder does not follow that pattern. Despite this being mocked data, we could still makes this work live.
Originally posted by @bdougie in https://github.com/open-sauced/hot/pull/216#discussion_r927049498

Hi @bdougie
If I may ask, is the code block below the function you are talking about??
  async function handleVoteUpdateByRepo(votes: number, repo_id: number) {
    user_id && capturePostHogAnayltics('User voted', 'voteClick', 'true');

    const updatedVotes = await updateVotesByRepo(votes, repo_id, user_id);
    updateVotesState(updatedVotes);
  }

Yes

Great
will make the needed changes on the hot repositories section

I get this error in my chrome dev tools whenever I try to upvote any of the repos.
The numbers will increase and upon refreshing of the page, it will return back to the default

I get this error in my chrome dev tools whenever I try to upvote any of the repos.
The numbers will increase and upon refreshing of the page, it will return back to the default

This one is for me, will check it out now!

I believe this works based on the #216 merge. It did introduce this bug https://github.com/open-sauced/hot/issues/230


Override default selection background for active match

When the editor is focused and .ProseMirror-active-search-match exists, the selection background color (usually in light blue) and the orange active decoration will overlap. This PR fixes the issue by only applying background color to ::selection.
A tiny issue is that, in Chrome, the <span> and ::selection has different height, which causes the orange .ProseMirror-active-search-match is slightly higher than the yellow .ProseMirror-search-match. I think this doesn't bother users. Just FYI.
Before
https://github.com/ProseMirror/prosemirror-search/assets/24715727/68614750-d7ad-4666-9d45-139f7ce78759
After
https://github.com/ProseMirror/prosemirror-search/assets/24715727/8a0cbc4e-e2b6-433a-86cd-838c7e38b1ec

I don't think it is a good idea to change the color of the selection when it is on a search match. I'm not sure the overlapping backgrounds are an issue at all.

The .ProseMirror-active-search-match class only appears when the text selection exactly matches it. This means the orange color will never be shown without mixing to users, which seems odd from a design perspective.
Additionally, I noticed that the background of the text selection and the span decoration have different heights, creating some padding, shown below:

These issues wouldn't arise if the text selection and .ProseMirror-active-search-match don't need to cover the same range. It seems that most rich text editors like Google Docs and MS Word, unlike code editors, don't change the selection while searching, so they don't have this color overlapping issue.
That said, I'm satisfied with the current API. It's flexible, allowing me to easily implement the styling on my end. It also makes sense that this library provides basic implementation and lets users do further customization. Feel free to close this PR.


[FEATURE REQ] Configurable Connection timeout for eventhub

Library name
Azure.Messaging.EventHubs 5.8.1
Please describe the feature.
Due to satellite connection and a congested network channel, we're constantly hitting an eventhub connection timeout limit.
Switching to a better connection is not a viable option.
With current version of EventHub library, there's no way to configure a custom timeout for AMQP connection setup.
The default timeout is set to 30 seconds, but it's hardwired in property AmqpConnectionScope.SessionTimeout
Manually changing just the parameter to 1 minute and recompiling the Eventhub assembly solved our problem.
Can you please make this parameter user configurable ?
Best regards,
Stefano

Hi @stefanop.  Thank you for reaching out and we regret that you're experiencing difficulties.  Can you help me understand your end-to-end scenario better?  Specifically, I'd like to see the error message and stack trace that you're running into.
All Event Hubs service operations - including establishing a connection - are intended to be governed by the TryTimeout of the retry policy of the client.  You're correct in that we're creating a hard-cap of 30 seconds when opening AMQP links, which I've traced back to behavior from the previous generation that we brought forward - but I honestly have no idea why.   I'm going to try to run that down and will verify with the Event Hubs service team what the impact of removing that cap would be.
If all goes well, we'll look to either remove the cap being imposed and allow TryTimeout to perform its intended role or introduce a new connection option that governs just establishing connections/links.

//fyi: @m-redding

Hi @stefanop.   After looking things over, the 30 second cap was a mistake carried forward from the previous generation library which is not needed and runs counter to our goal of having a single timeout govern each service operation end-to-end.  We've decided to remove that 30 second cap, which will allow the retry policy's TryTimeout to take over.   In your scenario, if you need more than the default 60 seconds, you can extend the timeout using the RetryOptions when creating the client.   An example can be found here.
The change will go out with our next release, tentatively scheduled for the second week of May.   Thanks for bringing this to our attention.   I'm going to mark this as addressed, but please feel free to unresolve if you'd like to continue the discussion.


Move prebuilt activities to the person-risk card.

It was previously below the call to action.
Make some related UX tweaks too:

adjust header styles to deemphasize location and highlight
activity/relationship now that location autofills

remove explicit step numbers now that location autofills

adjust prebuilt activities box instructions so that the most important
words for understanding are clustered at the beginning of the phrase

add magnifying glass affordance to prebuilt activity search to more
strongly indicate that it's for typing (it's indistinguishable from
the other dropdowns. This isn't a perfect solution but will do for now.)

I took the colour changes out of this particular PR and will incorporate them later. I think those are better grouped together elsewhere as general focus management improvements.

I took the colour changes out of this particular PR and will incorporate them later. I think those are better grouped together elsewhere as general focus management improvements.

Ah, I forgot about input group prepends in Bootstrap. Good use! Thanks for putting in that fix! I'll merge now.

Ah, I forgot about input group prepends in Bootstrap. Good use! Thanks for putting in that fix! I'll merge now.


Work on Pink Theme

Work on Pink Theme

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add add_default_auth_status method

The purpose of this method is to let the UI detect the status of the
default auth_status of the provider.
In a specific case there is a need to override it with 'true' instead
of relying on the value of default_auth_status_ok? :
Kubevirt provider can be managed both as an infrastructure provider and
as a sub-provider of the container manager. When it is being managed as
an infra provider, its auth_type is 'kubevirt' and the default_auth_status
refers to the parent provider.
There can be a case in which the parent provider is invalid, while the
kubevirt provider is valid.
Therefore, in that case, the default_auth_status should be 'true', to
allow editing the kubevirt infra provider.
The method will be used by https://github.com/ManageIQ/manageiq-ui-classic/pull/4137/files#diff-f402e411cd6dc158c724b14d72ead612R861
Fixes https://github.com/ManageIQ/manageiq-providers-kubevirt/issues/71

@himdel @agrare the purpose of this PR is to achieve the same as intended originally by https://github.com/ManageIQ/manageiq-ui-classic/pull/4069
The new method will be used from https://github.com/ManageIQ/manageiq-ui-classic/pull/4137/files#diff-f402e411cd6dc158c724b14d72ead612R861
and will be overridden by kubevirt's provider:
https://github.com/ManageIQ/manageiq-providers-kubevirt/pull/93/files#diff-ff17b05a7c525eec833f0ddb0134a0d9R122

@agrare
I noticed that 'default' as method's prefix already exists authentication_mixin.rb, i.e. default_authentication_type and default_authentication, but I understand your point about being too specific in a method that can be generalized.

Why not allow editing of the kubevirt provider if the auth for a different provider is invalid?

That's exactly what I'm trying to achieve with this and the UI patches:
The default_auth_status is false if the parent container provider is invalid. That causes the 'Save' button to be disabled, as not all dialog validations are passed. So even if kubevirt is valid and should be allowed to be saved, it's blocked.
What is needed here is an option to pass 'true' value to default_auth_status when the context is add/edit ems_infra and provider type is kubevirt. That way only kubevirt's properties are considered.
The other alternative I can see is cloning the entire section for ems_type 'infra' and for 'kubevirt' provider's type:  https://github.com/ManageIQ/manageiq-ui-classic/blob/master/app/controllers/mixins/ems_common_angular.rb#L397
till line https://github.com/ManageIQ/manageiq-ui-classic/blob/master/app/controllers/mixins/ems_common_angular.rb#438 and replace the 'default_auth_status' with true.
However, that will add more provider specific logic to the UI and I assume this is also not desired based on input from @himdel
The last option will be to disable entirely the ability to edit the kubevirt provider as an infra provider, so it can only be modified from the'virtualization' sub-tab of its parent container provider.

That's exactly what I'm trying to achieve with this and the UI patches:
The default_auth_status is false if the parent container provider is invalid. That causes the 'Save' button to be disabled, as not all dialog validations are passed. So even if kubevirt is valid and should be allowed to be saved, it's blocked.

I was thinking more along the lines of why is the UI not allowing a manager to be saved if another manager's auth is invalid?  Adding a method that we force to be true sounds like a workaround.  Maybe this is a workaround that is needed currently but I'm curious about @himdel's thoughts about this.  Is this by design or just something that is working this way.

The other alternative I can see is cloning the entire section for ems_type 'infra' and for 'kubevirt' provider's type

Yeah definitely not this :)

I was thinking more along the lines of why is the UI not allowing a manager to be saved if another manager's auth is invalid?

Details?
What are you referring to?
In the UI code, @ems is the manager being edited, not some other manager.

Yes, that's why I'm asking for details, since that's the first time I hear about this.
Any idea what the other manager is?

@masayag can you help here?  I assume the parent manager is kubernetes.

@agrare @himdel The parent provider is the container manager (kubernetes/openshit).
I've added a screenshot to show the problem.
It used to happen since the @ems.authentication_status_ok? was delegated to the parent provider.
In a PRs added recently (1 and 2), that method is no longer delegated to the parent provider. My bad for not mentioning it earlier and derailing the issue from the current cause of it.
When the 'edit' infra provider is loaded for kubevirt provider with an invalid token, the 'default_auth_status' is set to false, and will remain so, even when setting a valid token as the screenshot shows. Kubevirt authentication status should be determined by kubevirt_auth_status and not by the default_auth_status:

OK.. and why does Kubevirt provider have kubevirt_auth_status and not default_auth_status?
I would argue that if kubevirt anything has a kubevirt_something method, that is indeed the default.
Are you using default_auth_status for something else in Kubevirt?

Basically .. I don't see the point of this PR.
I'm almost sure authentication_status_ok? is the one that should know which authentication we're using and check that authentication's stauts, or am I wrong?

Yeah I agree @himdel maybe the delegated parent kubernetes credentials should be :auth_type => :kubernetes and the default should be kubevirt for the kubevirt provider?

Since kubevirt is managed both as a sub-provider and an infra provider, the 'default_' prefix is reserved to the parent provider (i.e. openshift) and kubevirt auth/endpoint attributes are identified by 'kubevirt_' prefix.
The default_auth_status isn't being used by kubevirt.

Since kubevirt is managed both as a sub-provider and an infra provider

What does this mean exactly?  Are there two different ways to deploy a kubevirt provider (i.e. standalone or as part of an openshift provider)?

I'm almost sure authentication_status_ok? is the one that should know which authentication we're using and check that authentication's stauts, or am I wrong?

The problem that default_auth_status is called once when the dialog is loaded.
If it is loaded when kubevirt's token is invalid, it is set to false which disabled the Save button.
Even if I enter a valid token and validate it, the default_auth_status remains false.
So there is no option to save a valid token if the existing token is invalid.
I can record a short video or provide an access to the env to make it clearer.
authentication_status_ok? is used for validating the status of the kubevirt provider.

What does this mean exactly? Are there two different ways to deploy a kubevirt provider (i.e. standalone or as part of an openshift provider)?

There is only one way to deploy kubevirt, via the virtualization tab of the openshift container.
Once kubevirt provider is added, you can view it under the virtualization tab or as an infra provider (to rename it or its token).

@masayag but default_auth_status just checks authentication_status_ok? so if you validate a good token won't that return true?

What does this mean exactly? Are there two different ways to deploy a kubevirt provider (i.e. standalone or as part of an openshift provider)?

There is only one way to deploy kubevirt, via the virtualization tab of the openshift container.
Once kubevirt provider is added, you can view it under the virtualization tab or as an infra provider (to rename it or its token).

Then I'm not sure what you mean by being managed as a sub-provider and as an infra manager.

@masayag but default_auth_status just checks authentication_status_ok? so if you validate a good token won't that return true?

authentication_status_ok? will return true only if the token that was saved in the DB is valid.
However, the dialog is loaded with 'Save' button disabled (since the current token in the DB is disabled), so there is no option to save the valid token.

Then I'm not sure what you mean by being managed as a sub-provider and as an infra manager.

sub-provider - when adding a container provider (let's call it the primary provider) and later on adding kubevirt to it. Kubevirt's endpoints are delegated to parent provider, as it determines the server details.
So maybe it is more like 'child-provider' since its parent is the container manager.
infra manager - that's the actual role of the kubevirt provider - to allow managing vms (ATM), same as oVirt provider.

authentication_status_ok? will return true only if the token that was saved in the DB is valid.
However, the dialog is loaded with 'Save' button disabled (since the current token in the DB is disabled), so there is no option to save the valid token.

Okay, not sure but this sounds like a UI issue, maybe? IDK
Is it because authentication_status_ok? checks the default creds and you're delegating the default creds?

sub-provider - when adding a container provider (let's call it the primary provider) and later on adding kubevirt to it. Kubevirt's endpoints are delegated to parent provider, as it determines the server details.
So maybe it is more like 'child-provider' since its parent is the container manager.
infra manager - that's the actual role of the kubevirt provider - to allow managing vms (ATM), same as oVirt provider.

Not sure how this has any impact on why kubevirt delegates default credentials to the kubernetes provider

@agrare originally I wanted to handle it on UI by https://github.com/ManageIQ/manageiq-ui-classic/pull/4069/files
I'll try to come up with a different PR for UI. Anyway, closing this one. Thanks for your feedback.


Refactor: import @walletconnect/ethereum-provider dynamically

Description
The @walletconnect/ethereum-provider package is a big dependency, and it was imported twice: statically and dynamically.
I've moved the static import to the dynamic one.

PLEASE NOTE- Checklist must be complete prior to review.
Checklist

[x] Increment the version field in package.json of the package you have made changes in following semantic versioning and using alpha release tagging
[x] Check the box that allows repo maintainers to update this PR
[x] Test locally to make sure this feature/fix works
[x] Run yarn check-all to confirm there are not any associated errors
[ ] Confirm this PR passes Circle CI checks

@Adamj1232 please let me know if I need to adjust anything. Cheers

Thanks @katspaugh! Ill give er a test first thing tomorrow! Cheers

@katspaugh after digging in there were a few other updates for the WC module. I have added them in this PR - https://github.com/blocknative/web3-onboard/pull/2036

Cool, thank you! I'll close this then :)


Bug:  Volvo Trucks Regression issues

How to reproduce

Go to the Volvo Home Page.
Check the footer.
Navigate to a different page (e.g. any PDP).
Check the footer.

Expected Result
The footer looks the same in all pages.
Actual Result
The labels from the footer in pages that are not the Home Page are in a smaller size and the text in the bottom of the footer also looks off.

confirmed as fixed
as tested on https://release-2--vg-volvotrucks-us-rd--netcentric.hlx.page/pdp


popup.addTo(scene)  报错

ncaught (in promise) TypeError: Cannot read property 'get' of undefined
at Popup.addTo (popup.js?5980:45)
at Popup.open (popup.js?5980:67)
at eval (Home.vue?76f2:192)
at Array.forEach ()
at _callee2$ (Home.vue?76f2:176)
at tryCatch (runtime.js?98b8:45)
at Generator.invoke [as _invoke] (runtime.js?98b8:271)
at Generator.prototype. [as next] (runtime.js?98b8:97)
at asyncGeneratorStep (asyncToGenerator.js?1da1:3)
at _next (asyncToGenerator.js?1da1:25)

2.0 版本已经去掉了 addTo
通过Scene.addPopup() 添加
https://l7.antv.vision/zh/docs/api/component/popup/#添加-popup


Refreshing encryption key and policy caches on resource deletion

Refreshing encryption key and policy caches on resource deletion
.NET PR - https://github.com/Azure/azure-cosmos-dotnet-v3/pull/2403
which depends on https://github.com/Azure/azure-cosmos-dotnet-v3/pull/2371

fixed with https://github.com/Azure/azure-sdk-for-java/pull/22616


register_content to pass appropriate flags.

A change was made in https://github.com/StackStorm/st2/commit/239d6f453f937f51328e7c9f35a8076f81910ea2 than needs to be moved to the st2_deploy.sh command. Repairs actions not properly registering on first-install, needing to run st2ctl reload to repair.
/cc @Kami @DoriftoShoes

/cc https://github.com/StackStorm/st2/pull/932

Good catch - I forgot to update st2_deploy script.
LGTM, +1


Updated Binder to use ccd4869e0971c7cccb43d6fd06b3f1de71d2543f.

Modern Computer Now

The ★binder branch is a special branch used to deploy to mybinder.org, and is not intended for merging to master. We'll work to clarify this in the future, thanks!


Add unique_id to yeelight light

Breaking Change:
None
Description:
This PR adds unique_id property to yeelight light entities. MAC address of the device is used.
Checklist:

[X] The code change is tested and works locally.
[X] Local tests pass with tox. Your PR cannot be merged unless tests pass
[X] There is no commented out code in this PR.
[X] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[X] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[X] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

There is no internal serial number that could be used instead?

There is no internal serial number that could be used instead?

The yeelight library provides light IDs (something like serial number) only during discovery - the ID isn't Bulb class property. The component during initialization of each light entity would have to discover all lights in the local network and match the ID to the bulb based on the IP address. Using the MAC address seems to be a simpler solution.

For me it looks good. Did you tested it with devices added in config, but unavailable during HA setup ? What happens then.

This will not work for those users who have their devices in other networks. That being said, I just checked with my devices (color, bslamp1) and they are responsive for the discovery via unicast, so I'm planning to pursue that road after/if I get a confirmation from the yeelight devs: https://forum.yeelight.com/t/topic/12894/383

I checked, you are right, getmac doesn't work with devices in subnetworks. A lot of peoples use separate subnetworks for IoT devices. This PR may cause a lot troubles. I will close it and try once again with bulb ID. Thanks.


PanicException: called Result::unwrap() on an Err value: H5Dread(): can't read data: file read failed

Dear CEMBA team,
I am trying to write fragments bed files to disk through an AnnDataSet object (with all 2.3M cells from all .h5ad sample files; AnnDataSet object with n_obs x n_vars = 2355842 x 5451055) using SnapATAC 2.4 below. ‘Class’ is a manually added column, using AIT21_annotation_freeze_081523.tsv (under this folder) for class-to-subclass conversion.
snapatac2.ex.export_bed(adataset,
                       groupby='Class',
                       suffix='.bed.gz',
                       out_dir = class_level_fragments_dir
                       )

and encountered the following error (I’ve replaced my personal paths with /path/to/):
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: H5Dread(): can't read data: file read failed: time = Sun Dec 17 07:33:29 2023
, filename = ‘/path/to/CEMBA180129_3A_rm_dlt.h5ad', file descriptor = 116, errno = 5, error message = 'Input/output error', buf = 0x50871390, total read size = 2096, bytes this sub-read = 2096, bytes actually read =<PHONE_NUMBER>3709551615, offset = 0', /root/.cargo/git/checkouts/anndata-rs-087569d311f7caf2/d54b207/anndata/src/container/base.rs:1160:85
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[51], line 5
      3 print(f'Writing outputs to f{class_level_fragments_dir}')
      4 print('='*50)
----> 5 snapatac2.ex.export_bed(adataset,
      6                        groupby='Class',
      7                        suffix='.bed.gz',
      8                        out_dir = class_level_fragments_dir
      9                        )

File ~/miniconda3/envs/conda_env/lib/python3.8/site-packages/snapatac2/export/__init__.py:55, in export_bed(adata, groupby, selections, ids, out_dir, prefix, suffix)
     52 elif isinstance(ids, str):
     53     ids = adata.obs[ids]
---> 55 return internal.export_bed(
     56     adata, list(ids), list(groupby), selections, out_dir, prefix, suffix,
     57 )

PanicException: called `Result::unwrap()` on an `Err` value: H5Dread(): can't read data: file read failed: time = Sun Dec 17 07:33:29 2023
, filename = ‘/path/to/CEMBA180129_3A_rm_dlt.h5ad', file descriptor = 116, errno = 5, error message = 'Input/output error', buf = 0x50871390, total read size = 2096, bytes this sub-read = 2096, bytes actually read =<PHONE_NUMBER>3709551615, offset = 0

I have verified that each cell is assigned the appropriate class label. At the time of this error (which occurred after hours of writing to disk), the line counts of the written bed.gz files do not match the number of fragments (grouped by class) in the metadata (Supplementary Table 2 of the paper), as expected.
Counting the number of lines (at the time of error) using zcat <filename> | wc -l yields:
{'HY_GABA.bed.gz': 226103206,
 'CTX-CGE_GABA.bed.gz': 290152494,
 'CTX-MGE_GABA.bed.gz': 550481310,
 'MH-LH_Glut.bed.gz': 34759628,
 'NP-CT-L6b_Glut.bed.gz': 996287794,
 'MY_Glut.bed.gz': 79333970,
 'Astro-Epen.bed.gz':<PHONE_NUMBER>,
 'CNU-HYa_Glut.bed.gz': 166911254,
 'LSX_GABA.bed.gz': 147192400,
 'DG-IMN_Glut.bed.gz': 336481046,
 'HY_Gnrh1_Glut.bed.gz': 783042,
 'OB-CR_Glut.bed.gz': 6790320,
 'CNU-MGE_GABA.bed.gz': 93358728,
 'MB_Glut.bed.gz': 815605023,
 'MY_GABA.bed.gz': 113820670,
 'HY_Glut.bed.gz': 177754869,
 'TH_Glut.bed.gz': 455732012,
 'CNU-LGE_GABA.bed.gz': 403055016,
 'P_GABA.bed.gz': 133969424,
 'Immune.bed.gz': 234100458,
 'Vascular.bed.gz': 262067892,
 'CB_Glut.bed.gz': 476247064,
 'OPC-Oligo.bed.gz':<PHONE_NUMBER>,
 'IT-ET_Glut.bed.gz':<PHONE_NUMBER>,
 'CNU-HYa_GABA.bed.gz': 285071090,
 'CB_GABA.bed.gz': 57209872,
 'MB_Dopa.bed.gz': 50937284,
 'P_Glut.bed.gz': 233308761,
 'MB_GABA.bed.gz': 572258267,
 'OB-IMN_GABA.bed.gz': 50274352,
 'MB-HB_Sero.bed.gz': 25413152,
 'OEC.bed.gz': 509394,
 'Pineal_Glut.bed.gz': 1446016}

while grouping the metadata by class (for all 2.3M cells) yields
Class
IT-ET Glut<PHONE_NUMBER>
OPC-Oligo<PHONE_NUMBER>
Astro-Epen<PHONE_NUMBER>
NP-CT-L6b Glut     784204001
MB Glut            750316389
CNU-LGE GABA       493293499
MB GABA            448868487
CTX-MGE GABA       443671252
CB Glut            427319942
TH Glut            356853925
DG-IMN Glut        339200608
P Glut             264327471
CNU-HYa GABA       238441380
Vascular           237619853
CTX-CGE GABA       233809950
Immune             210318453
HY GABA            205488208
LSX GABA           194601248
MY GABA            162226871
HY Glut            161648292
CNU-HYa Glut       143951176
P GABA             126838072
CNU-MGE GABA       115760498
MY Glut            105435880
OB-IMN GABA         72832099
MB Dopa             54655953
CB GABA             50524434
MH-LH Glut          34059796
MB-HB Sero          26239541
OB-CR Glut           7518986
Pineal Glut          2123205
OEC                   851273
HY Gnrh1 Glut         785959
Name: # of Fragments, dtype: int64

Strangely, I do notice that for the HY GABA class, the number of written lines (226103206) exceeds the number of cells belonging to that class in the metadata (205488208) by ~20M entries, so I’m not sure what is going on there.
I'm also not sure if this is due to the particularly large files, but the error occurred after much saving has already occurred. Would appreciate any thoughts on this!
Best,
Jay

@jayluo2
Hi Jay,

May I know if only "CEMBA180129_3A_rm_dlt.h5ad" has the reading error?
Since you have error on at least one file, I am not sure if there are some files failed so we have less number of fragments?

Songpeng

@jayluo2
This is how I export bed files for peak calling:
https://github.com/beyondpie/CEMBA_wmb_snATAC/blob/main/03.peakcalling/src/main/python/prepare_bedfiles.py
I use Snakefile to organize this, so that multiple groups can be run simultaneously.

I believe this would only be the first file that produced the error (since these are not just warnings), so I’m not sure if there are additional flies that may contribute. I’ve separately loaded “CEMBA180129_3A_rm_dlt.h5ad” using snapatac2.read() and the contents appear normal:

AnnData object with n_obs × n_vars = 7350 × 5451055
    obs: 'tsse', 'n_fragment', 'frac_dup', 'frac_mito', 'doublet_probability', 'doublet_score'
    var: 'count', 'selected'
    uns: 'reference_sequences', 'scrublet_sim_doublet_score'
    obsm: ‘insertion'

anndata.X yields
<7350x5451055 sparse matrix of type '<class 'numpy.uint32'>'
	with 56624600 stored elements in Compressed Sparse Row format>

and the .obs slot looks fine as well.

The HY GABA case I pointed out is strange because the # of written lines (in bed.gz file) should be at most the # of fragments (of HY GABA class) from the metadata (Supplementary Table 2), so it is weird that for HY GABA, more fragments are written to disk (226103206 lines) than are present in the metadata (205488208 fragments).

@jayluo2

So I am not sure what happened on the files. It looks OK from my view.
@wkl1990 Kangli, do you have any comment about this? Is this because of filtering of blacklist regions?

@beyondpie .obsm['insertion’] on this h5ad file (CEMBA180129_3A_rm_dlt.h5ad) returns
<7350x2725521370 sparse matrix of type '<class 'numpy.uint8'>'
	with 95603440 stored elements in Compressed Sparse Row format>

using export_bed() with a column of ‘mock labels’ (alphabetical order) works just fine as well, with the following command (“anndata” = same h5ad file)
snapatac2.ex.export_bed(anndata,
                              groupby='mock_labels',
                             out_dir=output_dir,
                       suffix='.bed.gz')

The fragments files also appear to be fine. e.g.
zcat a.bed.gz | head
chr1	4186842	4186843	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	4187029	4187030	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	4187030	4187031	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	4187283	4187284	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	5070349	5070350	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	5070467	5070468	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	7138690	7138691	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	7138914	7138915	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	7378213	7378214	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA
chr1	7378408	7378409	AGCGATAGAACCAGGTGAAGTATGGGCTCTGA

and
zcat u.bed.gz | head
chr1	3588059	3588060	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	3588103	3588104	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	5017916	5017917	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	5017979	5017980	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	7397876	7397877	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	7397997	7397998	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	22296964	22296965	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	22297030	22297031	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	24611574	24611575	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC
chr1	24611626	24611627	TCTCGCGCAAGAGGCAGCGTAAGAGTACTGAC

@beyondpie Regarding your second question, I identified some more discrepancies similar to HY GABA above:
Discrepancy for HY_GABA: 205488208 fragments (metadata), but class-level bed.gz file has 226103206 rows
Discrepancy for CTX-CGE_GABA: 233809950 fragments (metadata), but class-level bed.gz file has 290152494 rows
Discrepancy for CTX-MGE_GABA: 443671252 fragments (metadata), but class-level bed.gz file has 550481310 rows
Discrepancy for MH-LH_Glut: 34059796 fragments (metadata), but class-level bed.gz file has 34759628 rows
Discrepancy for NP-CT-L6b_Glut: 784204001 fragments (metadata), but class-level bed.gz file has 996287794 rows
Discrepancy for Astro-Epen:<PHONE_NUMBER> fragments (metadata), but class-level bed.gz file has<PHONE_NUMBER> rows
Discrepancy for CNU-HYa_Glut: 143951176 fragments (metadata), but class-level bed.gz file has 166911254 rows
Discrepancy for MB_Glut: 750316389 fragments (metadata), but class-level bed.gz file has 815605023 rows
Discrepancy for HY_Glut: 161648292 fragments (metadata), but class-level bed.gz file has 177754869 rows
Discrepancy for TH_Glut: 356853925 fragments (metadata), but class-level bed.gz file has 455732012 rows
Discrepancy for P_GABA: 126838072 fragments (metadata), but class-level bed.gz file has 133969424 rows
Discrepancy for Immune: 210318453 fragments (metadata), but class-level bed.gz file has 234100458 rows
Discrepancy for Vascular: 237619853 fragments (metadata), but class-level bed.gz file has 262067892 rows
Discrepancy for CB_Glut: 427319942 fragments (metadata), but class-level bed.gz file has 476247064 rows
Discrepancy for OPC-Oligo:<PHONE_NUMBER> fragments (metadata), but class-level bed.gz file has<PHONE_NUMBER> rows
Discrepancy for IT-ET_Glut:<PHONE_NUMBER> fragments (metadata), but class-level bed.gz file has<PHONE_NUMBER> rows
Discrepancy for CNU-HYa_GABA: 238441380 fragments (metadata), but class-level bed.gz file has 285071090 rows
Discrepancy for CB_GABA: 50524434 fragments (metadata), but class-level bed.gz file has 57209872 rows
Discrepancy for MB_GABA: 448868487 fragments (metadata), but class-level bed.gz file has 572258267 rows

I agree with you and think the source of these discrepancies must be in how the "# of Fragments” column in Supplementary Table 2 is generated. Do you have thoughts on which one of the two (i.e. Supplementary Table 2 or bed.gz line counts from export_bed()) is the most filtered/updated version?

@jayluo2
Thank you so much for the feedback!

So you can actually get the bed files, then I am a little bit confused with your first comment about the error. If you later notice that there is no such error any more, that would be great!
I think the hdf5 files I shared are the ones we used. So you can use them for later analysis. Just remember that

we did not filter the black list for the fragments. (Here is one black list file you can use: https://github.com/beyondpie/CEMBA_wmb_snATAC/blob/main/meta/mm10.blacklist.bed)
please check if mitochondrial fragments and/or the fragments not from Chr1-Chr19, X and Y are kept. For our analysis, we only use the fragments from Chr1-Chr19, Chr X and Chr Y.
Here is how I got the raw h5ad without doublet removal:https://github.com/beyondpie/CEMBA_wmb_snATAC/blob/8d090924b7a75ef6bf12f6a6ee396092d9437559/00.data.preprocess/sa2.01.preprocess.py#L98C9-L98C9.

I will keep this open and see if @wkl1990 later will give us some feedback. But currently, I suggest you just use the ones you got and temporally ignore the inconsistencies between yours and the ones in Supplementary File 2.
Sincerely,
Songpeng

@beyondpie Thank you for the details!

I suppose the discrepancies in from Supplementary Table 2 and the fragments stored in the h5ad files may be due to the filtering steps. The "Feature selection” section of the paper appears to state that blacklist was used to filter out genomic bins (which I presume are used for subsequent clustering), and so the actual Tn5 insertions in the .h5ad files on catlas.org were not removed for blacklist overlap?
Based on some spot checks, I don’t think there are mitochondrial fragments in the fragments files.
I am able to use ‘groupby’ in export_bed() to write class-grouped fragments bed files to disk, but the error I initially mentioned (PanicException) do not allow fragments to be written to completion. I’m not sure if this error is related to the fact that the Tn5 insertions in the h5ad files are not filtered for the mm10 blacklist.

@jayluo2

Based on the link I shared with you, I should include all the fragments located in Chr1-19, ChrX and ChrY without any blacklist filtering. I am not sure about the discrepancies you saw, which I need to check with my colleague.
Good.
I haven't had the failures before. But I usually store the data into subclass or L4 level. So not sure if class-level is kind of too big to be written into a file (or memory or other issue)? Would you mind to test to work on with smaller groups?

Best,
Songpeng

@beyondpie
Thank you for clarifying! I tried grouping by subclass with the same command,
snapatac2.ex.export_bed(adataset,
                       groupby='Subclass',
                       suffix='.bed.gz',
                       out_dir = output_dir
                       )

but received the same error:
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: H5Dread(): can't read data: file read failed: time = Wed Dec 20 16:47:56 2023
, filename = ‘/path/to/CEMBA181022_6B_rm_dlt.h5ad', file descriptor = 142, errno = 5, error message = 'Input/output error', buf = 0x62c1890f, total read size = 33416, bytes this sub-read = 33416, bytes actually read =<PHONE_NUMBER>3709551615, offset = 0', /root/.cargo/git/checkouts/anndata-rs-087569d311f7caf2/d54b207/anndata/src/container/base.rs:1160:85
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[40], line 5
      3 print(f'Writing outputs to f{subclass_level_fragments_dir}')
      4 print('='*50)
----> 5 snapatac2.ex.export_bed(adataset,
      6                        groupby='Subclass',
      7                        suffix='.bed.gz',
      8                        out_dir = subclass_level_fragments_dir
      9                        )

File ~/miniconda3/envs/ledidi/lib/python3.8/site-packages/snapatac2/export/__init__.py:55, in export_bed(adata, groupby, selections, ids, out_dir, prefix, suffix)
     52 elif isinstance(ids, str):
     53     ids = adata.obs[ids]
---> 55 return internal.export_bed(
     56     adata, list(ids), list(groupby), selections, out_dir, prefix, suffix,
     57 )

PanicException: called `Result::unwrap()` on an `Err` value: H5Dread(): can't read data: file read failed: time = Wed Dec 20 16:47:56 2023
, filename = ‘/path/to/CEMBA181022_6B_rm_dlt.h5ad', file descriptor = 142, errno = 5, error message = 'Input/output error', buf = 0x62c1890f, total read size = 33416, bytes this sub-read = 33416, bytes actually read =<PHONE_NUMBER>3709551615, offset = 0

I am not sure if this is due to any particular h5ad file. Would it be possible to upload subclass-level fragments bed files to catlas.org?
Best,
Jay

@jayluo2
Sorry to hear that you still have the errors.

Would you mind to share the whole script with me? Let me read your codes firstly.
I will say if I can share you the h5ad files through Google Drive link, and you can later try that again. Ideally, they should be the same one, but let's just double check this.
@wkl1990 Would you mind to upload the subclass-level bed files (if catlas is not easy, we can have a google drive link, and share that with @jayluo2 firstly).

Songpeng

@beyondpie Apologies for the delay in uploading bed files. I will upload them to Catlas and notify you once it's completed.

@beyondpie @wkl1990 Thank you both for your help, and that all sounds great!
@beyondpie Below is the code:
import os
import numpy as np
import pandas as pd
import snapatac2

# paths
cell_level_metadata_txt_path = os.path.join(‘path/to/Supplementary_Table_2.txt')
subclass_class_mapping_metadata_txt_path = os.path.join(‘path/to/AIT21_annotation_freeze_081523.tsv')
raw_anndata_dir = ‘/path/to/raw_anndata_files'
anndataset_dir = ‘/path/to/save/anndataset_object'
output_dir = ‘/path/to/output/fragments‘

# Load metadata (all 2.3M cells)
whole_mouse_brain_2023_metadata_df = pd.read_table(cell_level_metadata_txt_path)

# Get a list of absolute paths to sample h5ad files
adata_paths_l = [os.path.join(raw_anndata_dir, filename) for filename in raw_h5ad_filenames_l]
adatas_l = [(path.split('/')[-1].split('_rm_dlt.h5ad')[0], path) for path in adata_paths_l]

# Get AnnDataSet object
adataset = snapatac2.AnnDataSet(
    adatas=adatas_l,
    filename=anndataset_path
)

# Ensure ‘obs_names’ are unique
# use "." delimiter because that's what's done w/ cell-level metadata
unique_cell_ids = [sa + '.' + bc for sa, bc in zip(adataset.obs['sample'], adataset.obs_names)]
adataset.obs_names = unique_cell_ids
assert adataset.n_obs == np.unique(adataset.obs_names).size

# Get Cell-to-Subclass mapping
cellid_to_subclass_dict = dict(zip(whole_mouse_brain_2023_metadata_df['CellID'], whole_mouse_brain_2023_metadata_df['Subclass']))

# Create a Series from adataset.obs_names
cell_barcodes_series = pd.Series(adataset.obs_names)

# Map the ’Subclass' from 'cellid_to_subclass_dict' to the Series
subclass_labels = cell_barcodes_series.map(cellid_to_subclass_dict)

# Replace all spaces with underscores in the 'Class' column
subclass_labels = subclass_labels.str.replace(' ', '_', regex=False)
assert subclass_labels.isna().any() == False

# Add the class labels as a new column to adataset.obs
adataset.obs['Subclass'] = subclass_labels

# Export subclass-level fragments BED files
snapatac2.ex.export_bed(adataset,
                       groupby='Subclass',
                       suffix='.bed.gz',
                       out_dir = output_dir
                       )

Best,
Jay

@jayluo2

Here we also uploaded the hdf5 files (see supplementary file: GSE246791_RAW.tar)
https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE246791. You can give this a try.

I did not see apparent bugs in your codes. But previously I found that I cannot write some columns easily into SnapATAC2 AnnData/AnnDataSet. But the groupby parameter supports an independent list: https://github.com/beyondpie/CEMBA_wmb_snATAC/blob/8d090924b7a75ef6bf12f6a6ee396092d9437559/03.peakcalling/src/main/python/prepare_bedfiles.py#L83
This is how I export the bed files.

@wkl1990 Thank you so much! Did you have any errors before for this task?

Songpeng

@beyondpie @jayluo2 The fragments in the subclass have been updated in Catlas. You can download them here: http://catlas.org/wholemousebrain/#!/downloads

@wkl1990 @beyondpie Thank you so much! I will proceed with those files for now and close this issue, but would appreciate any updates regarding the h5ad files as well. Thank you both for all your help!

@beyondpie @wkl1990 I see that there are 338 bed.gz fragments files in the uploaded directory. Could you please clarify why there are 275 cCRE files (which appear to be subclass level) but 338 subclass fragments files? I am specifically referring to this part of the paper: "Finally, 275 out of 338 subclasses were annotated to the 1,482 subtypes. This includes 253 out of 315 neuronal subclasses, covering 28 neuronal classes and 7 neurotransmitter types, as well as 22 out of 23 non-neuronal subclasses, covering 5 non-neuronal classes.”
My interpretation of this is that only 275 of the 338 scRNA-seq subclasses anchored properly to the snATAC-seq data. Is this correct? In that case, I presume the corresponding fragments files for these 275 subclasses are the most relevant files for downstream analyses?
Best,
Jay

@jayluo2
Would you mind to check the number of the files? Kangli just add subclass_id for each of the file, so "338_blabla" only means the index not the number of files.

@beyondpie I see 275 files as expected — thank you!


Make maximum number of blocks scanned in a single call to getEvents configurable

Currently, Juno will happily scan thousands of blocks trying to find an event that matches the given filter. The request is terminated early only if the number of events accumulated reaches a limit provided by the user.
This is a pretty easy to execute DoS attack. With a single RPC request, with a filter that does not match any events, one can make Juno scan the entire header chain.
We should make Juno only scan maximum of N blocks before responding to the request with a continuation token.

If this issue is not being worked on, I'd be willing to take it.

@Exca-DK thanks! assigning it to you

One question for clarity: Should this value be exposed to the user or constant is fine?

Let's make it configurable with a sane default. "Sane default" could be something like "a number that roughly caps the runtime of a single getEvents call around 100ms"


Tooltips show [missing %{name} value] for Order Araneae and Family Ichneumonidae

From https://forum.inaturalist.org/t/missing-name-value/15566
Mouseover Spiders and Ichneumonid Wasps on https://www.inaturalist.org/observations/56936240 and https://www.inaturalist.org/observations/56922216 and you'll see

It's happening in apparently every taxon that's not a species or subspecies. Cassidini, Lepidoptera, Charidotella, Braconidae, Oxalis, Ionoxalis (section), Oxalis (subgenus)_, and so on. Sterna trudeaui and Viola tricolor spp. tricolor don't have any problem, for example.
I use iNat in Brazilian Portuguese.

Just checked in the Crowdin platform the translation for Brazilian Portuguese and there's an untranslated string "%{rank} %{name}" that was added yesterday. The context text is "When a scientific name is displayed with its rank, e.g. Family Noctuidae or
Genus Noctua (but generally not Species Noctua pronuba)".


Enable btn#1

buttons enabled #1  , now the prompt will take the number of clicked button as a input.

Great Job @prasoonjain006 ! I am merging your pull request. You may also want to work on the latest Issue #8 ! All the best!

Thanks @Apoorve73  for giving me this opportunity 😃!
Yes I'm ready to work on another problem!


Hack Session 599 ✼ Remotely

25.10.2023
Wrap-up of our Gōng-fu I/O Weekly · Hack Session #599 ✼ Remotely ✼ meetup.
Projects
• Shoelace and Astro — @olange and @rudifa
• ‹Bouboucle› generative art with Javascript — @andreaskundig and @david-hodgetts
See also

#598 Previous session
#600 Next session

Feedback on session #599 of @olange
Studied following articles:

Preact › Blog 06.09.2022 › Introducing Signals by Marvin Hagemeister and Jason Miller
@lit-labs/preact-signals
@lit-labs/signals-core

As well as (unrelated to Signals):

@lit-labs/router
@lit/ReactiveElement

See also

#587 Previous feedback


一時ファイルのディレクトリを環境変数から取得

Codeforces だと K:\codeforces73\e0f971a623bf424b18406b1457686d11\run-085ce89b7376d5d89465fcffeb6fbfec\run\tmp6t7cy_k1 のようなディレクトリが使われていて、codeforces71 codeforces72 codeforces73 がその時々で変わるため、ハードコードだと大変。

https://codeforces.com/blog/entry/4088

You are forbidden to modify files or directories' permissions in the file system.
あー

Can-do's and Can't-do's

It is forbidden to obfuscate the solution code as well as create obstacles for its reading and understanding. That is, it is forbidden to use any special techniques aimed at making the code difficult to read and understand the principle of its work.

ここも怪しいのかも、“aimed at making the code difficult ...” ではないけど


List index out of range - generate_tfrecords.py

Hello,
For TF 2.x, I tried generating the tfrecords from csv and images directly. However, it failed saying
  File "generate_tfrecord.py", line 147, in <module>     class_dict = class_dict_from_pbtxt(args.pbtxt_input)   File "generate_tfrecord.py", line 114, in class_dict_from_pbtxt class_dict[names[i]] = ids[i] IndexError: list index out of range

Hello,
Thanks for the article. I got the same problem, if create a tfrecords from more than one id in label_map.pbtxt file.
I checked the object list in csv file and it was exactly the same as in pbtxt. Please correct it and thank you in advance.
my label_map.pbtxt
item {
id: 1
name: 'A'
}
item {
id: 2
name: 'B'
}
item {
id: 3
name: 'C'
}

Please test the new master branch, as I believe the problem has been fixed. Unfortunately, I don't have an image dataset to test the fix, but I fixed the specific bug that caused the reported issue.

@khany27 @mineymax can you confirm that the problem was resolved?

I tested yesterday, the problem was solved very well, thank you very much.
However, I would like to keep it updated some line of code to be compatible with tensorflow 2.x (i.e. tf.gfile.GFile update to tf.io.gfile.GFile and tf.python_io.TFRecordWriter to tf.io.TFRecordWriter)

There is already a pull request with that change, which I will merge as soon as possible.


[SMALLFIX] Enable WorkerMetadataSyncIntegrationTest

What changes are proposed in this pull request?
Remove the @Ignore annotations. It works now.
Why are the changes needed?
improve code quality.
Does this PR introduce any user facing changes?
no.

alluxio-bot, merge this please


ban and warn anonym

make ban and warn anonym cause some users get a few insults

I don't like that I think you should know who has banned/warned you. If that's really needed a setting to hide the mod would be better.

I don't like that I think you should know who has banned/warned you. If that's really needed a setting to hide the mod would be better.

This also becomes a slight legal issue @dys0n-V-nT if someone were to send something bad we would be in trouble so the mod being there shows we or the bot isnt sending it the mod is

This also becomes a slight legal issue @dys0n-V-nT if someone were to send something bad we would be in trouble so the mod being there shows we or the bot isnt sending it the mod is


Transaction info: hint about the rate should be shown below "Amounts" field only for multiple CA transactions details

On the transaction info screen comment about exchange rate should be under the amount. It should be shown below "Amounts" (plural) field only for multiple CA transactions details

Change color of the word "Amount" to gray

Checked


Error when using a default gateway outside of the allocated range

When setting up a default gateway, libnetwork will verify that it belongs to the bridge network, but will also try to allocate it from the allocated range which, I think, is the --fixed-cidr flag from docker.
However, the default gateway may very will be outside of the allocated range, such as:
% docker -d --bip=<IP_ADDRESS>/16 --fixed-cidr=<IP_ADDRESS>/24 --default-gateway=<IP_ADDRESS>

Currently, this fails, although it should work.
I think this code, https://github.com/docker/libnetwork/blob/master/drivers/bridge/setup_ipv4.go#L112, should only be called if the default gateway belongs to the allocated range.

Thanks for reporting this.
I think I know where the issue is. Will propose a patch soon.

Thank you for working on this. I'm sorry, but I can't comment on your solution as I really don't know how libnetwork works. I've tested a much cruder patch that removes the call to ipAllocator.RequestIP, it works but it's obviously not appropriate :)
Do you think there's any chance this could make it to docker 1.7.0? I've been waiting for the possibility to set --default-gateway for a long time, but can't use without a fix to this bug.
No pressure though, I get it if you consider this only affects one use case for --default-gateway and is not a blocker.

+1. This bug prevents --default-gateway from working in many important cases.


Run a main() application with an error in the code

In my code I have some main() function to generate some static data. If I run them with a class that doesn't compile (for example, a missing ;) I get this error:

Error:Android Gradle Build Target: java.util.ServiceConfigurationError: com.android.ide.common.blame.parser.PatternAwareOutputParser: Provider com.squareup.sqldelight.intellij.parsing.OutputParser could not be instantiated

Instead of:

Error:(199, 31) error: ';' expected

If I try to run my android app or my tests the message error is the correct one.

weird, that looks like an issue with the jps-plugin. I'll investigate, thanks

I have the same error.

Error:Android Gradle Build Target: java.util.ServiceConfigurationError: com.android.ide.common.blame.parser.PatternAwareOutputParser: Provider com.squareup.sqldelight.intellij.parsing.OutputParser could not be instantiated

Error:Android Gradle Build Target: java.util.ServiceConfigurationError: com.android.ide.common.blame.parser.PatternAwareOutputParser: Provider com.squareup.sqldelight.intellij.parsing.OutputParser could not be instantiated
Even I have the same error

Why has not yet been repaired？

i believe this has been resolved recently and was a bug in android studio. will reopen if it pops up again


Make hasCorrectArity handle tuples properly

This completes the work that started in PR #33069, and fixes #32835.
(Includes that PR in a separate commit.)
There are probably two additional related changes that are needed to
make this more complete:

Fix the code that composes the error message (see the first two
FIXMEs in callWithSpread3.ts).

Fix the code that checks the argument types (second two FIXMEs).

There is also an error in genericRestParameters1.ts which changed
but should not be an error in the first place.  Added a FIXME there
too.  (Probably will work if the previous iterm is done.)

In addition, getEffectiveCallArguments munges the arguments in case of
a spread in the last argument which might be better to avoid.  (I think
that there are cases where it wouldn't work anyway, such as a spread of
an array followed by a spread of an empty array.)

@typescript-bot test this
@typescript-bot user test this

(After this is merged, there should be two new issues for the FIXMEs in the test, one for the bad error message on a bad arity, and one for checking the argument types.)

I just looked at the user test failures and there's nothing new there.

Nothing in the RWC failures either.


Unable to launch Kubernetes dashboard from azure cli

I m trying to launch the Kubernetes dashboard from the azure cli using the below command:
az aks browse --resource-group groupName --name clustername
I get the below error:
The command failed with an unexpected error. Here is the traceback:
[WinError 193] %1 is not a valid Win32 application
Traceback (most recent call last):
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\knack\knack\cl
i.py", line 206, in invoke
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core\commands_init.py", line 351, in execute
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core\commands_init_.py", line 409, in run_jobs_serially
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core\commands_init.py", line 402, in run_job
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\six\six.py", l
ine 693, in reraise
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core\commands_init.py", line 379, in run_job
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core\commands_init.py", line 171, in call
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-core
\azure\cli\core_init_.py", line 451, in default_command_handler
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-qvpg2q31\azure-cli-acs
azure\cli\command_modules\acs\custom.py", line 1381, in aks_browse
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\subprocess.py", lin
e 336, in check_output
**kwargs).stdout
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\subprocess.py", lin
e 403, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\subprocess.py", lin
e 709, in init
restore_signals, start_new_session)
File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\subprocess.py", lin
e 997, in execute_child
startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application
I have installed Python 3.7 64 bit version in my system. Not sure what is the problem. Can you please help  me ?
Thanks,
Swathi S-

Looks like a python 32-bit vs. 64-bit issue. Do other az commands work? Closing for now but please reopen if this is specifically impacting the aks browse command.