Add startup flag --light-weight-webhook to trim down outbound webhook payload

When we create a credential offer with image, the image is repeated in following objects (dict)
"credential_request", "cred_request", "credential_proposal", "cred_proposal", "credential_offer", "cred_offer", "credential_preview", "cred_preview", "values", "credentials~attach", "offers~attach"
a Credential offer with 30KB image resulted in a webhook sized at 227KB in my use case and resulted HTTP 413 as well as unnecessary stress on WAF rule regex processes on those encrypted string and base64 image string.
This startup flag will remove those objects only on outbound webhook and size went down to 7KB at most.
I did preserve 1 object "cred_issue" which has the "rev_reg_id" and "cred_rev_id".
To me this is a more like a nice to have flag if someone ever encounter problem with image and webhook.
Previous related discussion
PR #725
Issue #1222
Signed-off-by: Victor Lee<EMAIL_ADDRESS>

Interesting problem. Looking forward to folks reviewing this and the best approach to solving the very real issue.  I don't know nearly enough to know what is the right way to handle this, so adding @ianco @dbluhm @frostyfrog @andrewwhitehead to the discussion.
It would be a huge change, but perhaps the webhooks should only send new information to the controller (vs. the full state) and leave it to the controller to hold the data it needs to make decisions as the protocol proceeds.  Or do that, and add in an endpoint that says "send all state for protocol X" whenever it is needed by a controller?
Thanks for the PR!

I'll agree that this is an interesting problem and I'm certain it will only get worse when implementors don't take file size into consideration, especially if upper management requests higher resolution imagery. For example, an image from an iPhone that controls embedded 3D data for biometric systems.
This issue, in terms of credentials, is prevalent during not only credential issuance, but verification/proof and endorsement. There are also plugins implementing features (such as the Q and A protocol) which may also cause increased webhook payloads if the system chooses to accept binary or non-fixed width payloads (I'm sure some malicious user is just waiting to send a script from Shakespeare as an answer to "why are you visiting the country").
I believe that what you've got here is a great first step towards solving the issue, but I believe that there's a lot more that can be done overall. Making sure to decrease unnecessarily duplicated data is one thing. Limiting the size of the payload in the face of untrusted data is another. I recommend that, for lightweight payloads, we send the minimal amount of data needed to request the full data where possible. Such as a presentation exchange ID during proof and allowing the system to retrieve the full record manually if they need it. This may be considered a bit extreme, but since we can't quite trust that the third party agent won't send us something massive, I feel as though it's the best compromise to prevent errors.
P.S. I hope that's all readable. Woke up and sent it from my phone. Really interesting problem we've got here :)

@ianco could you take a look at this one?  Seems valuable to me.

@ianco could you take a look at this one? Seems valuable to me.

ok will do.

Would it make sense to make this a breaking change with a “—useOldStyleCredExchangeProtocolStateObject” (or something equally weird) for backwards compatibility?  It seems this is something we would rather have than not.
We are doing a 0.8.0 breaking release change Real Soon Now, so this would fit there.

I think the default is the old style verbose webhooks, no?  so not a breaking change ...  (unless you want the default to be the more terse format)

PS  failing test:  FAILED aries_cloudagent/messaging/models/light_webhook.py::FLAKE8

I’m suggesting that we want the default to be the terse form, and only if needed use the long form. AFAIK, all of the data is available in the long form — it’s just the extra copies of the data that are in the short form.
Put another way, we could be smarter about not repeating content in the Protocol State Object by blindly adding in each message, where the message repeats data from a previous message.
Or is that a bad idea? I guess things could be lost if the content changes across messages — then we would only have the latest version vs. all versions.  How is that handled in this case?  Is there checking that the content hasn’t changed between messages, and if so what is done?

I’m suggesting that we want the default to be the terse form, and only if needed use the long form.

I think this would be fine, however if we do this we should probably implement a terse form for the presentation exchange as well (this PR only addresses the cred exchange).  (I think all the other exchanges are prob ok as is)

So maybe go with this now, and add an Issue to change the presentation model as well, and then change to terse by default  in the next breaking release?  In theory, we could get it in for the final 0.8.0.

sorry my bad, still getting use to black and flake8...ha...

@ianco @swcurran Thank you very much!
I think next step is to create new issue on proof exchange webhook and terse form by default so that we can carry the discussion further.

Thanks @victorlee0505 — agree on that as the next step.


Initial changes

updates to readme
updates to job
replace container (always cortex) with name
update label name

With these changes, all but 2 of the Compactor and Compactor Resources dashboards have data.

Thanks Tyler!  Once we get them all working I wouldn't expect to have to rebuild them again unless we redeploy Cortex with a new name...


Rename classes in interface folder so they better reflect their functionality

StatusBars and MenuBars don't need to be Element

Tooltip neither


jaxb2-basics plugins should not be included as dependency by default

Currently org.jvnet.jaxb2_commons:jaxb2-basics is added as (runtime) dependency by default, which adds some xjc plugins. Should we really have this added by default?
The site docs says this should be added as a dependency to the plugin when needed. I therefore suggest we remove. This could break some existing builds though that rely on this (undocumented) behavior. It would easily be solved though by following the documentation.

Fixed by 2cc5e57


Added bf16 lowerings for cuda backend

should we also enable it for check_cuda_streams suite?

Forgot there were two cuda targets. Enabled.


DDLogLevelAll versus DDLogLevelVerbose

Whats the difference?

As you can see from the code DDLogLevelAll encompasses all levels, both current and future.
In practice and right now they are the same but that could change at some point or with custom levels.


Alternative Endpoints (Or Effect Network Proxy)

Greymass endpoint seems spotty on the Asian continent, it would be nice to be able to select alternative endpoints if greymass is not working. Perhaps eos.newdex.one
Two DAO members have had issues with this, however, a VPN did fix this. An alternative solution may having the webserver be available as a proxy for making requests so the strange region specific issues are no longer a problem.

ProtonVPN in particular was the one used
https://protonvpn.com/

Let's investigate what are the current most stable nodes used in the ecosystem. Maybe there is even a proxy service available or one we can host


[SITE BROKEN] NovelUpdates.com

pasting the NU link in the telegram bot gives nothing.
But NU is updated a one of the sources.

Novel Updates does not work right now, see #135

Alright I think the best easiest semi-automated solution (as well as backup solution for future NU shenanigans):

user has to go to "novel up dates com/series/xxxx-xxxxx/" and click on the "show all chapters“ button
user has to save "novel up dates com/series/xxxx-xxxxx/„ and save the site as mhtml file/save full site
than open the file in Novel Grabber
everything works as previously with novelupdates url

On the developer side, you’d just make the scraper look for the links & chapter numbers in the  „my_popupreading_wrapper“ in the mhtml/HTML file.
Or if they still had the "my_popupreading_wrapper" popup opened when they save it, it should work without changing a thing how the Novel Grabber’s NU scraper works now.

I tested it and https://www.novel up dates.com/extnu links aren’t behind hCaptcha. Well…yet.

Maybe it could be even automated with browser extension, maybe not the clicking on "show all novels button“ (idk really how sensitive hCaptcha is to that), but definitely the - generate html file, save it, open it NovelGrabber, parts could be automated.
I don’t see this backup way (with extension) as massive downside since most users start with browsing to novelupdates to find the Novel link first anyway.
It’s not elegant or automated, but it should at least give a backup solution for user to just download a novel, when that niche novel isn’t anywhere else but NU & NU adds another layer of protection breaking the scraper.

I do get a 403 error on the /extnu links as well, so this won't help. Also, while not fully automated, you could do this with manual grabbing already: Paste a table of contents url, fetch links (doesn't matter if you get a 403, it's needed to work) then manually insert chapter links. There are probably browser addons to copy links from a selection.

I don’t get 403s when I use headless browser, without it I also get them.
What you say about grabbing & pasting the content from the NU pop-up window - that suprisingly works well.
If you meant from pasting ToC & chapter links from the website that hosts the novel - there you run into problem if you have 2+ translation groups working on a novel. Or that their ToC is properly updated for newest chapter links. Or that their site isn’t all badly sitemaped/formatted.
But regardless of which one you meant - both require quite a lot of manual work from the users and presumes that they have a tool & knowleadge so they can extract URLs from text - apart from maybe also reversing lines (NU).
That’s pretty learning heavy, slow & unautomated for normal users.

@Flameish
While we’re on that topic, it would be great if you could post multiple ToCs from multiple sites that have the novel to get the links.
Now whenever you add new ToC link, the currently fetched links disappear.
It would be great if they only disappeared when you click on the trash icon.

Also useful if you spent 10mins editing the chapter links to not have them gone by loading the url again.

And a feature where you can just paste links to the links window if they are text links.
Meaning „Chapter 30“ text that also is a URL link to Chapter 30.
For example:

Headless does indeed work with the /extnu links, my bad.
For manually grabbing to work you need to click on "fetch links" once, doesn't matter if you get a 403/forbidden error, it's an initialization thing. After that you can manually copy and paste the links from NU to it.
I agree that it can be quite a hassle to accomplish that and I like your idea of parsing html to chapter + links directly, even if this doesn't quite work in browsers since they just copy  text as far as I know. You would need something like [this addon](https://addons.mozilla.org/en-GB/firefox/addon/copy-selected-links/ to copy links), it is also what I use for NU in conjunction with this bookmark script:
javascript:(function(){var x = document.querySelectorAll('li a[title]');var i;for(i = 0; i < x.length; i++){x[i].remove();}})();

When executed, this script removes those "Going to chapter" links from the popup window so that you can copy all the links once for the chapter name and once more with the addon for the links.
Nevertheless please keep this issue on topic and create new ones for your feature requests; but be advised that I'm currently in the process of rewriting Novel-Grabber, which is why I'm not adding new features to this version. Separate issues are easier to keep track of than having these requests buried within others.

Yep, I’ll create seperate issues. I was just pitching you the ideas if they make sense or not while we were on topic.


Modified the ios_string.rb script to create a swift 3 compliant syntax

Thanks, I have update the tools to swift3.


istio resource informer

Is your feature request related to a problem? Please describe.
I want to do a API level operation on the resources of istio
K8s provides a informer mechanism for the operation of a resource object, but when I analyze istio, I find that istio's client does not provide the watch method, and the restclient in client is private, so I think it is more troublesome to construct watch by myself. I would like to ask what methods should be used if we want to achieve monitoring of resources.
example:
`import (
"istio.io/istio/pilot/pkg/config/kube/crd"
"yunzhicloud.com/istio/utils/k8s"
)
type Gateway interface {
List(ns string) (list *crd.GatewayList, err error)
}
type gateway struct {
client *crd.Client
}
func NewGateway() (g Gateway, err error) {
client, err := k8s.NewIstioClient()
if nil != err {
return nil, err
}
return &gateway{client: client}, nil
}
func (c *gateway) List(ns string) (*crd.GatewayList, error) {
return &crd.GatewayList{}, nil
}`
c.client not found watch method?
and  I look at the source code found (Istio-0.8)：
https://github.com/istio/istio/blob/master/pilot/pkg/config/kube/crd/controller.go
line 108--120 There is such a code snippet
rc.dynamic.Get(). 				Prefix("watch"). 				Namespace(namespace). 				Resource(ResourceName(schema.Plural)). 				VersionedParams(&opts, meta_v1.ParameterCodec). 				Watch()
but the restclient is  not expose
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context about the feature request here.

Istio does not store the resources. You can use the standard informers on k8s CRDs related to Istio

I think https://github.com/istio/istio/issues/8772 will cover this. Closing this as a dupe.


pkg/prometheus: Update Thanos to 0.1.0

This supersedes #1934, as I can't push to that branch.
The only change is reverting to using tsdb.path for the sidecar.
I've tested this branch running on my cluster and so far it seems good.

lgtm on green


HELM-331: PerfDS React, various query fixes

HELM-331: Performance Datasource Fixes

fix variable interpolation for node and resource in Attribute queries
Query Editor, get initial state from existing state (passed in props)
encapsulate and provide strong typing for API requests
check whether queries are hidden, filter them out before sending via API
more comprehensive check for valid Filter queries, don't send to API if invalid
ensure query targets added or concatenated as arrays to existing query
various other fixes

External References

JIRA (Issue Tracker): http://issues.opennms.org/browse/${JIRA-ISSUE-NUMBER}
Continuous Integration: CircleCI

I think Resources and Atribut should allow * and | and generate multiple queries. We can address that after.

I think that this works, need to double check. I think it's handled on the server side, but we do pass * and |.


flesh out API documentation

Everything in rest_api.md is filled out now. There's still some more to do --- I'd like to get rid of the function names, and apidesc.md is still there and pretty much unchanged (i.e. very bitrotted), but I think this is a good self-contained change.
This builds on top of #455, so that should be merged first.

Suggestion: We can point out that any API call will fail with 404 if you reference a non-existent resource in a non-creation API call.

+1

Looks good. Thanks for the documentation Ian =) +1


[stable/instana-agent] Chart version 1.0.24

What this PR does / why we need it:
Update README to include chart installation and uninstallation instructions for both Helm 2 and Helm 3
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

cc @wiggzz
cc @nfisher

cc @wiggzz

cc @nfisher

/lgtm


Job template as configmap

To run a training job, controller relies on a helm chart under
templates/job. Some variables (e.g., endpoints of apiserver and
notifier) of files in the folder are hard coded. When those endpoints
need to be updated, a new image needs to be built, which is time
consuming. Also, a user needs to modify several files, which can cause
misconfigurations.
To address the problem, now the templates are moved into configmaps so
that it can be updated easily at deployment time.

Codecov Report

Merging #156 (1b22662) into main (f8b3c45) will not change coverage.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##             main     #156   +/-   ##
=======================================
  Coverage   21.29%   21.29%
=======================================
  Files          34       34
  Lines        1503     1503
=======================================
  Hits          320      320
  Misses       1173     1173
  Partials       10       10

Impacted Files
Coverage Δ

cmd/controller/app/job/default_handler.go
0.00% <0.00%> (ø)

cmd/controller/app/job/handler.go
0.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8b3c45...1b22662. Read the comment docs.


Automatic Consul Registration for Quarkus Applications

Enhance the integration with Consul to enable applications to automatically register themselves during startup, even if the application doesn't explicitly know its IP address. Consul will handle the communication, minimizing configuration effort.
When a Quarkus application starts, if the stork-service-registration-consul is present, it should automatically register with Consul, including its IP address, public port, and other metadata. To provide flexibility, we will introduce the following new configuration options:

Public IP / Name: Optional. If not provided, Consul will use its detected IP.
Public Port: Optional. Defaults to quarkus.http.port.
Service Name: Optional. Defaults to quarkus.application.name.
Service ID: Optional. Consul-specific. Defaults to a generated UUID (quarkus.uuid or similar).
Health Check URL: Optional. If a health check is configured, use the liveness endpoint. Note that it may run on a separate management interface, with a different public IP and port, so manual configuration may be needed.

For full Docker mode, this will work assuming the private port is exposed on the network.
Consul Failure Management
That would be a request for enhancement for the Vert.x consul client.
Another critical aspect is how the client manages Consul failures. Currently, even in distributed mode, the Consul client is unaware of all Consul instances and doesn't switch to a new leader when a failure occurs.
Proposed solution:

On client creation, query Consul for its nodes and store them.
In case of a failure:

Query any available Consul server ($ANY_CONSUL_SERVER/v1/status/leader) to find the new leader.
Create a new client and retry the request with the new leader.

To optimize performance, clients should be reused instead of being created every time.
Additional Considerations

Handling scenarios where the health check is on a different IP/port than the application itself.

cc @cescoffier @melloware @FranckD-Zenika @@FranckDemeyer

I've been working a bit on this today again.
There is a problem when consul returns an empty address. The problem is in StorkClientRequestFilter line 50 :
      String host = instance.getHost() == null ? "localhost" : instance.getHost();

I'm creating a PR soon for fixing it.
Anyway, we still have a problem about auto-registration. Consul is not able to detect the public IP of the application sending the request. It just uses the agent's address If not provided, and for some reason when the agent's address is registered it returns an empty address when a discovery request is sent.
So, we can do automatic registration in quarkus when de dependency is present but need the public ip and public port to register via the configuration. For the service name we can default to the application name if not provided.

@aureamunoz any progress on this?

Not yet. I'm planning to work on it once a few things are done.


Mint - Run subsequent system tests even if a test fails

Is your feature request related to a problem? Please describe.
Whenever the Mint test suite is executed, a single failing test in minio-java or minio-go causes all the remaining tests from the same "test package" to be skipped. The disadvantage is, that you don't see all failing tests but only the first failing test (at least in minio-java and minio-go tests). Ok the advantage is, that you don't need to run the whole test suite to see if "something" is broken.
Describe the solution you'd like
We suggest to introduce an optional runtime variable RUN_ON_FAIL.
By setting the variable RUN_ON_FAIL=1 all tests from all packages should be executed even if a preceeding tests is failing.
Describe alternatives you've considered
Changing the default behavior to ALWAYS run every test independent of failing tests.
This would change the default behavior of the Mint test suite.
Additional context
We implemented this feature for minio-go and minio-java and we will create appropriate PRs and link them to this issue, so that there is a single place for discussions for this.

There are still some open PRs related to this issue.
-> https://github.com/minio/minio-go/pull/1475
-> https://github.com/minio/minio-java/pull/1180
And a change in minio-java that needs to be reverted: https://github.com/minio/mint/commit/721525fa3894306c117e36f7dcea6e1af369f133

The two PRs are now up to date:
https://github.com/minio/minio-go/pull/1475
https://github.com/minio/minio-java/pull/1180
The change in the minio-java call (721525f) does not need to be reverted.
I passed the variable RUN_ON_FAIL as environment variable and additionally (that was the error) as parameter to minio-java.


Add support for other entities as part of a collection - #262

Change log:

Update Collection schema to reflect support for multiple entity types

@libpitt It looks like before this refactor we had two identical functions named link_collection_to_datasets, you updated one but I think we can delete the other inside schema_triggers.py.
There is also a function named link_collection_to_datasets that should be updated to link_collection_to_entities with an updated parameter: dataset_uuid_list -> entity_uuid_list


Clean up release branches

Hello 👋,
Thanks for this great tooling. I've got a minor feature request which would help save a few clicks on the consumers end - would we be able to automatically delete release branches after they have been merged or marked inactive?

The core release-please library can't do this because it doesn't listen to events. It's possible to add something on the GitHub app however, with v13, we now re-use branch names. This re-use should alleviate the need to delete a bunch of branches.


Mysql query lower(name) does not use index

Following query is issued every time we tag something and looks like AATO calls SQL lower function, which in turn makes Mysql not use index on "name" column.
mysql> EXPLAIN SELECT tags.* FROM tags  WHERE (lower(name) = 'web-development' OR lower(name) = 'entrepreneur' OR lower(name) = 'opensource');
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows   | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
|  1 | SIMPLE      | tags  | ALL  | NULL          | NULL | NULL    | NULL | 213164 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+--------+-------------+
1 row in set (0.00 sec)
In mysql name has unicode_ci collation so all queries against name column are not case sensitive by default. YMMV with other dbs though. I'm on Mysql 5.5.

Any news on this issue? Since #354 is now closed, is there any plan to solve this?

PostgreSQL also did not use index for me
Solved by adding a lowercase index
CREATE UNIQUE INDEX lower_case_tag_name ON tags ((lower(name)));

PostgreSQL used 100% cpu before this index when creating objects with tags. This index is a massive improvement.


perf tests

@cereallarceny I made what I think are minimal repros of #108. It looks like having an object inside of an array causes a lot of issues immediately, likely because of what needs to be done for _key and _type to be correct. Just by the nature of typescript perf, it'll be pretty difficult to identify if this is the exact cause until we fix it and determine if it also solves your issue.

Your perf test related to #108 looks to be spot-on what I'm encountering. Array of objects, nested, ultimately with an image at the leaf node.


Change copyright year to 2021

HI, @sophianyberg
It is better to create a single PR for the same change. Can you move all your changes in a single PR? :slightly_smiling_face:

HI, @sophianyberg
It is better to create a single PR for the same change. Can you move all your changes in a single PR? :slightly_smiling_face:

I cannot.

I cannot.

You can merge your branches locally into a single branch and then close other PRs. And keep a single PR open for the branch in which you have all the changes. Hope it helps.

You can merge your branches locally into a single branch and then close other PRs. And keep a single PR open for the branch in which you have all the changes. Hope it helps.

Hello @rohit2sharma95 ,
I am on an iPad and I can not merge this pull request until you approve it. Can you please approve this and the other pull requested?
Thanks,
@sophianyberg

Hello @rohit2sharma95 ,
I am on an iPad and I can not merge this pull request until you approve it. Can you please approve this and the other pull requested?
Thanks,
@sophianyberg


ComposableStablePoolAMPModifiedFactory

For Goerli

@baloonsky as I mentioned to you on Discord we first merge changes to staging Subgraphs before promoting to production. Could you please rebase your branch with the new base?

done, thank you


Run fails on systems with snap gnome-system-monitor

Describe the Issue
When the task for Set home perms runs, it finds a link to a folder on a read-only filesystem and fails with OSError: [Errno 30] Read-only file system
Expected Behavior
Links to different filesystems outside of the home dir fs should not be followed or modified.
Actual Behavior
TASK [/home/user/UBUNTU18-STIG : MEDIUM | UBTU-18-010451 | PATCH | All local interactive user home directories must have mode 0750 or less permissive. | Set home perms]
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 30] Read-only file system: '/snap/gnome-system-monitor/174/data-dir/themes'

Control(s) Affected
010451
Environment (please complete the following information):

branch being used: devel (==main)
Ansible Version: 2.9.27
Host Python Version: 2.7.17 (old, I know)
Ansible Server Python Version: 2.7.17
Additional Details:
Ubuntu Desktop 18.04.05 LTS
snap gnome-system-monitor 42 installed

snap list gnome-system-monitor
Name                  Version  Rev  Tracking         Publisher   Notes
gnome-system-monitor  42.0     181  latest/stable/…  canonical✓  -

Additional Notes
The find command has -not -type l which correctly finds /home/[user]/snap/gnome-system-monitor/174/.
But that folder contains a link to a read-only filesystem
ls -al /home/[user]/snap/gnome-system-monitor/174/
total 20
drwxr-xr-x 4 user user 4096 Feb  1  2022 .
drwxr-x--- 5 user user 4096 Oct 21 10:10 ..
drwx------ 7 user user 4096 Feb  1  2022 .config
-rw-rw-r-- 1 user user   31 Feb  1  2022 .last_revision
drwxr-x--- 3 user user 4096 Feb  1  2022 .local
lrwxrwxrwx 1 user user   46 Feb  1  2022 .themes -> /snap/gnome-system-monitor/174/data-dir/themes

And then it tries to change permissions on /snap/gnome-system-monitor/174/data-dir/themes which will never succeed because all /snap/ subdirs are mounted ro.
mount | grep /snap/gnome-system-monitor/174
/var/lib/snapd/snaps/gnome-system-monitor_174.snap on /snap/gnome-system-monitor/174 type squashfs (ro,nodev,relatime,x-gdu.hide)

Possible Solution
More careful conditional construction of ubtu_18_010451_audit.results??

Sorry for the delay in this, taking a look at it today and will address it.

Upon trying to replicate your you error I could not.  I am testing on a ubuntu18 box with the snap gnome-system-monitor installed and mine is not producing the error you have gotten.  Will continue to look into it today when i have some free time.

hi @edlins
Thank you again for raising this issue. It has taken longer than expected to get to this (client priorities), we have added a new fix which should resolve the problem you are seeing.
I am not sure if this is still an issue for you?
I will close this issue as it has been open for some time, please feel free to reopen if this is still outstanding.
many thanks
uk-bolly


build! paths oddities

build! chokes on paths where show! works.
user=> ((requiring-resolve 'nextjournal.clerk/serve!) {:browse? true})
Clerk webserver started on http://localhost:7777 ...
{:browse? true}
user=> ((requiring-resolve 'nextjournal.clerk/show!) "/home/kevin/src/ir-notebook.clj")
Clerk evaluated '/home/kevin/src/ir-notebook.clj' in 4619.99975ms.
nil
user=> ((requiring-resolve 'nextjournal.clerk/build!) {:paths ["/home/kevin/src/ir-notebook.clj"]})
Clerk evaluated 'jar:file:/home/kevin/.m2/repository/io/github/nextjournal/clerk/0.13.842/clerk-0.13.842.jar!/nextjournal/clerk/builder_ui.clj' in 989.224615ms.
👷🏼 Clerk is building 0 notebooks…
🧐 Parsing… Done in . ✅
🔬 Analyzing… Execution error (ExceptionInfo) at nextjournal.clerk.builder/throw-when-empty (builder.clj:120).
nothing to build
user=>

This appears to be down the to fact that show! doesn't use fs/glob and build! does.
build! using fs/glob causes a two issues:

fs/glob doesn't descend into symlinks by default (my ~/src is a symlink)
fs/glob doesn't like absolute paths

The path handling seems like it should be uniform between show! and build!, and at the very least it would be nice to support absolute paths. I think symlinks should be supported too, but there may be security implications, not sure.

fs/glob also doesn't like relative paths that contain ".."
user=> (fs/glob "." "../../Sync/src/ir-notebook.clj" {:follow-links true})
[]
user=> (.exists (java.io.File. "../../Sync/src/ir-notebook.clj"))
true
user=>


EmbeddedKafkaBroker consume with seekToEnd does not reliably seek to the end

Backport of gh-3660

Fixed via 71b571c03d29f39544494d15791a36d49a2256b7


The incoming request has too many parameters. The server supports a maximum of 2100 parameters. Reduce the number of parameters and resend the request.

I am getting this error when downloading packages in VS Code. I have generated my own package. We have little over 7000 objects.

I've generated my symbols manually using the Microsoft.Dynamics.Nav.SymbolReferenceGenerator.exe
This works fine. At least, it looks like that for now.

That error is not issues when downloading but when you have generated the symbols.
How did you create your symbols? By starting finsql.exe with generatesymbolreference flag and in the UI you have compiled all objects? Then this issue will happen. That is a bug fixed for CU1.
But this step should not be needed. The command generatesymbolreference does it all for you.
Also SymbolReferenceGenerator eventhough it will generate symbols it cannot generate as good symbols as C/SIDE. You may expect scenarios where you will not be offered as good services(e.g intellisense, lookup for symbols etc) as symbols generated from C/SIDE. An example would be if anything in our compiler services that depends on the type of the sourecexpression of a control field

All objects compile. I've started the finsql.exe with the generatesymbolreference.
After that there is nothing in the [NAV App Resource] table (Except the system symbols)
This error happens when I run the download symbols from VSCode. I see this error in the windows event log.
If the error is fixed in CU1 then it is ok for me. I can continue now. I just succesfully published my first extension in my hybrid database. I am already spending way more hours getting this to work than I want.
Hope you understand.

My suggestion is still just let the finsql.exe command=generatesymbolreference .... (The command line version, not the UI started with the flag called the same.)
That should create your symbols. I know that we have some issues right now that we have only published the System symbols and the Application symbols only with Cronus.
We will also publish soon all these required symbols to a downloadable location that anyone can publish them for themselves in their own database. Then hopefully things will be easier.

I'm good for now. I'll test again on CU1


Initial Angular schematics cleanup

This PR starts updating older schematics to use newer utilities and helper functions with the goal to reduce complexity as well as remove older helper code.

Nice cleanup!


Usage of Identify + Track in this API vs Analytics.NET

Previously we only had to call Track and supply the userId field as a parameter in order to populate the UserId field in the payload which is published to Segment for this Track event.
Are we expected to call Identify multiple times for a single user when switching between them in order to fill the UserId property in a subsequent call to track?
For example

Call to identify(user1)
Call to track an event from (user1)

Next usage of analytics client.

Call to identify(user1)
Call to track an event from (user1)

Would result in multiple identify calls being pushed to segment before each Track call, it seems unnecessary? Maybe it's your intention, just looking to understand best practice examples for this case.

hey @jn011, definitely don't do that. you can attach user id at a per event basis with the following plugin:
class UpdateUserIdPlugin : EventPlugin
{
    public override PluginType Type => PluginType.Enrichment;

    public override RawEvent Execute(RawEvent incomingEvent)
    {
        if (incomingEvent is not TrackEvent modified)
        {
            return incomingEvent;
        }

        modified.UserId = modified.Properties.GetString("userId");
        modified.Properties.Remove("userId");

        return modified;
    }
}

and then track as following
analytics.Add(new UpdateUserIdPlugin());
analytics.Track("MyEvent", new JsonObject
{
    ["userId"] = "YOU USER ID"
});

you can modify almost any value of the event with this approach. here is an example of modifying context. you can do the same for anonymousId, integrations, etc

@jn011 that's a good question. the main reasons behind this decision are:

support client-side (mobile/web/game) use cases.
unify APIs across all SDKs.

yes, it's a bit counter intuitive and feels more cumbersome, but this is a trade-off between flexibility and usability. this trade-off is beneficial to make because we can't capture all the use case for customers. some may want to modify userId/anonymousId/context/integration, some may not. if we have to support all the use cases, the logic of the sdk becomes bloated and performance of the sdk slows down because of unnecessary checks. thus, it's a good idea to delegate this to the customer, so that the sdk is simply and works with only the necessary logic. in addition, with this flexibility, you can always write plugins to fit your needs without having to raise a feature request and wait for a release from us, reducing the feedback loop.
if you check this method, you'll find the side-effects that identify call could bring can be eliminated by overwrite the value of userId and anonymousId. you can write the plugin to do so to make the API calls idempotent


Add Condition When Iterating Over Variable Parts

This makes it explicit to only add to the variable length list if the length of the part is non-zero

Ops, I accidentally clicked approved. Please ignore that

value = [b"foo", b"bar", b"test", b"\x42\x42", b"eth2", b"very", b"wow"]

def serialize(v): return v if isinstance(v, bytes) else v.to_bytes(4, 'little')

def is_variable_size(v): return b"t" in v

BYTES_PER_LENGTH_OFFSET = 4
BITS_PER_BYTE = 8

# Recursively serialize
fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in value]
variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in value]

# Compute and check lengths
fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts]

variable_lengths = [len(part) for part in variable_parts if len(part) != 0]
print('pr: ', variable_lengths)
variable_lengths = [len(part) for part in variable_parts]
print('now: ', variable_lengths)

assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE)
# Interleave offsets of variable-size parts with fixed-size parts
variable_offsets = [serialize(sum(fixed_lengths + variable_lengths[:i])) for i in range(len(value))]
fixed_parts = [part if part != None else variable_offsets[i] for i, part in enumerate(fixed_parts)]

print("fixed parts", fixed_parts)
print("var parts  ", variable_parts)

# Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts
x = b"".join(fixed_parts + variable_parts)

print(x.hex())
print(str(x))
print(len(x))

# slice out of bounds: valid in python, obfuscates bug in PR.
print(variable_lengths[:1000000])

Output:
pr:  [4, 4]
now:  [0, 0, 4, 0, 4, 0, 0]
fixed parts [b'foo', b'bar', b'\x17\x00\x00\x00', b'BB', b'\x1b\x00\x00\x00', b'very', b'wow']
var parts   [b'', b'', b'test', b'', b'eth2', b'', b'']
666f6f6261721700000042421b00000076657279776f777465737465746832
b'foobar\x17\x00\x00\x00BB\x1b\x00\x00\x00verywowtesteth2'
31
[0, 0, 4, 0, 4, 0, 0]

If you're looking for a more realistic way of encoding offsets, take a look at the pyspec implementation:
https://github.com/ethereum/eth2.0-specs/blob/6f82480df2acd5627fee600aeccf0b975c8a1022/test_libs/pyspec/eth2spec/utils/ssz/ssz_impl.py#L72
Or, alternatively:
PySSZ (MIT license): https://github.com/ethereum/py-ssz/blob/master/ssz/sedes/base.py
ZSSZ (disclaimer: my code, experimental, MIT license): https://github.com/protolambda/zssz/blob/77bb42f406abe74d9da0cad199dbb0859d161484/types/elem_var_size_series.go#L12

Closing this PR:

introduces a bug in a 1 line change
syntax is off (placement of if in list comprehension)
don't see the alternative interpretation of if len(0) != 0 else 0 helping either. As else can't be returning None, since we also sum it.

But thanks for looking into improving the ssz code :+1:


Franz is firing the discrete gpu and draining the battery

Franz is firing my discrete gpu and draining my battery, can this be fixed?

#217

https://developer.apple.com/library/content/qa/qa1734/_index.html

@joltcan
Franz's Info.plist already contains:
<key>NSSupportsAutomaticGraphicsSwitching</key>
<true/>


Develop

Coverage increased (+0.005%) to 97.632% when pulling fe751f59f54782bfc48ed919f28dcc01e6a943e3 on develop into 764ec277c8cd3aee01ff3b931b449e6f6c400f22 on master.


Project: Simple Todo Console

Project name
Simple Todo Console App
Description
This is a simple Microsoft Todo Console app that is developed using Microsoft Graph .NET v5 SDK. It's login using Device Code and will save the access token into cross-platform cache MSAL so you don't need to login every time. The functions and menus available for the Todo console app is like below:
0. Exit
1. Display access token
2. Clear access token cache
3. Clear screen
4. List All To Do Task Lists
5. List All To Do Tasks
6. Display To Do Task
7. Add To Do Task
8. Update To Do Task

It provides some basic Todo info/functions for you to view/add/update via console app. Besides that, every time you start the console, it will display upcoming task(s) due within two days. The listing function supports paging and for search Todo task, it will either get the first matched task or display TOP x matched tasks for you to select.
Repo URL
https://github.com/sanme98/hack-together-simple-todo-console
Team members
sanme98

Thank you for your submission @sanme98! Would you be willing to record a short video to demonstrate the app? It's not required for the hackathon but it would help us showcase your work internally at Microsoft and publicly in the community 😊

![Simple Todo Console Demo]


little config touches for singularity

cleaned singularity-related comments from configs in favor of https://github.com/mirnylab/distiller-nf/issues/112
added docker:// to the container URI to make it compatible both with docker and singularity
added path for trace,report and timeline, so they do not pollute the distiller's "root" folder

Nope, nothing worked at the end..
We still need to manually modify the container to create /nl and/project there.
Don't close it though - I'll revert the docker://  - which seems to cause an error when used with actual docker,instead of singularity
The rest should be good - some comments cleanups and  folders for performance reports

@golobor this became an innocent cleanup and addition of maxForks #119 #100 and also beautiful nextflow html reports are now in a  folder pipeline_info instead of being in main-distiller folder - nice for cases where one has to resume a lot, which results in having trace.txt.1 trace.txt.2 ... in the main distiller-folder
can be merged if it is needed

I thought that one could get  trace.txt.{number}  only if the pipeline failed/terminated and one resumed it  - several times ...
That way indeed you get only successfull part of trace in each trace.txt.{number}
Did you get trace.txt.9 by resuming ? running different projects using the same instance of distiller ?
Also checkout report.html - it has the same info as trace, it's harder to parse but it's beautiful and maybe usefull for tuning requested resources


CIViC biomarker issues

mixed bag of things. will look into it and discuss with Darren on what needs to be done

What this means is that civic data contain various types of biomarker types, including things like 'overexpression' (which is not actually a problem, as this just means increased levels of ) and mutations and maybe others as well. Also, there is no cross-reference to the actual entity identifier in column D of the output. Instead the gene is indicated there. The gene information is useful, but is not the correct thing to put in that column. I'd expect something like a dbSNP identifier.


SPI: pin configuration is not flexible enough

Currently, we take Output pins for both MOSI and MISO. This creates some problems with devices like M5CoreS3 SE which uses a full-duplex SPI for the SD card, and a 3-wire simplex SPI setup for the display, reusing the MISO pin as the DC output.
Currently we can't work with this device safely.
cc #2494

Hi!
I'm having the same issue with the "Cheap Yellow Display" (https://github.com/witnessmenow/ESP32-Cheap-Yellow-Display), in that board the GPIO39 (an input-only pin in the ESP32) is the MISO for the touch screen, since the .with_miso() function requires a PeripheralOutput... it doesn't work. .with_mosi() also requires a PeripheralOutput but that makes sense of course.
I made a local fork of esp_hal and changed it so that it'll take a PeripheralInput and it works.
Interestingly this is only in the master.rs file... in slave.rs it uses a PeripheralOutput for MISO and a PeripheralInput for MOSI (the input/outpit directions are reversed, which makes sense since it is from the slave's perspective).


New feat: battery draining; model/os identifier

Citation: New vulnerabilities in 4G and 5G cellular access network
protocols : exposing device capabilities

Citation: New Vulnerabilities in
5G Networks


Support for remote locations (via ssh)

I would love to just use rsync-sidekick, but apparently this is restricted for cases where source and destination directory are local.
It would be extremely useful if it would support remote source or target. In the meantime, I created a little script (tailored to my needs) that does what I want (if I knew Go or would be interested in learning it, I would of course have just contributed to rsync-sidekick).
If someone is looking for something like that but with remote support, see here for some infos and here for the gist and adapt it to your needs.
I am currently not interested in working it out more than that, so would be happy if rsync-sidekick would subsume all the features so I can abandon my weekend project script for good.

I implemented github.com/helinwang/sync_mv that supports remote syncing.

I highly recommend switching to BorgBackup, which is a production-ready tool built for remote syncing with minimal duplication.

@alexandervlpl  Correct me if I'm wrong, but I've looked at the docs a bit and I don't see any way to sync directory structures between two machines using Borg Backup.  Is this possible, if one disables de-duplication, encryption, compression, etc. to actually replicate files on the target device which can be accessed outside the backup program?

As far as I know, you can backup a directory structure to a backup repository located on another machine. So no direct sync to a directory structure.
However, you can restore or mount the backup repository afterward. This would allow to finally recreate the same directory structure on the other machine. But this is indeed not the same as directly syncing two directory structures.
PS: I actually prefer Kopia over Borg as a backup solution. I find it more intuitive and maybe it is a little bit more modern. Though, it basically has a very similar command line interface.
PS2: For direct remote syncing two directory structures with deduplication, there is still syncthing. But with this you really need to do some set up steps and it is not just like a simple rsync command via ssh.

@michael-ts if you need a usable mirror on the other side, Borg (or Kopia?) may work but it won't be ideal.
Yes, you'll need to mount the backup on the second machine. AFAIK this is read-only. Also since this will be a FUSE filesystem and there's all this deduplication magic happening in the background, performance will probably be slow. You do want the deduplication though, that's what prevents all extra useless copying we want to avoid.

Did anyone try syncthing yet for such a usecase?
Using a backup tool instead of rsync seems like a "misuse" of tools. Even though it might work, goals are different.

Syncthing to me is more of a "Dropbox, but decentral and open source" use-case and in fact, I do not want it to happen continuously, because sometimes I can mess up my primary copy and want to use the "backup" - if you don't keep a "history" on the backup side, continuous sync sounds not like the right solution.

I thought synching might have a one-sync mode which only runs one. But that does not exist.

You can mount remote file system using sshfs and then use rsync-sidekick like normal on local file system.

@MOJNICK Technically yes, but it's worth noting that sshfs is super slow even with simple file operations in ideal conditions on a LAN. Something like rsync-sidekick will probably be unusable unless you're working with a very small number of files.

@alexandervlpl That's true.


[FR]: how to use rules_ts_dependencies(ts_version_from = "//:package.json") with bzlmod example?

What is the current behavior?
Using bzlmod it is not possible to sync typescript version with package.json
Describe the feature
I would like to be able to pass ts_version_from into rules_ts_ext.deps()

Looks like this was fixed by #378


Add Flutter to existing apps，flutter engine leak or doesn't release on iOS platform

➜  ~ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, v1.1.10-pre.193, on Mac OS X 10.14.2 18C54, locale zh-Hans-CN)
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
[✓] Android Studio (version 3.1)
[✓] Connected device (1 available)

• No issues found!

Closed #27149 via https://github.com/flutter/engine/pull/7610

when  should  i call the destoryContext method to  release the flutterviewcontroller?


Increment version for devcenter releases

Increment package version after release of Azure.ResourceManager.DevCenter

API change check
API changes are not detected in this pull request.


Capitalize

Version: 3.0-SNAPSHOT
Build: 2018-03-18 00:21
OS: Windows 10 10.0
JDK: 1.8.0_161 amd64 Oracle Corporation

Is it possible to make a feature so the "Title" and "Original Title" always is capitalize. (First Letter Is
Capitalized In Every Word)

nice idea.
Let's see, where we could integrate such checkbox...

where do you expect this feature?
at scraping, at renaming? only for GUI?

At scraping would be nice, maybe both. Its ok only for GUI for me.
Thanks.

Does capitalize coming soon???

we've migrated v3 to GitLab (https://gitlab.com/tinyMediaManager/tinyMediaManager) along with all issues. The issues for v2 at GitHub will be closed - we will respond to the corresponding issue in GitLab in the future
br
Manuel


Add support for Search bar

Add the ability to use search in the actionbar. Maybe return the search as a variable when enter is pressed.

any progress on this?

I don't think this is an active project anymore


Parsing NPM registry from .npmrc should support scope-specific registries

Currently, readRegistryFromNpmRc() only supports "global" registries:
https://github.com/oss-review-toolkit/ort/blob/db219febb94825b87e01a5d9e60239998050adff/analyzer/src/main/kotlin/managers/utils/NodeSupport.kt#L147
But we should support the syntax for scope-specific registries, too, see
# last modified: 01 Jan 2016
; Set a new registry for a scoped package
@myscope:registry=https://mycustomregistry.example.org

Ah, thanks for raising this. Now I know why I couldn't get our custom npm package to work.

This was obsoleted by https://github.com/oss-review-toolkit/ort/pull/5162 which removed readRegistryFromNpmRc() altogether in favor of a better solution based on npm view.


OS X Dark theme

Gitify doesn't seem to be visible when using the dark theme, it would be nice if that was supported somehow. Will look into this myself tomorrow.

According to this you need to use template images to support the dark theme.

Hi @jeroenvisser101 !  Was just reading this one! Looks like it's what we need!

If you have time now, it think that's better, because I have a very busy weekend :+1:

Alright will have a look then!

Cool! Let me know what you find, and if you need an extra hand, let me know!

@jeroenvisser101 It was a lot simpler than I thought! Ready for next release! :rocket:

Nice! Thanks for fixing this so fast :+1:


Passing class array as parameter causes huge memory usage

I tried to reproduce the behaviour in pseudo-code, quite close to the original one. Given enough memory, TSC will finish successfully but otherwise it gets killed with the following stacktrace:
<--- Last few GCs --->

[16386:0x5605d8e1a6b0]    50125 ms: Mark-sweep 1406.6 (1486.2) -> 1406.6 (1455.2) MB, 1177.1 / 0.0 ms  last resort
[16386:0x5605d8e1a6b0]    51212 ms: Mark-sweep 1406.6 (1455.2) -> 1406.6 (1455.2) MB, 1087.1 / 0.0 ms  last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xa5540ea66a1 <JS Object>
    1: set [native collection.js:~253] [pc=0xc9b5bc6502a](this=0xd7ada93c5d9 <a Map with map 0x33aba6684359>,q=0x10528320ada9 <String[13]: 724020,957028>,y=1)
    2: /* anonymous */ [/home/pasztet/Workspace/systemtests/node_modules/typescript/lib/tsc.js:909] [pc=0xc9b5b10ca69](this=0x25c15adccac1 <JS Global Object>,value
=1,key=0x10528320ada9 <String[13]: 724020,957028>)
    3: arguments adaptor ...

TypeScript Version:  2.3.2 / - Already occured on 2.2.x
Code
class Base {
    protected async weightedInit(): Promise<[this, number]> {
        return [this, 0];
    }
// A simple factory
protected static async sortInit<T extends IConstructor>(type: [{ new (...els: any[]): T }], ...els: any[]): Promise<T> {
let res = [];
        for (let t of type) {
            let created = new (Function.prototype.bind.apply(t, [null].concat(els)));
            let created = created.weightedInit();
            if (created) res.push(created);
        }
}
return res.sort((a, b) => b[1] - a[1])[0][0];
}
}
// And we call it with a multi-inheritance class
class A extends Base {
    public async init() {
        await Base.sortInit<A>([A, B, C, D, E], 0);
    }
}
class B extends A { }
class C extends A { }
class D extends B { }
class E extends D { }

Expected behavior:
The compiler will use a 'reasonable' amount of memory.
Actual behavior:
Depending on the amount of subclasses, it can use +4GB memory.
Workaround:
Change the factory signature to:
    protected static async sortInit<T extends IConstructor>(type: any, ...els: any[]): Promise<T> {

TSC will complete the operation in a matter of seconds.

can you give typescript@next a try? if it does not work for you either, we will need a self contained repro to diagnose further.

Automatically closing this issue for housekeeping purposes. The issue labels indicate that it is unactionable at the moment or has already been addressed.


UnicodeDecodeError occurs when running Python in a Rye environment with unicode strings on Windows

Steps to Reproduce
Environment

Windows 11 22H2 with Japanese location settings.

Steps

Create Rye environment with a name containing unicode strings.
Change the directory to the directory created in the 1st step.
Run rye sync
Run rye run python

Example
rye init ryeテスト
cd ryeテスト
rye sync
rye run python

Expected Result
PS C:\temp\rye_テスト> rye run python
Python 3.12.3 (main, Apr 15 2024, 17:48:16) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Actual Result
PS C:\temp\rye_テスト> rye run python
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1176, in exec_module
  File "<frozen site>", line 624, in <module>
  File "<frozen site>", line 607, in main
  File "<frozen site>", line 539, in venv
  File "<frozen site>", line 394, in addsitepackages
  File "<frozen site>", line 236, in addsitedir
  File "<frozen site>", line 188, in addpackage
UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 26: illegal multibyte sequence

Version Info
rye 0.35.0
commit: 0.35.0 (a1dbc56d4 2024-06-24)
platform: windows (x86_64)
self-python<EMAIL_ADDRESS>symlink support: true
uv enabled: true
Stacktrace
No response

While investigating this issue, I found https://github.com/python/cpython/issues/117793.
So, I checked the pth file in the .venv/Lib/site-packages directory and found 2 pth files (_rye_テスト.pth, _virtualenv.pth). Each file's text encoding was UTF-8.
I then changed the encoding of these files to Shift-JIS(CP932), which resolved the issue.
I hope this information is helpful to you.


urgent： multiple charts in one page,select legend to change the different charts, how to set legend?

check the bellow link, I reproduce it in this link.
http://gallery.echartsjs.com/editor.html?c=xgSLl-2GF0&v=1
pls help me to solve this, thks.

I modified it a bit http://gallery.echartsjs.com/editor.html?c=xgSLl-2GF0&v=2

@DorisOu What do you mean by "first line's name"? Can you show it in your demo?


WIP Fix render crash

Currently it crashes more, but I discovered that two threads are accessing SKCanvas which is not Thread Safe. So I have to introduce Locking so that only one Thread at a time Access SKCanvas and then the crashes should go away.

The current shape of the Code could change as I discover more things. So the caching could go back to less fine grained things.

Currently I'm debugging RasterizingTileLayer with ShapeFile. Several renderers access the same RenderCache and Render simultanously.

@inforithmics Thanks for looking into this. It is important to get this completely straightened out. I like to help. I have some questions/remarks.

My first suggestion is to invest in some rewriting to make debugging easier, without having the intent the solve the problem.

When two threads access the same resource they crash only occasionally. You could store the first thread in a static field. If a second thread comes in throw an exception, so you know right away things have gone wrong.
A thread in C# has an ID. Store the ID in a dictionary, how many entries do you get in the dictionary?
Check if a second thread comes in and in that case let the code stop on a breakpoint so you could check the stacktrace.

You mention MapInfo being on another thread accessing the SKCanvas. But I would think in our tests we don't usually do a MapInfo, we mostly just render a map. Perhaps this is seen in manual tests, but we need another explanation for the test fails?
The SKCanvas is not thread save. The MapControl renderer runs on the SkiaSharp Paint events (which is always on the same thread). A second thread could be MapInfo. Any others?
Could this be an phenomenon caused by the tests? Tests can run in parallel? Do they? Or can you reproduce the phenomenon running one test?
You talks about multiple renderers. Are those MapRenderer instances? If yes put a breakpoint in the constructor to check where those are created.
I am not a fan of locks. If it looks like we need them I like to look into a possible rewrite to avoid having to use them.
What is the best way for me to reproduce the problem?

I think, that problem would be much easier to solve, if we have different draw and paint routines. A draw could be done in multiple threads, a draw should be done always in the same thread.

This could solve it too. The first Bug I fixed with the lock was that the same shape was rendered at two places at the same time. And this caused an exception.
During editing the drawing of the map was active and the editing itself triggered a mapinfo request which too triggered a drawing operation causing the same skpath to be drawn at two canvases at the same time, which it didn't like it. So the current fix was a lock that only one Render operation is done in the renderer at a time.

@inforithmics Thanks for looking into this. It is important to get this completely straightened out. I like to help. I have some questions/remarks.
* My first suggestion is to invest in some rewriting to make debugging easier, without having the intent the solve the problem.

  * When two threads access the same resource they crash only occasionally. You could store the first thread in a static field. If a second thread comes in throw an exception, so you know right away things have gone wrong.

Good idea I'll try that, my first approach was to use the Thread Id as a cache key but didn't work at the first time, Intrestingly If I disable the caching than it works too. So it definitly has to do with the shared resource beeing used on two threads simultanously. The current crashing RasterizingTileLayerWithShapeFile has several threads accessing the same skpath objects rendering them on several seperate Canvases. And this causes crashes. So I plan to see what happens when I lock the shape/paint to see if it still crashes.

  * A thread in C# has an ID. Store the ID in a dictionary, how many entries do you get in the dictionary?

In the Editing sample there where 2 threads the ui thread (mapinfo request) and the render thread because of Fetch.
In the RasterizingTileLayer with Shapefile, there are more threads (and not locking yet) so it crashes. What I plan to do here is investigate if every thread has it's own Vector Cache if things work better. Was previously the case until I made the Cache Thread Safe.

  * Check if a second thread comes in and in that case let the code stop on a breakpoint so you could check the stacktrace.

When the crash happens with the option view all threads I see that two threads are rendering at the same time maybe accessing the same object.

* You mention MapInfo being on another thread accessing the SKCanvas. But I would think in our tests we don't usually do a MapInfo, we mostly just render a map. Perhaps this is seen in manual tests, but we need another explanation for the test fails?

The failing tests come because of RasterizingTileLayer uses more than one thread to faster render the tiles. I guess If I limit this to one thread, it will be slower but does not have anymore crashes.

* The SKCanvas is not thread save. The MapControl renderer runs on the SkiaSharp Paint events (which is always on the same thread). A second thread could be MapInfo. Any others?

The RasterizingTileLayer and RasterizingLayer Render images that run on other threads, because of Async Fetch. Somehow using the same SkObject at the same time on different threads isn't a good idea in Skiasharp it seems.

* Could this be an phenomenon caused by the tests? Tests can run in parallel? Do they? Or can you reproduce the phenomenon running one test?

Intrestingly with my changes, It crashes now very reliable meaning every time. Before it was not every time.

* You talks about multiple renderers. Are those MapRenderer instances? If yes put a breakpoint in the constructor to check where those are created.

Yes every Thread in RasterizingTileLayer has it's own MapRenderer but not it's own RenderCache.

* I am not a fan of locks. If it looks like we need them I like to look into a possible rewrite to avoid having to use them.

Me too. When I learn more I try to remove these.

* What is the best way for me to reproduce the problem?

In Main, you can open the Editing Sample and move around the Shape it crashes very quickly.

In this Branch you can open the RastizeringTileLayer with ShapeFile Sample and the it crashes immediatily, That's why the Unit Tests fail now.

Maybe to avoid locking I have to make a differnt Cache Scheme, When I use a SkPath object I have to remove it from cache so that it isn't used at the same time und when I'm finished I return it. But I first want to verify that the parallel usage of one SkObject is the problem. Or I could mark it as in use and if I need it I create one more. So that under the same key objects that are reused often have more than one cached instance.

About the lock. I now think a lock may be the best solution in the current architecture. There is no async/await within the renderer so it can be a simple lock (no SemaphoreSlim).
An alternative would be to sync everything to the same thread using queues and messages. That might be an option for MapInfo but it might be more complicated to get this to work for the tile rasterizer.
I would suggest to put the lock in the root methods of the renderer to keep the code simple to understand. In the alternative (using fine grained locks) there is more bloat added by the locking code and it is more likely to introduce errors in the locking, like forgetting to lock one render type, or making errors in the nesting of  locking. So, after having a short glance I think it should be in the public methods Render and RenderToBitmapStream.
When rendering is redesigned this might all work different, so lets keep the code easy to understand and not try to optimize for performance.

if every thread has it's own Vector Cache

This worries me. Caching is always a problem in all the apps I work with. We have too much growth of caches in different places already. We should get this under control. A problem with a cache per thread is that this could grow too big. The same data could be cached in different vector caches. If new thread are created every time. This causes the kind of bugs that you only find after running the app for an hour.

With the lock I discovered that the Skia objects can only used once on a Rendering surface and not in parallel. So I have an idea to improve the Cache so that it gives back a skia object and if it needs another one it creates a new one. With disposing I keep track of how many Skia objects are out there. So Objects that are only used once have one cache item and objects that are used more often like colors have one or more cached items. This would avoid the lock. and the several caches.

I managed to fix it without global locks, I track if a cached instance is in use if it is in use I create temporary a new one that is disposed after use. I tested caching more than one instance but, the caches didn't reuse so not worth the extra effort.

I'm trying to reduce the number of casts, in the caches.

I got more stable with my changes but crashes now at another test. When I introduce an Iteration lock the crashes go away. So I think I let this lock there until I have a better idea to get rid of it.

@inforithmics Sorry, for causing a conflict.

@pauldendulk no problem, I only changed the Regression Image because it changed showhow. I guess it could be something Framework related.


Firefox 8+ compatible please?

Can you fix this so it stops breaking every time firefox increments their version number?

RESTClient 1.4.0 will support Firefox 11.0


Allow health and liveness checks per-process type

Relates to #879
After health checks are moved routeable process types by default, we should support setting process-type specific health checks.
deis healthcheck:{list,set,unset} should take a new argument --type=<proc type>
The list option should either enumerate checks for all process types by default or take an --all flag.

So I assume you'd be able to provide a readiness check for each process type? Our use case is we want the default for cmd (webapp) and then an exec readiness check for sidekiq

Exactly, that type of separation is what we are shooting for!


Isolate specs from std lib when possible

Ping @myronmarston I went through and isolated out specs from the standard library.

This LGTM.  I'd still like to understand what problem you feel this solves, though.  In #730 you said:

That's only half my concern. The rest of it is external test suites.

I still don't understand what "the rest of it is external test suites" means.  Can you explain a problem external test suites would have that this will solve?

Yeah I think I might have been wrong with the initial implications of using require in our specs but I'd still like to do this to be on the safe side, initially the missing require was highlighted by a build over on rspec-core but I'd still like us not to have these components loaded if we don't need to. I feel it reduces our cross section and makes us less likely to accidentally rely on the standard library.
IMO this is important and I actually noticed Travis was far more sensitive to missing components than my own machine, I'm hypothesising that this is related to me using rvm and not bundler bin stubs...

IMO this is important and I actually noticed Travis was far more sensitive to missing components than my own machine, I'm hypothesising that this is related to me using rvm and not bundler bin stubs...

Do you use --standalone (as our travis builds do and I do)?  Besides speeding things up by avoiding the bundler tax at runtime, the act of avoiding bundler/rubygems at runtime means any stdlibs used by those aren't automatically loaded, which could affect here.
Anyhow, merge away.

Do you use --standalone (as our travis builds do and I do)? Besides speeding things up by avoiding the bundler tax at runtime

Nope, I have rvm gemsets instead, same performance enhancement.


Saving re-attached entities configured with ForNpgsqlUseXminAsConcurrencyToken() throws DbUpdateConcurrencyException

Scenario:
When detached entity is attached to new context and set to Modified state in order for SaveChanges() to pick-up updates the DbUpdateConcurrencyException is thrown if the entity is also configured with ForNpgsqlUseXminAsConcurrencyToken().
minimal example

I should probably add that in the example there are no concurrent updates going on. In fact there's single write requested, so it's clearly a bug.


Pyca 593

Further instructions added for making a decision in a refugee case -
the agent needs to create a further to do

Yep. Looks good. Merging. 👍


updated vagrant incompatiblity with virtualbox 5.1

Updated known compatibilities list in deploy and linked to the troubleshooting doc.

Looks good. Thanks!


Linear Neural Networks/Implementation of Softmax Regression from Scratch

Job d2l-zh/PR-557/1 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/2 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/3 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/4 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/5 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/6 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/

Job d2l-zh/PR-557/7 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-557/


[FLINK-32299][gateway] Upload python jar when sql contains python udf…

… jar

What is the purpose of the change
Upload python jar only when sql contains python udf jar.
Brief change log

Do not add python jar in the initialization of DefaultContext.
Add python jar to resourceInfos in resolving python functions.

Verifying this change
This change is already covered by existing tests, such as PyFlink e2e.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? (not applicable

Thanks @KarmaGYZ, LGTM +1


Search Page UI updated

Linked Issue #77
Changes made
Changed the UI according to the new flutter design, also due to some bugs and errors the color scheme is changed a little bit.
How does the solution address the problem
This PR will focus on the UI enhancement of the search repo page and will give the user a better experience of using the App.

@dk-a-dev sorry for the delay. Can you please review it ASAP so that this contribution will be counted in hacktoberfest ?

Yup sure buddy i am on it, I also want to get it merge before tomo
plz be active if I post a review to make quick fixes if any

the new UI doesnt look neat and there bugs in colour scheme

contributors page card not updated

the repo card needs a lot of improvement, its not like one in design

@SarveshChavan

@SarveshChavan if u are struggling with something let me knw, i can help, @Lomesh21 is also willing to help

Sorry for the mistake but when I click wo dark theme in my machine the theme is changing properly and also the issue with repo card is because we can't fet h the followers and the contact details from our api (the response of get repos method). Also the slimy widget has some constraints of its own such as the cards height should be 150 minimum and if I increase it for the other 2 fields(followers and contact) the card height get increased and when we increase the slimy widget card height it produces an extra blank card after each slimy widget. But I'll still try to make it possible and I'll let you know. I'll change it tomorrow and also will take the pull to avoid conflict.

The switch button working fine the theme is getting changed, @dk-a-dev review the PR, and also the changes that have been made in the contributors page and drawer need to be changed according to theme.


SPARK-43166: name docker users

What changes were proposed in this pull request? / Why are the changes needed?

Currently, the official Spark docker images run as the UID 185, but no corresponding entry exists in /etc/passwd. This causes issues when libraries try to fetch the current unix username.
This PR invokes the useradd function as part of the Dockerfile, to ensure that a name exists for these users.

Does this PR introduce any user-facing change?

No. No user-facing changes are expected.

How was this patch tested?

Docker build succeeds locally.

@DerekTBrown Mind creating a JIRA and add it to the PR title please? See also https://spark.apache.org/contributing.html

@DerekTBrown I have an alternative solution https://github.com/apache/spark/pull/40831, which should cover your case

Looks good. Closing in favor of #40831


Better processors: processing any field

Currently built-in processors can only manipulate fields Record.Key and Record.Payload.After. We should change processors so they can be configured to work on any of the openCDC fields, maybe even multiple fields at the same time (e.g. copy payload data into metadata).

The new list of processors and the way they will address fields is described in the design document, will be tackled in https://github.com/ConduitIO/conduit/issues/1290.


2.13: sbinary fails to extract

https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/651/console
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[sbinary] [warn] 	::          UNRESOLVED DEPENDENCIES         ::
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[sbinary] [warn] 	:: org.scala-lang.modules#scala-xml_2.13.0-pre-12b8cfb;1.0.5: not found
[sbinary] [warn] 	:: org.scalacheck#scalacheck_2.13.0-pre-12b8cfb;1.13.4: not found
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::

note that we unforked sbinary in 99e8031c6aa80f490f9979fa41072755b55b58e5. this failure started when I merged the change onto 2.13.x

perhaps https://github.com/sbt/sbinary/pull/10 will fix it. even if not, it's a good baseline to start from

hmm, at https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/654/consoleFull we still see:
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[sbinary] [warn] 	::          UNRESOLVED DEPENDENCIES         ::
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[sbinary] [warn] 	:: org.scalacheck#scalacheck_2.13.0-pre-12b8cfb;1.13.5: not found
[sbinary] [warn] 	::::::::::::::::::::::::::::::::::::::::::::::

this is a tough one...
I suspect this snippet of code in sbt-houserules:
// WORKAROUND https://github.com/sbt/sbt/issues/3393
def addSbtPlugin(m: ModuleID) =
  libraryDependencies += Defaults.sbtPluginExtra(
    m,
    (sbtBinaryVersion in pluginCrossBuild).value,
    (scalaBinaryVersion in update).value
  )

dependency extraction shouldn't normally cause library dependencies to be retrieved, but perhaps (scalaBinaryVersion in update).value is the culprit

this is a tough one. I don't have a clue what's going on. why would dbuild/sbt attempt to actually retrieve a dependency during dependency extraction?

I should enable dbuild's debugging mode and look at the resulting log; that's an obvious thing I haven't tried yet.

full debug-mode log: https://gist.github.com/SethTisue/e74c824f82daff525a16db2a428482c0
I don't see any new clues :-\

asked for volunteer extra pairs of eyes on Slack. anyone watching this repo who wants to take a look, be my guest. not sure if there's something weird and elusive going on here? maybe it's something obvious staring me in the face that I'm somehow not seeing

@cunei any insight?

It seems you are using "2.13.0-pre-12b8cfb" as a Scala version during extraction. The failed dependency is not one of the main build level, but one of the plugins level. So sbt is just trying to compile the build definition, before dependency extraction, when the failure occurs. There may just not be a scalacheck published against 2.13.0-pre-12b8cfb. The same error occurs if you check out sbinary from scratch, change the scalaVersion in the build file, and launch "sbt" pointing to an empty ivy cache directory, before you even get to the prompt.

I am not sure why the plugin level would try to use that scalaVersion rather than 2.10, however

exactly

I'll have a look.

thanks!

The failure is due to the different version of sbt-houserules. Version 0.3.3 works ok, but version 0.3.4 leads to the incorrect dependency.

0.3.4 adds:
+sbtVersion in Global := "1.0.0" // must be Global, otherwise ^^ won't change anything

maybe that's confusing dbuild? sbinary's own project/build.properties still says 0.13.16

this, in 0.3.4, also looks like it could trip up dbuild:
+// WORKAROUND https://github.com/sbt/sbt/issues/3393
+def addSbtPlugin(m: ModuleID) =
+  libraryDependencies += Defaults.sbtPluginExtra(
+    m,
+    (sbtBinaryVersion in pluginCrossBuild).value,
+    (scalaBinaryVersion in update).value
+  )

dbuild is not involved. You get the same result if you check out commit 2a1cfe45192719e118bd8a87b1c8ae2284fc6d61 of sbinary, change scalaVersion to 2.13.0-pre-fa78b6d, and start sbt.

hmm. scalaBinaryVersion in update seems especially suspicious?

I dealt with it (in 87ce08eae13235aa382c12e22e809edbd4e5dc29) by forking sbinary and downgrading sbt-houserules in the fork.
it should work to unfork once sbt 0.13.17 is out that fixes the underlying problem (sbt/sbt#3393) and once a new sbt-houserules has been released that removes the workaround code inserted in 0.3.4 for that issue
thx @cunei for your help!

ugh, I should have seen this coming: downgrading sbt-houserules also downgrades from sbt-bintray 0.5 to bintray-sbt 0.3, and then we run afoul of the usual dbuild/bintray incompatibility

scalacommunitybuild/sbinary@3fa07bf377b81c0579e78936b54e2d181c4fc542 just removes sbt-houserules altogether. done.

over at https://github.com/sbt/sbinary/pull/11 I've submitted a PR that, once merged, should permit unforking


Customizing result out?

Is there any way to define callbacks or handlers to modify the search result output?
I'm hosting Pagefind from under /static, so when it renders links to results, a page that should be located at /somepage.html is linked to as /static/somepage.html.
How do I configure Pagefind to remove it's own prefix from the search results, or otherwise edit the result link and correct it?

Nevermind. I overlooked the process result section in the docs.


kafkasql - operator does not reconfigure the deployment if kafka bootstrap servers is modified

looks like it only happens if the pods are failing, because the first configured values are wrong, and then when you try to fix it changing the config values the pod is not replaced with a new one with the updated config values

weird, it actually works properly, maybe my cluster lagged


Fail e2e test docker script on command failure

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/test eks-anywhere-e2e-presubmit

/retest


Create View "Project Overview" Backend

User story

As a {}
I want / need {}
So that {}

Acceptance criteria

Criterion 1
Criterion 2
...

Definition of done

Added only after week 5
The same for all features

Issues was aborted


Re-add Minification Mapping Input Suggestion

Looks like this ability used to be in the compiler, but was removed long ago. A piece of the functionality was re-added for #433, but not the piece allowing for an input mapping to use as a starting point.
It's a very common practice for website to use hashed file names to allow browsers to cache unchanged portions of scripts, especially as a part of a webpack bundle, which allows only certain portions of the bundle to need to get re-downloaded by users in the new version.
However, if you're using Closure Compiler as a minifier (perhaps via the webpack plugin), the instability of the minification will mean that all chunk contents will likely change every build, even if there were no real changes to the contents of some of the chunks.
In an ideal world, we'd be able to give Closure the minification map of the previous build, which it could then use as a "suggestion" for the current mapping. If there's a big enough upside, it can choose to re-map any or all of the identifiers. This means that for cases where there's no upside to re-mapping, Closure could choose stability, perhaps allowing more of those chunks to have unchanged contents.
The main pitch here is to make minification slightly less "chaotic" (in the sense that small changes anywhere within the scope can lead to big output differences everywhere) for cases where there is little-to-no upside.

Also good for any diffing use, including differential updates.

To expand on the use case described - if one wishes to use Compress Dictionary Transport (https://github.com/WICG/compression-dictionary-transport) this is likely a must. Without it, the typical diff between builds will be too large to be useful.
I found that sometimes even small actual difference results in a size diff of over 40% due to renaming which is inconsistent between builds.
What would it take to get this prioritized?

It looks like the feature might still be present, just not plumbed to the command line?
https://github.com/google/closure-compiler/blob/558e96f386c378ff8ef9cbdc36a3ba266d008517/src/com/google/javascript/jscomp/AbstractCommandLineRunner.java#L2759C20-L2769
Have you tried generating the prop/variable maps and then reading them using this to confirm it doesn't work? I recognize that the commit linked at the top claims that it isn't intended for this purpose, but the tests seem to confirm that it works, and there are compiler passes that check for the presence of these input maps and make use of them to keep names consistent:
https://github.com/google/closure-compiler/blob/558e96f386c378ff8ef9cbdc36a3ba266d008517/src/com/google/javascript/jscomp/RenameProperties.java#L220-L243
etc

This is what I'm thinking when I looked at this code earlier.
I plan to try it and will report results here.

Update on my testing - when enabling input maps on a local build of the compiler I was able to create a brolti diff of 10% between 2 production versions of our code. This is compared to the 40% diff without it. This means that this feature is very important and will play a significant part in our efforts to roll out Compress Dictionary Transport (CDT)(https://github.com/WICG/compression-dictionary-transport)
I'm working on submitting a PR with the changes.


Failed in running cr.tl.lineages() on Ubuntu 20.04 LTS

Hello there!
I created a clean virtual environment with conda and installed cell rank by pip install cellrank in it. All the upstream processing with scvelo is superb, however I cannot proceed with the absorption probability calculation by cr.tl.lineages(adata). It didn't show any error message but just stopped responding without any progress. I used to run cell rank on macos and it was fine but since I changed my work on a linux based station recently, I could not figure out why.
Computing lineage probabilities towards terminal states
Computing absorption probabilities
0%
0/60 [00:00<?, ?/s]

KeyboardInterrupt                         Traceback (most recent call last)
 in
1 #absorption probabilities
----> 2 cr.tl.lineages(adata)
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/cellrank/tl/_lineages.py in lineages(adata, backward, copy, return_estimator, **kwargs)
84
85     # compute the absorption probabilities
---> 86     mc.compute_absorption_probabilities(**kwargs)
87
88     logg.info(f"Adding lineages to adata.obsm[{lin_key.s!r}]\n    Finish", time=start)
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/cellrank/tl/estimators/_base_estimator.py in compute_absorption_probabilities(self, keys, check_irreducibility, solver, use_petsc, time_to_absorption, n_jobs, backend, show_progress_bar, tol, preconditioner)
472
473         # solve the linear system of equations
--> 474         mat_x = _solve_lin_system(
475             q,
476             s,
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/cellrank/tl/_linear_solver.py in _solve_lin_system(mat_a, mat_b, solver, use_petsc, preconditioner, n_jobs, backend, tol, use_eye, show_progress_bar)
485         )
486
--> 487         mat_x, n_converged = parallelize(
488             _solve_many_sparse_problems,
489             mat_b,
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/cellrank/ul/_parallelize.py in wrapper(*args, **kwargs)
98             pbar, queue, thread = None, None, None
99
--> 100         res = jl.Parallel(n_jobs=n_jobs, backend=backend)(
101             jl.delayed(callback)(
102                 *((i, cs) if use_ixs else (cs,)),
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/parallel.py in call(self, iterable)
1039             # remaining jobs.
1040             self._iterating = False
-> 1041             if self.dispatch_one_batch(iterator):
1042                 self._iterating = self._original_iterator is not None
1043
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
857                 return False
858             else:
--> 859                 self._dispatch(tasks)
860                 return True
861
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/parallel.py in _dispatch(self, batch)
775         with self._lock:
776             job_idx = len(self._jobs)
--> 777             job = self._backend.apply_async(batch, callback=cb)
778             # A job can complete so quickly than its callback is
779             # called before we get here, causing self._jobs to
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
206     def apply_async(self, func, callback=None):
207         """Schedule a func to be run"""
--> 208         result = ImmediateResult(func)
209         if callback:
210             callback(result)
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/_parallel_backends.py in init(self, batch)
570         # Don't delay the application, to avoid keeping the input
571         # arguments in memory
--> 572         self.results = batch()
573
574     def get(self):
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/parallel.py in call(self)
260         # change the default number of processes to -1
261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262             return [func(*args, **kwargs)
263                     for func, args, kwargs in self.items]
264
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/joblib/parallel.py in (.0)
260         # change the default number of processes to -1
261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262             return [func(*args, **kwargs)
263                     for func, args, kwargs in self.items]
264
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/cellrank/tl/_linear_solver.py in _solve_many_sparse_problems(mat_b, mat_a, solver, tol, queue)
246     for b in mat_b:
247         # actually call the solver for the current sub-problem
--> 248         x, info = solver(mat_a, b.toarray().flatten(), tol=tol, x0=None, **kwargs)
249
250         # append solution and info
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/scipy/sparse/linalg/isolve/iterative.py in gmres(A, b, x0, tol, restart, maxiter, M, callback, restrt, atol, callback_type)
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/scipy/_lib/_threadsafety.py in caller(func, *a, **kw)
42         def caller(func, *a, **kw):
43             with self:
---> 44                 return func(*a, **kw)
45         return scipy._lib.decorator.decorate(func, caller)
46
~/miniconda3/envs/cr1.3/lib/python3.8/site-packages/scipy/sparse/linalg/isolve/iterative.py in gmres(A, b, x0, tol, restart, maxiter, M, callback, restrt, atol, callback_type)
598         olditer = iter_
599         x, iter_, presid, info, ndx1, ndx2, sclr1, sclr2, ijob =
--> 600            revcom(b, x, restrt, work, work2, iter_, presid, info, ndx1, ndx2, ijob, ptol)
601         if callback_type == 'x' and iter_ != olditer:
602             callback(x)
KeyboardInterrupt:
Versions:
Package             Version

anndata             0.7.5
argon2-cffi         20.1.0
async-generator     1.10
attrs               20.3.0
backcall            0.2.0
bleach              3.3.0
cellrank            1.3.0
certifi             2020.12.5
cffi                1.14.5
click               7.1.2
cycler              0.10.0
decorator           4.4.2
defusedxml          0.7.1
docrep              0.3.2
entrypoints         0.3
environment-kernels 1.1.1
future              0.18.2
future-fstrings     1.2.0
get-version         2.1
h5py                3.2.1
ipykernel           5.5.3
ipython             7.22.0
ipython-genutils    0.2.0
ipywidgets          7.6.3
jedi                0.18.0
Jinja2              2.11.3
joblib              1.0.1
jsonschema          3.2.0
jupyter-client      6.1.12
jupyter-core        4.7.1
jupyterlab-pygments 0.1.2
jupyterlab-widgets  1.0.0
kiwisolver          1.3.1
legacy-api-wrap     1.2
llvmlite            0.35.0
loompy              3.0.6
MarkupSafe          1.1.1
matplotlib          3.4.1
mistune             0.8.4
natsort             7.1.1
nbclient            0.5.3
nbconvert           6.0.7
nbformat            5.1.2
nest-asyncio        1.5.1
networkx            2.5
notebook            6.3.0
numba               0.52.0
numexpr             2.7.3
numpy               1.20.2
numpy-groupies      0.9.13
packaging           20.9
pandas              1.2.3
pandocfilters       1.4.3
parso               0.8.2
patsy               0.5.1
pexpect             4.8.0
pickleshare         0.7.5
Pillow              8.1.2
pip                 21.0.1
progressbar2        3.53.1
prometheus-client   0.9.0
prompt-toolkit      3.0.18
ptyprocess          0.7.0
pycparser           2.20
pygam               0.8.0
Pygments            2.8.1
pygpcca             1.0.2
pyparsing           2.4.7
pyrsistent          0.17.3
python-dateutil     2.8.1
python-igraph       0.9.1
python-utils        2.5.6
pytz                2021.1
pyzmq               22.0.3
scanpy              1.7.1
scikit-learn        0.24.1
scipy               1.6.2
scvelo              0.2.3
seaborn             0.11.1
Send2Trash          1.5.0
setuptools          54.2.0
sinfo               0.3.1
six                 1.15.0
statsmodels         0.12.2
stdlib-list         0.8.0
tables              3.6.1
terminado           0.9.4
testpath            0.4.4
texttable           1.6.3
threadpoolctl       2.1.0
tornado             6.1
tqdm                4.59.0
traitlets           5.0.5
typing-extensions   <IP_ADDRESS>
tzlocal             2.1
umap-learn          0.4.6
wcwidth             0.2.5
webencodings        0.5.1
wheel               0.30.0
widgetsnbextension  3.5.1
wrapt               1.12.1

Python 3.8.5

Hi @tslimax ,
thanks for the log. From what I can see, you have 2 terminal states and you're solving 60 lin. equations using scipy and the gmres solver. How many cells do you have in your AnnData object?
I think the problem is the scipy's solver, since the KeyboardInterrupt happened during the opt. procedure - for large or ill-conditioned problems, it may not work as well as solvers from PETSc.
Another possibility is that there's something wrong with the joblib's parallel backend ('loky' is used by default), but I don't think this should be causing the issue, since you don't specify n_jobs=2 or more). Just to be sure, you can try:
cr.tl.lineages(adata, backend='threading')

but I don't think this will do anything.

Hi @tslimax, did this help?

Thanks for your reply, and I have actually found the solution today after receiving some hints from @michalk8.
The number of cells in my current dateset is exactly 10k, I guess that's why cr.tl.lineages(adata) didn't work as expected in the first place. I have found the solution in the low-level mode tutorial where it stated that PETSc library should have been used if the data set is large.
Low-level mode tutorial

So cr.tl.lineages(adata, use_petsc=True) solved the problem, otherwise it will full load my cpu and stop responding.
I have tested the same code and environment (pip install 'cellrank[krylov]', python=3.8.5) across macos, ubuntu and centos and the same error appeared in all devices if the number of terminal states is 2 and without adding use_petsc=True while calculating absorption probability. The reason why it can run in my old macbook was because that dataset has more than 2 terminal states.
This error is reproducible if there are only 2 terminal states without adding use_petsc=True, for example:

g.compute_macrostates(cluster_key='clusters')
g.set_terminal_states_from_macrostates(["1","2"])

Or

g.compute_macrostates(n_states=2, cluster_key='clusters')
g.compute_terminal_states()

but it wasn't affected if I manually set the 2 clusters as terminal states by
g.set_terminal_states()


Support multiple independent kernels

This is a initial attempt at supporting multiple independent kernels.
There are now three commands to start a new kernel:

"Start Local Kernel" (default when executing code in a new file)
"Connect to Existing Kernel" (replaces "Switch Kernel")
"Connect to Remote Kernel"

Main Changes in this PR:

Kernels are indexed by file path
By default a new ZMQ kernel is started when users runs code in a new file
Users can connect multiple files to the same kernel via "Connect to Existing Kernel". This also supports remote kernels
"Switch Kernel" command is removed since it's now obsolete
"Select Kernel" command is renamed to "Start Local Kernel"
New "Kernel Monitor" panel to manage all running kernels (open with "Toggle Kernel Monitor") :

This should be ready for testing and review.
Fixes #40

nice work!


Consider replacing FluoFileOutputFormat with an Accumulo Key Builder

See discussion on ngageoint/geowave#157

Should this be marked as 1.0.0-beta?

Should this be marked as 1.0.0-beta?

yeap!


[Code Cleaning]: Use "HaveLen()" instead of "len(...).To(Equal(..)" in vmi preset admitter test

What this PR does / why we need it:
This small PR replaces all appearences of Expect(len(X)).To(Equal(Y)) to Expect(X).To(HaveLen(Y)) for better readability and error presentation.
Release note:

NONE

/retest

/retest

/test pull-kubevirt-e2e-k8s-1.23-sig-storage-cgroupsv2

/lgtm


Update post-lecture-quiz.md

Hi, please don't submit the answers to quizzes here but rather fork the repo and do the quizzes on your own, thank you￼

Oh sorry


tests_tls: generate certs on managed nodes to avoid installing local RPM

[citest pending]

BaseOS CI does not have openssl and it's generally cleaner to not install additional RPMs on control node when possible.


When I tried to type command "sudo apt-get install clang cmake ninja-build pkg-config libgtk-3-dev" in my Linux mint 20.1.
This what I get
.
.
pkg-config is already the newest version (0.29.1-0ubuntu4).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
clang : Depends: clang-10 (>= 10~) but it is not going to be installed
libgtk-3-dev : Depends: libatk-bridge2.0-dev but it is not going to be installed
Depends: libatk1.0-dev (>= 2.15.1) but it is not going to be installed
Depends: libcairo2-dev (>= 1.14.0) but it is not going to be installed
Depends: libfontconfig1-dev but it is not going to be installed
Depends: libgdk-pixbuf2.0-dev (>= 2.40.0) but it is not going to be installed
Depends: libglib2.0-dev (>= 2.57.2) but it is not going to be installed
Depends: libpango1.0-dev (>= 1.41) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

The issue tracker use for Flutter bug reports and feature request; apt is not part of Flutter.
If you are having issues installing prerequisites for Flutter, try a general support forum for your operating system.


Hyphenate adjective?

Sorry, total nit, if "Swagger-specific" is a term describing "annotations", maybe it's easier to read with a hyphen?
I don't think what's there is wrong, I was just scanning this document and got tripped up over "The Swagger specific annotations...", so if I tripped, maybe others will too.
Just a thought. Overall, love the project!

@fehguy thanks! Wow, you are fast! :)

except when i'm not :)


Example of Rotate Tool error when run with ipad pro enviroment dev tool

Hi guys,
As title, i have test your example of rotate tool with ipad pro and some error threw. Please check this issue

Interesting. The only viewer I've used that had touch support, used the RotateTouchTool to accomplish it. Rotate tool is likely failing because it doesn't have the eventdata it expects? (although, it should still be normalized and available?)
https://github.com/cornerstonejs/cornerstoneTools/blob/master/src/tools/RotateTool.js#L59-L72

Related: https://github.com/cornerstonejs/cornerstoneTools/pull/1112

Hi @dannyrb,
Can you explain how to use multitouch rotate, i can rotate 360