[rfr] EmbargoTerminationApproval

:revolving_hearts:
https://openscience.atlassian.net/browse/OSF-5641
:bouquet: :confetti_ball: :bouquet:
Purpose
We want a way for users to be able to end embargoed registrations public early. Borrowing from the patterns for Registrations, Embargoes, Retractions, etc, we will notify admins on the registration tree and request that they either approve or disapprove this action. If an admin fails to respond within 48hrs, they have granted implicit approval. If a user is the only project administrator, then the Embargo will be lifted immediately.
Changes

Sanction and its subclasses are moved to their own module (website.project.sanctions)
add new Sanction subclass EmbargoTerminationApproval
add NodeLog.EMBARGO_TERMINATED/EMBARGO_TERMINATION_APPROVED/ EMBARGO_TERMINATION_CANCELLED and supporting templates (TODO)
update nodesPrivacy.js/mako to support making Embargoes public
move NodePrivacy instantiation into NodeControl, and pass onPrivacyChanged handler into NodePrivacy
add tests for Node sanction states (is_pending_registration, is_embargoed, etc)

Side Effects
The logic for restricting what can be made public/private is fairly scattered, so I wouldn't be surprised if some places cause issues (see: https://github.com/samchrisinger/osf.io/commit/0d1821dae05a24c2f5e1c2ea869ca1f837b0d2e8#diff-7a6d21a50624153d15e63a5f643401c7). Hopefully testing will resolve most or all of these.
Additionally this adds a new auto-approve script (like Embargoes and ReigstrationApprovals), that will needed to be added to celery-beat (?).
TODOS

[x] email templates
[x] log templates
[x] UI when embargo termination request is pending
[x] tests and testing
[x] EmbargoTerminationApproval auto-approve scripts

Screenshots

Punchlist—for @samchrisinger

[ ] make Travis happy

Investigations—for @whit537

[ ] read code
[ ] run code

@whit537 waiting on https://github.com/CenterForOpenScience/osf.io/pull/5371 to fix Travis woes

Having some trouble standing this up. Starting from a fresh database, checkout, and assets build: I login and I get a blank / and a blank /myprojects/. I see a 404 in the console that might be a clue ...

My expectation was that I'd be able to add projects from /myprojects/. Hmm ...

@whit537 Are you running webpack locally? It looks like you might be loading an older version of the dashboard. The /v1/dashboard/*  route(s) no longer exist.

Are you running webpack locally?

I thought I was ...

It looks like you might be loading an older version of the dashboard. The /v1/dashboard/* route(s) no longer exist.

Hmm, okay, let's see here ...

+1 to chrisseto's comment. Perhaps run
inv clean
and make sure
inv assets -dw
is running

inv clean

Hmm ... that just removes *.pyc files (which I suppress globally with PYTHONDONTWRITEBYTECODE). inv clean_assets on the other hand ...

Yessssssssss!
Okay! Thanks for the tip. :-) Proceeding with review ...

Okay, now with 6188fda I get a SyntaxError inside Mithril. Hmm ...

@whit537 Its probably an exception from the API as HTML. Mithril is trying to parse it as JSON.

Thanks.

Alright, I threw a breakpoint (using the Chrome dev tools) in mithril. I hit it three times when loading /myprojects/ ... LOL
Okay, this is getting ridiculous. :-)
One one run through the breakpoints, /myprojects/ loaded (more or less?) as expected. And then on the next ...

@whit537 seems like your HD may be close to full.
There are no changes introduced into the dashboard/myprojects code in the PR.

seems like your HD may be close to full.

Right. I guess 5.2 Gi isn't enough. :(
$ df -h
Filesystem      Size   Used  Avail Capacity  iused   ifree %iused  Mounted on
/dev/disk1     112Gi  106Gi  5.2Gi    96% 27949960 1371766   95%   /
devfs          191Ki  191Ki    0Bi   100%      660       0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%        0       0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%        0       0  100%   /home
$

There are no changes introduced into the dashboard/myprojects code in the PR.

I know, that's why this is so weird. I don't believe I've touched the OSF since the last time I was here on this PR.
I'll stop posting here to keep the noise down ...

I am unfrobulated (#5419). Sorry for the distraction, @samchrisinger @chrisseto. I guess the lesson is "check your local.py". Okay! Back on track here ...

I can create am embargoed registration. This PR is about lifting embargoes early, I guess ...

Made the project public, not seeing the embargo notice ... because I'm still on aa1042c. ;-)

Alright, I'm on 6188fda and still not seeing the "End Embargo Early" notice, gonna have to try harder ... maybe it's a clue that the registrations are still in draft even though I thought I registered them?
Oh well, more tomorrow. :-)

I'm trying to get a registration out of "Draft Registrations" and into "Registrations."

I'm now trying with an open-ended registration and no embargo, just to get something working. When I "Continue" on the registration choice dialog I see my new registration listed under "Registrations," but then when I refresh it is gone. Maybe something with file copying failing?

Hmmm ... I left it a long time, and then refreshed. Now it's still there, but I can't find anything to do with it. Nothing is clickable.

Eep! Now it's gone again. :disappointed:

All these drafts have a disabled "Register" button.

I'm sorry to be so dense, @samchrisinger. Can you give me the steps to create an approved, embargoed registration? It looks from the Jira description that that's the starting point for the functionality on this PR, but I don't know how to get there. :disappointed:

Digging into this some more ...

@whit537 sorry, haven't had a ton of time to respond to this.
Are you running rabbitmq and celery locally?

Are you running rabbitmq and celery locally?

Yes, I am.

I find some "registration" collections in Mongo ...

sorry, haven't had a ton of time to respond to this.

Maybe I will reach out to @csoderberg, she gave me my first product orientation to OSF. Perhaps she can help me understand the registration and embargo system.
@csoderberg Up for a Skype sometime today or tomorrow? :-)

I've also reached out on Jira.

@whit537 responded on flowdock with options

@whit537 see updated PR description. No problem creating approved embargoed registrations on my end.

Okay, so Step 2.v is where I'm hung up, and the message I get in the UI suggests that a malfunctioning waterbutler may be in teh way of me and my email. Reopening https://github.com/CenterForOpenScience/osf.io/issues/5419 to investigate ...

Files are being copied to the newly created registration, and you will receive an email notification when the copying is finished.

Okay, reunfrobulated! I have an approved embargoed registration!

And making the registration public does give me the "End Embargo Early" dialog, which does appear to work.

Running through with multiple admins ...

Sweet mercy. I'm really sloggin' through the mud here, aren't I? 😳  😩
I signed up a second contributor in a different browser session, and when I try to add the second contributor from the first session I get a 500 from the search endpoint.
[werkzeug]  INFO: <IP_ADDRESS> - - [11/Apr/2016 15:39:07] "GET /api/v1/user/search/?query=bar&page=0&_=1460407105699 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/whit537/workbench/cos/osf.io/framework/routing/__init__.py", line 118, in wrapped
    data = fn(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/framework/auth/decorators.py", line 22, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/views.py", line 208, in search_contributor
    exclude=exclude, current_user=user)
  File "/Users/whit537/workbench/cos/osf.io/website/search/search.py", line 19, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/search.py", line 89, in search_contributor
    exclude=exclude, current_user=current_user)
  File "/Users/whit537/workbench/cos/osf.io/website/search/elastic_search.py", line 79, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/elastic_search.py", line 593, in search_contributor
    if current_user and current_user._id == user._id:
AttributeError: 'NoneType' object has no attribute '_id'

@whit537 maybe try running inv rebuild_search

Still not out of the woods on emails getting spit out properly. This system is feeling pretty brittle to me right about now. :-(
Done for the day, more later ...

They pyrax import in that script is a year old.

(Sorry, wrong thread ...)

This is going into release/0.68.0, but it has a (relatively small) merge conflict:
CONFLICT (content): Merge conflict in website/project/views/node.py
<<<<<<< HEAD
        'archiving': node.archiving or getattr(node.root, 'archiving', False),
=======
        'is_embargoed': node.is_embargoed,
        'archiving': node.archiving,
>>>>>>> ee682f9bb28ffde4d11fc8d6677e07cc23aa0476

Behavior for single-admin embargoes looks right, but multi-admin behavior is broken -- emails are never sent out, API throws a 500 when building emails:
Traceback (most recent call last):
  [DRF trace omitted]
  File "/Users/matt/Devel/osf.io/api/base/serializers.py", line 871, in update
    ret['data'].append(self.child.update(resource, data))
  File "/Users/matt/Devel/osf.io/api/nodes/serializers.py", line 258, in update
    node.update(validated_data, auth=auth)
  File "/Users/matt/Devel/osf.io/website/project/model.py", line 1429, in update
    save=False
  File "/Users/matt/Devel/osf.io/website/project/model.py", line 3084, in set_privacy
    self.request_embargo_termination(auth=auth)
  File "/Users/matt/Devel/osf.io/website/project/model.py", line 3447, in request_embargo_termination
    approval.ask(admins)
  File "/Users/matt/Devel/osf.io/website/project/sanctions.py", line 249, in ask
    self._notify_authorizer(contrib, node)
  File "/Users/matt/Devel/osf.io/website/project/sanctions.py", line 317, in _notify_authorizer
    context = self._email_template_context(authorizer, node, is_authorizer=True)
  File "/Users/matt/Devel/osf.io/website/project/sanctions.py", line 953, in _email_template_context
    context = super(EmbargoTerminationApproval, self)._email_template_context(user, node, is_authorizer, urls)
TypeError: _email_template_context() takes at most 4 arguments (5 given)
[20/Apr/2016 16:24:17] "PATCH /v2/nodes/ HTTP/1.1" 500 59

Following either link in the multi-admin case also doesn't work:

Pass complete. Aside from the above bugs, looks good. Please address those issues and add a few tests for a multi-admin embargo cancellation case. Should be RTM soon :fire:

Emails are now working, but when I follow the approval link as the second admin, I get... mixed signals:

Why do two emails go out when there are two admins, but zero are sent when there's one admin?

Closing, will reopen against 0.68

Ah, gee, I just realized I've been commenting on the wrong PR. Disregard the comment re: the unrelated code.


Make output node information and format consistent across ockam node create | show | list commands

Current behavior:
» ockam node create
Node: 1c399708, Status: [✓], Worker count: 17, Pid: 77790, Transport count: 1

» ockam node show 1c399708
Node: 1c399708, Status: [✓], Worker count: 17, Pid: 77790, Transport count: 1

» ockam node list
Checking state for node '1c399708'
Checking state for node '207f75d1'
+-----------+----------+------------------+-----------------------------------------------------------------------------------------+
| Node name | API port | Running          | Log path                                                                                |
+-----------+----------+------------------+-----------------------------------------------------------------------------------------+
| 1c399708  |    62526 | Yes (pid: 77790) | /Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-1c399708/1c399708.log |
+-----------+----------+------------------+-----------------------------------------------------------------------------------------+
| 207f75d1  |     4001 | Yes (pid: 80602) | /Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-207f75d1/207f75d1.log |
+-----------+----------+------------------+-----------------------------------------------------------------------------------------+

Desired Plain Output:
» ockam node create

Node Created!

Node:
  Name: 1c399708
  Status: Running
  API Address: <IP_ADDRESS>:62526
  Pid: 77790
  Worker count: 17
  Transport count: 1
  Log Path: "/Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-1c399708/1c399708.log"

» ockam node show 1c399708

Node:
  Name: 1c399708
  Status: Running
  API Address: <IP_ADDRESS>:62526
  Pid: 77790
  Worker count: 17
  Transport count: 1
  Log Path: "/Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-1c399708/1c399708.log"

» ockam node list

Node:
  Name: 1c399708
  Status: Running
  API Address: <IP_ADDRESS>:62526
  Pid: 77790
  Worker count: 17
  Transport count: 1
  Log Path: "/Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-1c399708/1c399708.log"

Node:
  Name: 207f75d1
  Status: Running
  API Address: <IP_ADDRESS>:4001
  Pid: 80602
  Worker count: 17
  Transport count: 1
  Log Path: "/Users/mrinal/Library/Application Support/io.ockam.ockam-cli/node-207f75d1/207f75d1.log"

Table rendering is brittle on many terminal setups. If we support it, maybe we won't make it the default, or maybe we'll have the above vertical output as a fallback. Something we can consider later.
The code for the node commands is here
https://github.com/build-trust/ockam/tree/develop/implementations/rust/ockam/ockam_command/src/node

We love helping new contributors!
If you have questions or need help as you work on your first Ockam contribution, please leave a comment on this discussion.
If you're looking for other issues to contribute to, checkout issues labeled -  https://github.com/build-trust/ockam/labels/good first issue or https://github.com/build-trust/ockam/labels/help wanted

If you do not mind, I will have a crack at this issue. Looks simple enough to finish over the weekend :)

@kpagacz awesome thank you for picking it up!

Closed in https://github.com/build-trust/ockam/pull/3090 by @leonzchang. Thank you.
@kpagacz if you would like to pick a different issue, I would love to help you find one. https://github.com/build-trust/ockam/labels/good first issue. Please let me know.


Don't generate PDB for libiconv on Windows

Fixes https://github.com/crystal-lang/crystal/pull/11480#issuecomment-980469203. At the moment, static libraries for Windows builds are all release builds and do not have any debug information, so we make sure to strip it from libiconv too; more specifically, we cannot pass the /Zi flag to cl.exe. Later we should revisit the issue of whether to include PDB files for those libraries as they might affect call stacks on Windows.
This PR switches to kiyolee/libiconv-win-build instead, since its MSVC build files are more organized than the one before (which is also incorrect in certain places).

Ummmm isn't the license in that project  straight up GPLv3-only?
https://github.com/kiyolee/libiconv-win-build/blob/master/src/iconv.c

That's GNU iconv, the command-line frontend to GNU libiconv, and we don't use it anywhere, before or after this PR.

OK but all other files are also GPL only.
https://github.com/kiyolee/libiconv-win-build/blob/master/lib/cp858.h

That linked file clearly refers to LGPL???

OMG I was so confused.
"Library General Public License" huh.
I was looking for "Lesser" everywhere.
Sorry

There are still warnings
https://github.com/crystal-lang/crystal/runs/4340598526?check_suite_focus=true#step:37:34
Run .\crystal-cross.exe build src/compiler/crystal.cr -Di_know_what_im_doing -Dwithout_playground --link-flags=/STACK:10000000
iconv.lib(iconv.obj) : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance
Generating code
Finished generating code
iconv.lib(iconv.obj) : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance

Did some further digging and the actual GNU libiconv uses libcharset/include/localcharset.h instead of srclib/localcharset.h; they have identical code, only their licenses are different. So this is a potential licensing issue in that newer MSVC repository.

@HertzDevil It's unfortunate... this kind of thing is exactly something that can make this PR linger for months :D
What solution do we have for now?
Is it possible to stick with the previous implementation?
Otherwise, I'd like to see a solution sooner rather than later, even if it has to be passing -Dwithout_iconv in CI for now.

Is the /GL warning still there?
A few options if we want to get around the license issues:

We use the old repository, but because the DebugInformationFormat and WholeProgramOptimization elements are absent we must override it from outside or find a way to inject it into the .vcxproj correctly.
We fork the new repository and fix the include directories and/or header licenses ourselves.
We invoke cl.exe directly. But note that we must still rely on either of the repositories because this skips the Linux-only configuration step.


Adding test for checking for V2 vacancy in nationwide apprenticeship search results

@dev4valtech if it fails because vacancies aren't there then it is okay because that will tell us that there is something wrong with the index and the first of the three tests will fail. If it fails because V2 vacancies aren't there then yes there is a dependency on us making sure there is at least 1 live v2 vacancy on each non-prod environment but this should be very rare occurrence. On live this could happen since we only have 1 current live V2 vacancy which closes on 17/9/2018 hwoever we aren;t running these tests on live.


Only create the required sending socket

The channel always creates both an icmp and udp sending socket, regardless of the chosen mode.  Whilst this is not known to be causing an issue it is wasteful.

To clarify my understanding, the socket gets created but doesn't get used depending on the mode?

@c-git correct; channel.rs needlessly creates both icmp and udp (raw) send sockets, when they are only used in icmp and udp protocol modes respectively.


server: split monitoring and readiness mux

Currently debug/monitoring and readiness share the same mux. As a result of that the debug and monitoring handlers are also available on 8080 port. We have a requirement to not expose these debug endpoints outside of pod - So we configured monitoringAddr accodinlgy. But we can not configure httpAddr like that because it exposes readiness probe. This PR splits the mux such that debug/monitoring has one mux used by monitoring server and httpServer has a separate readinessMux.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[ ] Does not have any changes that may affect Istio users.

@ramaraochavali: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
318bf8a1150591bc2f0556989c618551b0abc8fb
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio

@ramaraochavali: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
318bf8a1150591bc2f0556989c618551b0abc8fb
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-pilot-k8s-tests_istio

@ramaraochavali: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
318bf8a1150591bc2f0556989c618551b0abc8fb
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Don't you want to expose monitoring?

Don't you want to expose monitoring?

We do not use prometheus directly. We use a local scraper sidecar that runs in the same pod, scrapes that endpoint, converts monitoring stats and pushes it our stats backend in the format needed.

Yeah, there is a TODO to remove one server.

Yes.  /version is on 15014 https://github.com/istio/istio/blob/master/pilot/pkg/bootstrap/monitoring.go#L48

Kiali Operator will attempt to auto-detect the port (it looks for the new istiod endpoint, otherwise uses the old pilot endpoint):
https://github.com/kiali/kiali-operator/blob/v1.22.0/deploy/kiali/kiali_cr.yaml#L498-L504
https://github.com/kiali/kiali-operator/blob/v1.22.0/roles/default/kiali-deploy/tasks/main.yml#L236
The kiali server helm chart will install a Kiali Server that assumes the default of istiod port 15014: https://github.com/kiali/kiali/blob/master/config/config.go#L388
So if you use the operator, or if you install the server via the server helm chart, you are good.
Note that if all these auto-detections or defaults fail, you can always set the external_services.istio.url_service_version to the version URL endpoint.

Also based on your use case description, you only need to disable debug and monitoring - no need to move them to a separate port ? We still want to have debug on 8080 with ability to control the bind address ( <IP_ADDRESS>) and have monitoring/readiness on <IP_ADDRESS> ports.

I have thought about a release notes but we have not documented that 8080 is the debug port. Our documentation refers to 15014 as the debug port https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/. That is why I have not added a release notes.

I think this is the PR that caused debug to move.

No. that is not correct. The debug was exposed on both 8080 and 15014 -  This PR just stopped exposing on 8080 - I have not moved the debug port.
Do you prefer to retain the debug port on both 8080  and 15014?
I certainly do not want to create BC issue - But if you want to conditionally add via flag to both - I am fine with that. I can add that env var  default to  true  i.e. add to both in 1.8 and we can remove it in 1.9.  Please let me know what you prefer.

My preference is to keep the debug on 8080 for 1.8, deprecated - because scripts and tools may still use it. That was the original port for debug - 15014 was added later for metrics.
Technically this didn't cause debug to move - but to stop working the way it did in the past.
It would be great to add env variable to turn off metrics - if you have this use case.
Also add an env variable for /debug - to bind it to localhost or disable it. Unfortunately to bind it to localhost we would need readiness to move, and I think having readiness and monitoring on the same port would be better than monitoring and debug, because monitoring clearly can't be localhost only.
My new PR is doing the opposite of your - allowing everything (including gRPC) to use a single port. That's for a different use
case, and even if they share a port it should still be possible to selectively disable different features.
Re. release notes - all my debug scripts use 8080, it looks John also was used with it. We should document if debug moves, and we should also document we are planning to bind it to localhost and gradually replace it with the new 'XDS' based debug, which will require authentication. No doubt that unauthenticated and exposed /debug must go away - just need to do it carefully.
BTW - does anyone know if Prometheus requires the monitoring port to be declared as part of the Service ? I thought they look at container port directly ( so they can scan client-only workloads ).

Re. release notes - we clearly need to announce the deprecation and how to turn off or do localhost-only debug, and probably update some of the docs.

https://github.com/istio/istio/pull/26536 adds env var for 8080 and release notes.
I agree monitoring and readiness should be together because they can not be localhost only.  But as you said that needs readiness port to change. Let us do it little later.


[PartEditor] Revise update without backend

This will revise update from PartEditor to PartGraphSelector to remove
backend as it is not used anymore.
ONE-vscode-DCO-1.0-Signed-off-by: SaeHie Park<EMAIL_ADDRESS>

for #822 from draft #851


Add account related txs and events

I compiled+tested this prior to rebasing onto the openapi generated type changes. It will need to be rebased again once the codegen follow-up changes are merged.
It might be useful to go commit by commit, but only for the first 4 commits.
Open questions:

The openapi spec calls consensus events ConsensusEvents but consensus transactions Transaction, which seems inconsistent. I'm in favor of renaming to Events, but we could also rename Transaction->ConsensusTransaction
Since we support filtering events by type, we need to have some mapping of all the valid event types. This used to be adjacent to the Event type the api types, but now that event types are autogenerated the mapping is separate from the ground truth. This could(?) drift, but it seems like this would be a rare occurence.
I'd like to go ahead and rename all block fields -> height. We discussed it at the last product design meeting of December and the follow up was to do some user research, but the general sentiment seemed to be that height >> block. I'll do this in a separate PR, but for now the new type introduced uses Event.Height.

Regarding the open questions from the PR description:

The openapi spec calls consensus events ConsensusEvents but consensus transactions Transaction, which seems inconsistent. I'm in favor of renaming to Events, but we could also rename Transaction->ConsensusTransaction

I've been wanting to do the Transaction->ConsensusTransaction rename myself already :). I vote for that one, since in some senses, consensus is analogous to yet another (special) runtime. In general, I prefer saying ConsensusFoo over Foo, because I don't think it's obvious that Foo relates to consensus. This applies less to consensus specifics like governance voting, but definitely applies to generic things like events.

Since we support filtering events by type, we need to have some mapping of all the valid event types. This used to be adjacent to the Event type the api types, but now that event types are autogenerated the mapping is separate from the ground truth. This could(?) drift, but it seems like this would be a rare occurence.

Yeah I think we're OK. We could add a comment? In #281, I argue we don't even need the full list, because checking for a valid enum input is maybe more work than it's worth. My opinion would be different if Go had better supprot for enums, or of oapi-codegen supproted them better. But really I don't care very much either way; I'm fine with a different solution to the one I'm proposing.

I'd like to go ahead and rename all block fields -> height. We discussed it at the last product design meeting of December and the follow up was to do some user research, but the general sentiment seemed to be that height >> block. I'll do this in a separate PR, but for now the new type introduced uses Event.Height.

My intuition is to go the other way; in fact, I think I recently renamed height->block somehere, maybe in txs :). A block has a height, but a tx appears in a block (to which we happen to refer with a height). It's like how an event refers to a tx (well kinda ... tx_index really, because FKs are complicated there), not just to a hash or index., which are properties of the tx. We could rename to block_height? It's a little verbose. Anyway, let's talk during the next group meeting.

Open questions:

The openapi spec calls consensus events ConsensusEvents but consensus transactions Transaction, which seems inconsistent. I'm in favor of renaming to Events, but we could also rename Transaction->ConsensusTransaction

Agree with @mitjat on Transaction->ConsensusTransaction

Since we support filtering events by type, we need to have some mapping of all the valid event types. This used to be adjacent to the Event type the api types, but now that event types are autogenerated the mapping is separate from the ground truth. This could(?) drift, but it seems like this would be a rare occurence.

I'm also okay with a comment.

I'd like to go ahead and rename all block fields -> height. We discussed it at the last product design meeting of December and the follow up was to do some user research, but the general sentiment seemed to be that height >> block. I'll do this in a separate PR, but for now the new type introduced uses Event.Height.

I can go either way. Probably also defer to block.


[Feature Request] Support for '()' and '[]' in addition to '$$' and '$$$$' in LaTeX rendering

The application currently supports rendering LaTeX math expressions enclosed within the '$...$' and '$$...$$' delimiters. However, a widely adopted and recommended practice in LaTeX is to use the '(...)' delimiters for inline math and '[...]' for display math. The GPT-models themself seem to have a preference for the latter.
Requested Feature:
For text containing the '(...)' or '[...]' delimiters, the application should recognize and render the enclosed mathematical expressions using the LaTeX format, just as it does for the '$...$' and '$$...$$' delimiters.

This is rather a lack of feature, than a feature request

You can add this request to your system message, like this

Replace () or [] with $$ in your response when using Latex.


RESTful Client API

RESTful Client API. Issue https://github.com/lintool/Anserini/issues/24
Besides, TweetStreamIndexer can now be registered with its own IndexWriter, Using TweetSearcher's static field indexWriter(not instantiated) can cause problem when client runs alone.

Closing out of date PR.


Remove -i since docker run is not interactive

When using the -i argument, circleci.sh fails when called from a script:
Begin circleci config validation
the input device is not a TTY

This is making my pre-commit hook fail. Also it appears that there is no interactive mode for the tool, so it is unnecessary.

@felicianotech are you owning this repo? Could we get this reviewed?
Related discussion is here: https://discuss.circleci.com/t/reprise-allow-for-using-circle-ci-tooling-without-a-tty/23309/2

This change will only affect the Snap and Homebrew releases of the CircleCI Local CLI.

@felicianotech is the main repo somewhere else? Sorry I thought this was it.


fix: message input select media

Fix select-media event on DtRecipeMessageInput
Obligatory GIF (super important!)

:hammer_and_wrench: Type Of Change
These types will increment the version number on release:

[x] Fix

:book: Jira Ticket
No Jira
:book: Description
Changed the value emitted by select-media event to emit the files object instead of just the file names.
:bulb: Context
@bianca-artola-dialpad reported that message input component was only emitting file name and they need more information, emitting the whole object should fix their issues.
:pencil: Checklist
For all PRs:

[x] I have ensured no private Dialpad links or info are in the code or pull request description (Dialtone is a public repo!).
[x] I have reviewed my changes.
[x] I have considered the performance impact of my change.

For all Vue changes:

[x] I have made my changes in Vue 2 and Vue 3. Note: you may sync your changes from Vue 2 to Vue 3 (or vice versa) using the ./scripts/dialtone-vue-sync.sh script.

:link: Sources
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file

can we do the same when dropping images?  we already have add-media but is just returning the path


Change default view settings, while changes specific folders

Being able to set default view settings would be great - while still being able to change specific folders' view settings as well. It doesn't seem like this currently possible, as any changes made are universal...?

This is fixed in the preview version https://files.community/download/preview


New project creation cannot handle tick mark in VS file path

Based on external customer feedback reported here
Repro:

Create new VB proj in VS 2017 Community.
Rename "Microsoft Visual Studio 15" folder to "Microsoft  Visual Studio '15".
Should fail with error "An unexpected token was found at character position 37 in condition "Exists('D:\Microsoft Visual Studio '15\2017\Community\MSBuild\$(MSBuildToolsVersion)\Microsoft.Common.props')".

Team triage: closing due to age and lack of response.
Personal triage: this makes me very sad :(


Read Comics Book dead

Source name
Read Comics Book
Source language
English
Source link
https://readcomicsplus.net
Other details
Existing comics give a null error and WebView redirects to ComicExtra, which already has an extension.
Acknowledgements

[X] I have updated all installed extensions.
[X] I have opened WebView and checked that the source website is down.
[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a meaningful title with the source name.
[X] I will fill out all of the requested information in this form.

Add a :+1: reaction to issues you find important.

It's not down just redirect to https://comixextra.com/

My bad, didn't check


Manage all your cron jobs in one file

I guess run npm run format to fix database.json format then try again

Thanks for the patch, but it must be green to be considered. Due to the high volume, I'm not reviewing red PRs, just closing to clear out the queue. I apologize if I haven't addressed something. Please open a new PR.


Feedback for "What is SAP Business Application Studio?"

https://help.sap.com/products/SAP Business Application Studio/9d1db9835307451daa8c930fbd9ab264/8f46c6e6f86641cc900871c903761fd4.html?locale=en-US&version=Cloud
Test

Hi,
I hope the test worked :)
I'm closing the issue.
Reopen if there is still an AI.


ImageWidget needs to take graphic names as a kwarg

for the main image graphic that they add

perhaps we can just allow a dict of image_kwargs like {0: {kwargs}, 1: {kwargs}} .. where the numerical key corresponds to the data argument index

after #245 we now just call them "image_widget_managed", makes it easier to manage.


Flyt outside Equinor

User story
As another company we want to deploy Flyt easily so that we can map our processes
Value for Equinor
Vendors to map and improve their processes for better deliveries to us.
Commits to Flyt

The first step is to make the API public and easy to run.


Is the p-value of the Kruskal-Wallis test two-sided?

I have 2 (or more) non-normal groups of varying sizes, and so I looked at the Kruskal-Wallis test (when len(groups)>2) and the Mann-Whitney rank test (when len(groups)==2).
I know from the Mann-Whitney rank test docs [1] the the p-value returned is single-sided by default. However, I am unable to find documentation whether the returned p-value in Kruskal-Wallis test is two-sided or not. See docs here [2] and source code [3].
When comparing both tests head to head for len(group)==2, my experiments show that the Kruskal-Wallis p-value may be two-sided, one example here:
from scipy import stats
x = np.arange(1,41,1)
y = np.arange(2,42,2)
print ( stats.kruskal(x, y) )
print ( stats.mannwhitneyu(x, y) )

KruskalResult(statistic=0.02460383653043551, pvalue=0.8753582575468357)
MannwhitneyuResult(statistic=390.0, pvalue=0.4407715705950445)

While I understand they are different tests, the relationship between the p-values (Mann-Whitney p-value * 2  is almost equal to Kruskal-Wallis p-value) is very close and difficult to ignore. Any help would be much appreciated!
[1] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html
[2] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html
[3] https://github.com/scipy/scipy/blob/v1.4.1/scipy/stats/stats.py#L6484-L6601

see docstring description for alternative=None in mannwhitneyu
half the two-sided p-value is a historical artifact,
All other hypothesis test have correct two-sided p-values

Thanks @josef-pkt - I can assume that Kruskal-Wallis returns two-sided p-value then.

The distribution of the test statistic of the Kruskal test is approximatred by a chi2-distribution under the null hypothesis and the p-value is computed as chi2.sf(h, df), e.g. it only considers deviations to one side. I would call this a one-sided p-value.

@ahmedhosny, I think @josef-pkt's answer was about the note in the mannwhitneyu documentation:

alternative: None: computes p-value half the size of the ‘two-sided’ p-value and a different U statistic. The default behavior is not the same as using ‘less’ or ‘greater’; it only exists for backward compatibility and is deprecated.

But you also asked "However, I am unable to find documentation whether the returned p-value in Kruskal-Wallis test is two-sided or not."
As @chrisb83 mentioned, it is one-sided: the p-value is the survival function of the chi-square distribution evaluated at H.
Is gh-12565 a helpful addition to the documentation?


Makefile doesn't do much

atm the Makefile only builds our scanner.
Someone needs to get familiar with the corebuild API and modify the Makefile to also do stuff like ocamlyacc parser.mly and compile + link all that stuff with the ast.mli

@cerrno can we get this working asap? as soon as i finish on frontend, andy is gonna be blocked on this

So while i was working on the AST->parser stuff, I added a couple of things to make it do something. Not really sure how or what it's doing though, but at least it's doing something that's along the lines of trying to compile it.

For now I have a fully working pipeline with the dependencies set. We might wanna switch to corebuild. See commit.

We can switch to core build but I'm not sure what the large advantage would be. This project won't have many files, and most will just be additional objects to build with ocamlc

A main advantage is that corebuild organizes the binaries you have. After making, your directory doesn't look like a mess.

I'll take a look in the near future then. we can also considering moving to flat src directory in that case

Closing for now. Will improve as required.


runtime org.freedesktop.Platform branch 22.08 is end-of-life

Info: runtime org.freedesktop.Platform branch 22.08 is end-of-life, with reason:
   org.freedesktop.Platform 22.08 is no longer receiving fixes and security updates. Please update to a supported runtime version.
Info: applications using this runtime:
   org.kosmospredanie.Yakblt

Info: runtime org.freedesktop.Platform.VAAPI.Intel branch 22.08 is end-of-life, with reason:
   org.freedesktop.Platform 22.08 is no longer receiving fixes and security updates. Please update to a supported runtime version.
Info: applications using this runtime:
   org.kosmospredanie.Yakblt

Info: runtime org.freedesktop.Platform.GL.default branch 22.08-extra is end-of-life, with reason:
   org.freedesktop.Platform 22.08 is no longer receiving fixes and security updates. Please update to a supported runtime version.
Info: applications using this runtime:
   org.kosmospredanie.Yakblt

Info: runtime org.freedesktop.Platform.GL.default branch 22.08 is end-of-life, with reason:
   org.freedesktop.Platform 22.08 is no longer receiving fixes and security updates. Please update to a supported runtime version.
Info: applications using this runtime:
   org.kosmospredanie.Yakblt

Wrong repo, sorry
-> https://github.com/flathub/org.kosmospredanie.Yakblt


Eager only loads related many to many relation for first result

Hi, i'm having an issue where i'm trying to load a many to many relation using an eager query. But its only loading them for the first result.
This is the query which i'm trying to execute
module.exports = function (filter = {}) {

    return Task
        .query()
        .where(filter)
        .eager('assignees(select)', {select: builder => builder.select(['id', 'firstName', 'lastName'])});

};

Relation Mappings for the Task model
relationMappings: {
        client: {
            relation: Model.BelongsToOneRelation,
            modelClass: path.resolve(__dirname,'../models/client'),
            join: {
                from: 'task.clientId',
                to: 'client.id'
            }
        },
        project: {
            relation: Model.BelongsToOneRelation,
            modelClass: path.resolve(__dirname,'../models/project'),
            join: {
                from: 'task.projectId',
                to: 'project.id'
            }
        },
        team: {
            relation: Model.BelongsToOneRelation,
            modelClass: path.resolve(__dirname,'../models/team'),
            join: {
                from: 'task.teamId',
                to: 'team.id'
            }
        },
        assignees: {
            relation: Model.ManyToManyRelation,
            modelClass: path.resolve(__dirname,'../models/user'),
            join: {
                from: 'task.id',
                through: {
                    from: 'task_user.taskId',
                    to: 'task_user.userId',
                    modelClass: path.resolve(__dirname,'../models/task_user')
                },
                to: 'user.id'
            }
        },
        createdBy: {
            relation: Model.BelongsToOneRelation,
            modelClass: path.resolve(__dirname,'../models/user'),
            join: {
                from: 'task.userId',
                to: 'user.id'
            }
        }
    }

The result of the above query is this:
[
  {
    "id": 3,
    "title": "est est qui",
    "description": "minima illum nemo deleniti praesentium",
    "duration": 213,
    "link": "",
    "completedAt": null,
    "isComplete": false,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "ccd2de1a-a9e3-4bf0-be27-a42d0d40a9f3",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": [
      {
        "id": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
        "firstName": "akram",
        "lastName": "ali"
      },
      {
        "id": "22697a01-11c4-47fd-8d40-2da167befd45",
        "firstName": "Clifton",
        "lastName": "Cremin"
      },
      {
        "id": "a52640d3-08cd-41bf-8fd8-4d419b5a3a4f",
        "firstName": "Mallie",
        "lastName": "Koelpin"
      },
      {
        "id": "9b49f378-75e0-43fa-bb69-eeaa9e61304d",
        "firstName": "Matilde",
        "lastName": "Brown"
      },
      {
        "id": "49313dbf-dcbd-4725-a68e-c502e6aff0b9",
        "firstName": "Florine",
        "lastName": "Padberg"
      },
      {
        "id": "ccd2de1a-a9e3-4bf0-be27-a42d0d40a9f3",
        "firstName": "Desiree",
        "lastName": "Tromp"
      }
    ]
  },
  {
    "id": 11,
    "title": "accusamus ut modi",
    "description": "dolorem aliquid quos impedit asperiores",
    "duration": 147,
    "link": "",
    "completedAt": null,
    "isComplete": false,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "ccd2de1a-a9e3-4bf0-be27-a42d0d40a9f3",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 12,
    "title": "praesentium quia aliquid",
    "description": "aut eveniet qui dolor iusto",
    "duration": 31,
    "link": "",
    "completedAt": null,
    "isComplete": false,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "9b49f378-75e0-43fa-bb69-eeaa9e61304d",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 14,
    "title": "omnis iste harum",
    "description": "eaque natus eaque et culpa",
    "duration": 43,
    "link": "",
    "completedAt": null,
    "isComplete": false,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 17,
    "title": "voluptas fuga natus",
    "description": "voluptatibus magnam tempore ducimus neque",
    "duration": 158,
    "link": "",
    "completedAt": null,
    "isComplete": false,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "a52640d3-08cd-41bf-8fd8-4d419b5a3a4f",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 30,
    "title": "atque consequatur quia",
    "description": "ut deserunt non sequi voluptatum",
    "duration": 108,
    "link": "",
    "completedAt": "2017-05-27T18:45:32.388Z",
    "isComplete": true,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "49313dbf-dcbd-4725-a68e-c502e6aff0b9",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 31,
    "title": "pariatur mollitia tenetur",
    "description": "enim beatae veritatis reprehenderit debitis",
    "duration": 222,
    "link": "",
    "completedAt": "2017-05-27T18:45:32.388Z",
    "isComplete": true,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "49313dbf-dcbd-4725-a68e-c502e6aff0b9",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 44,
    "title": "nam autem nulla",
    "description": "eligendi veniam unde omnis nisi",
    "duration": 125,
    "link": "",
    "completedAt": "2017-05-27T18:45:32.388Z",
    "isComplete": true,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "49313dbf-dcbd-4725-a68e-c502e6aff0b9",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 45,
    "title": "est quia et",
    "description": "non enim recusandae voluptas quod",
    "duration": 137,
    "link": "",
    "completedAt": "2017-05-27T18:45:32.388Z",
    "isComplete": true,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  },
  {
    "id": 46,
    "title": "et debitis non",
    "description": "impedit recusandae modi minima asperiores",
    "duration": 132,
    "link": "",
    "completedAt": "2017-05-27T18:45:32.388Z",
    "isComplete": true,
    "teamId": 1,
    "clientId": 2,
    "projectId": null,
    "userId": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
    "createdAt": "2017-05-27T18:45:32.397Z",
    "updatedAt": "2017-05-27T18:45:32.397Z",
    "assignees": []
  }
]

As you can see, only the first result has its relations loaded, in the database every Task has at least 1 many to many relation with a user.
This can be shown by fetching one of the above models, that appear to have no related models, using a first() operator.
module.exports = function(filter = {}) {

    return Task
        .query()
        .where(filter)
        .eager('assignees(select)', {select: builder => builder.select(['id', 'firstName', 'lastName'])})
        .first();

};

This is the result of fetching task with the id of 46, which above shows has an empty assignees array
{
  "id": 46,
  "title": "et debitis non",
  "description": "impedit recusandae modi minima asperiores",
  "duration": 132,
  "link": "",
  "completedAt": "2017-05-27T18:45:32.388Z",
  "isComplete": true,
  "teamId": 1,
  "clientId": 2,
  "projectId": null,
  "userId": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
  "createdAt": "2017-05-27T18:45:32.397Z",
  "updatedAt": "2017-05-27T18:45:32.397Z",
  "assignees": [
    {
      "id": "fb4a6def-57b5-4da8-8ca2-c0639cb0aecd",
      "firstName": "akram",
      "lastName": "ali"
    },
    {
      "id": "22697a01-11c4-47fd-8d40-2da167befd45",
      "firstName": "Clifton",
      "lastName": "Cremin"
    },
    {
      "id": "a52640d3-08cd-41bf-8fd8-4d419b5a3a4f",
      "firstName": "Mallie",
      "lastName": "Koelpin"
    },
    {
      "id": "9b49f378-75e0-43fa-bb69-eeaa9e61304d",
      "firstName": "Matilde",
      "lastName": "Brown"
    },
    {
      "id": "49313dbf-dcbd-4725-a68e-c502e6aff0b9",
      "firstName": "Florine",
      "lastName": "Padberg"
    },
    {
      "id": "ccd2de1a-a9e3-4bf0-be27-a42d0d40a9f3",
      "firstName": "Desiree",
      "lastName": "Tromp"
    }
  ]
}

I'm not sure if its an issue on my end with the query, or possibly a bug. I'm using Postgres, with objection on 0.7.12. The algorithm for the eager operator, is the default algorithm. Thanks for your help

Sorry, I'm unabled to reproduce this. Could you create a minimal standalone app that I can easily run that reproduces the problem? I tried to create a similar query, and got all results (also there are multiple similar queries in the test set).

Hi, Thanks for the quick response, After a lot of testing i figured out It appeared to be something to do with the user model having a default join algorithm. Removing the default join algorithm would make it run fine. I created a small app as you requested and to my surprise it was working as intended. I then uninstalled objection, and installed the latest version and it's been working perfectly. So i'm guessing something must have changed recently that fixed the issue. Thanks for your help and the this great library. I will close this issue as everything appears to be working as intended.


Change the default number of letters in a message

By default, a message cannot be longer than 500 characters, but the instance where I have my account allows messages up to 2000.
Is there a way to change that figure in Andstatus? Right now Andstatus doesn't allow messages longer than 500

Hello @pygopero !
What's that instance? What is its "Social network type" and its version?

Hi!
Thanks for asking. The servers is masto.es, of course is a mastodon server. And version is 4.3.1
By the way, with the beta version of Andstatus,  I am able to sign in this server, with the previous version it was impossible.
Thanks again.

Thank you for the details,
For a long time Mastodon didn't have config option allowing a Client app to know the text limit, I see my notes in AndStatus about this https://github.com/andstatus/andstatus/blob/master/app/src/main/kotlin/org/andstatus/app/net/social/ConnectionMastodon.kt#L531
Looking at this now I see that Gargron added such config in https://github.com/mastodon/mastodon/pull/16485, so I will add this to AndStatus also.

@pygopero Please try v.62.02 at #456
Now AndStatus rereads Maximum number of characters from the server on every account verification. Go to Account -> "Reverify credentials" button.

I will test it right now!

It works!
With a little trick: The number of characters available in a reply is 2000 minus the length of the account I am answering to.
Is that correct?

I' m not sure about details, but think yes, I didn't correct that "2000" from your server.

I've checked and that is the usual, so everithing is ok.


Add feature to use PAT with GitHub

This extends rate-limiting limit for update/install of GitHub releases
for users who can authenticate with a PAT
Squashed replacement for #449

 on  Ubuntu, Debian and Mint

Great work @philclifford !


Interpreting DeepBench MMul results

Hi all,
I had a quick question on interpreting some of the results in the .xlsx results docs. Do the groupings for the dense matrix multiplication results have any significance? There seems to be a group of mmul configurations, followed by a space, then more configurations. I'm doing some benchmarking research, and it would be nice to know if these groups of configurations have any meaning.
Here's a link to one of the results sheets that shows this - https://github.com/baidu-research/DeepBench/blob/master/results/train/DeepBench_NV_V100.xlsx
Thanks in advance for the help!

The grouping of the configurations don't have any meaning.  The applications for each of the kernels are mentioned in the spreadsheet which may be helpful to you.


⚠ Cioos Pacific ERDDAP has degraded performance

In 2ea9da6, Cioos Pacific ERDDAP (https://data.cioospacific.ca/erddap/index.html) experienced degraded performance:

HTTP code: 200
Response time: 15775 ms

Resolved: Cioos Pacific ERDDAP performance has improved in b91c34e.


Upgrade analyzer dependency for compatibility with newer packages

Trying to use reflection factory with other packages causes conflict in getting dependencies (e.g. flutter_data: ^1.4.4)
flutter pub get
Because reflection_factory <1.0.24 depends on analyzer ^2.1.0 and reflection_factory >=1.0.24 <1.1.1 depends on analyzer ^3.3.1, reflection_factory <1.1.1 requires analyzer ^2.1.0 or ^3.3.1.
And because reflection_factory >=1.1.1 depends on analyzer ^3.4.1, every version of reflection_factory requires analyzer ^2.1.0 or >=3.3.1 <4.0.0.
Because no versions of flutter_data match >1.4.4 <2.0.0 and flutter_data 1.4.4 depends on analyzer ^4.1.0, flutter_data ^1.4.4 requires analyzer ^4.1.0.
Thus, flutter_data ^1.4.4 is incompatible with reflection_factory.
So, because scout_core depends on both flutter_data ^1.4.4 and reflection_factory any, version solving failed.
pub get failed (1; So, because scout_core depends on both flutter_data ^1.4.4 and reflection_factory any, version solving failed.)

Is upgrading the dependency possible?

I will try to update them in the next days. Thanks for the report 👍🏻

Resolved:
reflection_factory: ^1.2.0
https://pub.dev/packages/reflection_factory


Added Document functionality and applicable tests

What does this PR do?
Adds functionality to Documents Controller
Description of Task to be completed?

This PR adds extra functionality to the 'fetchone' method in the documents controller, to take into account user/Admin type

Coverage decreased (-0.7%) to 99.267% when pulling 3a0046eebccfe67b9e4be713262067bf11ef7b02 on feedback-implementation into 2b49790edefe443a2a631457759c9af2b8040d57 on develop.


Get node logic ex

Another PR to fix #107 plus some bonus deprecation fixes for the future.
Notes:

ideally I'd like to drop composer.lock as it is considered bad style to include that in libraries
I've fixed the supported versions of twig to 1.25 - 1.30.0 as master (1.30.1) does break
However, I've included a few minor BC changes to get around most deprecation warnings...

Might have got a bit carried away with the changes not related to the getNode() issue. If you prefer I can split this into individual ones...
Personally, I can think of at least 3:

getNode() fixed for #107
remove composer.lock
update composer.json with prefer-stable: true
another PR after those to follow interface changes on twig dev-master

Hmm yeah the composer.lock thing looks like a great thing to get rid of but you're right, belongs in its own PR. Same deal with composer.json I guess. Basically yeah, PRs that do one thing each are way way way easier to merge!

BTW I appreciate how patient you've been on this stuff. Trying my best to continue to devote some time to this lovely little repo ❤

Progress! I'll try to get the actual issue isolated into a new PR - ironically, I might not get to that until 2 weeks from now - its summer in New Zealand ;)

Hell yeah, get out and enjoy that summer!

Closed in favour of #112, #113, #114 (and counting...)


Add CommitFunctions and update config with defaults

Follow up of #735
Closes #688.
Made commit option similar to changelog option but with a default template when "true" is passed (keeping it backward compatible).
This currently doesn't allow disabling commit for one of add/version. Maybe that can be added too if the respective function returns null or doesn't exist?

All done from my side, the only thing remaining is the name of the skipCI options. See this comment

The PR is almost ready to be shipped, great work! Once all remaining items (mostly nits really) get addressed I will rethink the rollout strategy as we need to consider how this affects the version of the involved packages and how to ensure that this doesn't break any existing consumers.

I would love to see this PR merged as soon as possible. What's holding it back?
Really looking forward to being able to turn auto-commit back on without getting that docs(changeset) prefix (which makes no sense when the actual change is just one commit) in the git log.

What's holding it back?

Lack of the free time of maintainers 😉

@rohit-gohri thank you very much for your contributions and bearing with me. People like you keep OSS projects afloat 🚀

Thanks a lot guys, really awesome! Works as advertised for me.
However, it does look like config/schema.json was not updated to reflect the change?

However, it does look like config/schema.json was not updated to reflect the change?

Yeah, we forgot about it - PR with an update would be appreciated.


Upgrade OAuth proxy to 6.0.0

Hi, The oauth proxy image 5.x.x are out of date. There's changes to Azure OAuth that are now deprecated, but are updated in 6.0.0. Could you please upgrade the deployment to use the newest OAuth image?

Thanks @cccs-cat001 for reporting it -- it has been merged into master at #869, but release-1.6 branch was already on RC stage and a quick manual test showed e.g. grafana not auto logging-in from the provided http headers, so we decided to postpone it for next release -- we'll add it to (to be created) release-1.7 branch and properly handle it.
/cc @javsalgar


Added warning for list of embedded components

Q
A

Bug fix?
no

New feature?
no

Tickets

License
MIT

As discussed on slack with @weaverryan , I think the key trick for lists of embedded components is an easy footgun and thus should be pointed as soon as the docs introduce the subject. I'm open for better wording or placement.

This makes it just a LITTLE bit more visible - let's try it! Thanks Florent!

I didn't want to make it too obvious, but there sure is margin for a bigger warning


包的路径问题

用命令 “pip install galerkin-transformer”, 可以安装好这个包:

Installing collected packages: galerkin-transformer
Successfully installed galerkin-transformer-0.1.1

但是当我调用 “from galerkin_transformer.model import *” 语句时，会出现如下错误：

Traceback (most recent call last):
  File "xxx/lib/python3.8/site-packages/galerkin_transformer/model.py", line 2, in <module>
    from libs.layers import *
ModuleNotFoundError: No module named 'libs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_galerkin_transformer.py", line 7, in <module>
    from galerkin_transformer.model import *
  File "xxx/lib/python3.8/site-packages/galerkin_transformer/model.py", line 5, in <module>
    from layers import *
ModuleNotFoundError: No module named 'layers'

应该是包的路径管理有些问题，麻烦您debug一下（或者是我哪个地方搞错了？）

Thanks! Let me fix it tomorrow.

Sorry for the delay.
Please try
pip install galerkin-transformer==0.2.1

and then:
 from galerkin_transformer.model import *


[Dussion] What is the most mainly use case for DistKV

I am curious about the mainly use case for DistKV. Is it distributed cache ,distributed computing or others?  Because the different use cases needed different tradeoff between CAP and ACID.  It's means we need to use different protocol to solve data replicate.

Do you mean that when we are running in the cluster state, what protocol is used between storeservers?
We discussed before about using the raft protocol.

First at all, thanks for your proposal this issue, and it is actually a nice issue.
I think Distkv is used as a distributed k-v memory store, which has rich types as Redis at most time.
But the original intention of Distkv is make a distributed Redis, so the consistency and the highly available are guranteed almost. But on the other hand, we must tolerate some keys are unavailable when a shard failed and do the data migration. but in a shard, if a node failed, it will looks like nothing happen just restart a node to join the shard only if the most of the nodes are alive.
So actually it seems we should use 2PC between master and slaves.

Further more, We'll have 2 extension engines on the top of Distkv core:
(1) A lightweight relationship database:
Something like tidb or hbase, can support a lightweight sql.
(2) A cloud-structure engine
This feature is naturally came out because of the rich types. We call this feature as cloud-struct, and it looks like a distributed computing engine somehow. This engine will make users' code executed on the Distkv servers.


[Diagnostics] Removing FailureDiagnosis::visitCoerceExpr from CSDiag

Follow up #29011, fixing the remaining failures diagnostics and removing FailureDiagnosis::visitCoerceExpr from CSDiag.
cc @xedin @hborla

@swift-ci please smoke test

@swift-ci please smoke test

Thank you @xedin :))

@LucianoPAlmeida Thank you for taking care of it!

I think this may be causing the source compat suite to fail for a few projects (https://ci.swift.org/job/swift-PR-source-compat-suite-debug/2157), looks like anchor is nullptr in the check isa<CoerceExpr>(anchor) when we do witness matching (see for example this crash log).

@hamishknight I'm going to revert (https://github.com/apple/swift/pull/29084) but I'll also open a PR to address that. We definitely can get locators without an anchor but a valid path while comparing generics...

@xedin Sorry for the issue, let me know if I can help with that? :)

No problem, I got it covered!

No problem, I got it covered!

Thank you so much 💯

@LucianoPAlmeida Could you please try to reduce a simple test-case from project crashes so we could include it into validation test?

@xedin Sure 👍

@xedin Just added the test case #29095 :))


Please, release smtp.cr v0.2.0

Hi @raydf I'm having an issue building amber projects https://github.com/crystal-lang/shards/issues/183
I found this shard version don't follow common version pattern because is using v0.1 instead of v0.1.0 so you need to release v0.2.0 to be able to fix my issue, please 😅
Also don't release v0.1.1 because doesn't work, seems that shards install logic is v0.1.1 < v0.1 so v0.2.0 is needed to be able to fix this issue
Remember to update both, shard.yml and src/version.cr before create a tag 👍

Thanks @faustinoaq


helm: ensure that envoy daemonset is installed only when needed

As .Values.ingressController.enabled and .Values.gatewayAPI.enabled require .Values.l7Proxy to be set it's sufficient to just perform a check on that value.

Fixes https://github.com/cilium/cilium/issues/33430
No backport necessary as this only affects 1.16
helm: ensure that envoy daemonset is installed only when needed

/test


[Panic] ruff fix-import

File
The file is https://github.com/onnx/onnx/blob/7f0a9f2537348e7e53a825e0b716db1dbc997e47/onnx/backend/test/case/node/where.py
pyproject.toml
[tool.ruff]
target-version = "py37"
select = [
    "E", # pycodestyle
    "F", # Pyflakes
    "W", # pycodestyle
    "B", # flake8-bugbear
    "N", # pep8-naming
    "YTT", # flake8-2020
    "RUF", # Ruff-specific rules
    "TID252", # Relative imports
]
ignore = [
    "B9", # Opinionated bugbear rules
    "E402", # warns on imports coming after statements. There are important use cases
            # like demandimport (https://fburl.com/demandimport) that require statements
    "E501",
    "N803", # Argument casing
    "N806", # Variable name in function should be lowercase
]
ignore-init-module-imports = true
unfixable = [
    "F401", # Unused imports
]

[tool.ruff.flake8-tidy-imports]
# Disallow all relative imports.
ban-relative-imports = "all"

Command ran
RUST_BACKTRACE=full ruff --fix-only --exit-zero --config=pyproject.toml /Users/justinc/Documents/GitHub/onnx/onnx/backend/test/case/node/where.py

Error:
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', crates/ruff/src/rules/flake8_tidy_imports/relative_imports.rs:102:54
stack backtrace:
   0:        0x1054e8ed8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h2990a6f24ccff6db
   1:        0x104edb668 - core::fmt::write::h91497fd291c8b104
   2:        0x1054e3ea0 - std::io::stdio::_eprint::hfdc1bd7888eaa0e7
   3:        0x1054eca70 - std::panicking::default_hook::hdfe992d5fb29a991
   4:        0x1054ec798 - std::panicking::default_hook::hdfe992d5fb29a991
   5:        0x1052796d4 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
   6:        0x1054ed02c - std::panicking::rust_panic_with_hook::h2eb0e4b718773521
   7:        0x1054eceb8 - <std::panicking::begin_panic_handler::StrPanicPayload as core::panic::BoxMeUp>::get::h3f7cfefea3e19c93
   8:        0x1054eb8c4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h2990a6f24ccff6db
   9:        0x1054ecc74 - _rust_begin_unwind
  10:        0x1055b0eec - core::panicking::panic_fmt::ha46aa9af97eb193d
  11:        0x1055b0e0c - core::panicking::panic::h5713e7735cef5fa8
  12:        0x10531cce4 - <ruff::checkers::ast::Checker as ruff::ast::visitor::Visitor>::visit_stmt::h4b593c8c33a76cb3
  13:        0x105333440 - <ruff::checkers::ast::Checker as ruff::ast::visitor::Visitor>::visit_body::h0fc88a428b1d49d5
  14:        0x10534a4d8 - <ruff::checkers::ast::Checker as ruff::ast::visitor::Visitor>::visit_body::h0fc88a428b1d49d5
  15:        0x10535a298 - ruff::linter::check_path::h842469eb58db80b1
  16:        0x105364d20 - ruff::linter::lint_fix::h929bb0874a600095
  17:        0x105266a30 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  18:        0x10520086c - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  19:        0x105264aa4 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  20:        0x10527d370 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  21:        0x105274640 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  22:        0x10527eb30 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  23:        0x1051d47d8 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  24:        0x1051d4c80 - <reqwest::redirect::TooManyRedirects as core::fmt::Debug>::fmt::h082b790308395cc4
  25:        0x1054debc8 - std::rt::lang_start_internal::h94fa211da9d06d56
  26:        0x10528c58c - _main
fish: Job 1, 'RUST_BACKTRACE=full ruff --fix-…' terminated by signal SIGABRT (Abort)

Version
ruff 0.0.247

Taking a look...

Just confirmed locally that this was fixed by #2990. Will cut a new release today.

Thanks!

Thanks for filing!


Page does not explicitly say which Azure Key Vault permissions are required for an application to read an encrypted column

When creating an application which reads encrypted columns from Azure SQL, it's not entirely clear what minimum set of permissions are required in order for the application to work properly.
The CLI snippets (for Powershell and Azure CLI) under Create a key vault to store your keys have these lines respectively:
Set-AzKeyVaultAccessPolicy  -VaultName $vaultName  -ResourceGroupName $resourceGroupName -ServicePrincipalName $applicationId -PermissionsToKeys get,wrapKey,unwrapKey,sign,verify,list
az keyvault set-policy --name $vaultName --key-permissions get list sign unwrapKey verify wrapKey --resource-group $resourceGroupName --spn $applicationId
From this, I could infer that the application requires these Key Permissions:

Get
List
Sign
Verify
UnwrapKey
WrapKey

This took a long time to find, and would be easier to understand if it had a dedicated table of permissions. Otherwise, developers may just be tempted to grant all available permissions to their application, which is a security risk.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ddcad97f-02c6-4307-7ed8-85ba7ae23ecc
Version Independent ID: 51a5e20b-c221-ee05-cb44-a479bfcf1a00
Content: Configure Always Encrypted by using Azure Key Vault - Azure SQL Database
Content Source: articles/azure-sql/database/always-encrypted-azure-key-vault-configure.md
Service: sql-database
Sub-service: security
GitHub Login: @VanMSFT
Microsoft Alias: vanto

@LwsSt Thanks for the feedback !  I have assigned this issue to content author to investigate and update the document as appropriate.

Hi @VanMSFT
That text you've highlighted refers to creating a column key, and setting up encryption. And only refers to SQL Server Management Studio.
My issue was more in the context of an application (for example, a developer application written in ASP.NET Core), which permissions are required.
The text you've highlighted mentions Create as a required permission, and this is not required for a running application.

@LwsSt - Thanks for the clarification, Lewis. I can add another line just for application permissions, but your assumptions are correct. The permissions required is in the code itself.
@jaszymas @shohamMSFT - Can you guys confirm the minimum application permissions regarding AE with AKV?


Please Send Halp 💥

Describe the bug:
Actual real-world usage of Razor-based solution runs into scaling concerns and makes one doubt long-term scalability.  I've sent out a plea here:
https://developercommunity.visualstudio.com/t/Razor-Tooling:-Ginormous-Memory-Usage-Af/10146575#T-N10247503
But I am opening an issue here for greater visibility.
Version used:
Any within the past two years.  They all feature the same grief.
To reproduce:

Make a solution with ~150k lines of C# and over 35k lines of Razor:

Open solution (this is an older solution that does not have as much code but should still easily present the issues/challenges for you).
Do development
Observe grief

Expected behavior:
Several GB of RAM used, normal and effective GC to keep around such levels.
Actual behavior:
Nearly 8-12 GB ultimately used with frequent GC and subsequent grief.
Additional context:
As mentioned in the plea/link above I am running (or attempting to run 😁) a Microsoft for Startups-accepted startup based on your technology and am now actively generating revenue.  However, this has gotten to the point where I am beginning to doubt the long-term viability due to this and many other issues I am facing in Visual Studio.  The runtime tech works great but the tooling is massively challenging and has been for several years now.
I am heavily invested in Microsoft/Blazor/Razor so I can put up with this friction for a while longer, but I cannot see myself realistically convincing any other developer to endure the challenges that the environment poses, especially if they are from a React/Node/etc background.  I am certain they would tell me I am insane for putting up with this situation -- which, incidentally enough, is exactly how I feel ATM -- and would recommend switching to another tech stack altogether.
FWIW I have reached out to @NTaylorMullen and the suggestion was to reach out to @davidwengier (👋) which I suppose I could have done privately but I feel more comfortable doing this here with this issue. 😁
I am wanting to explore ways to improve my solution so that it better fits your current design so that I do not encounter constant existential battles of doubt from using your technology.  The obvious stab is to reduce the number of projects used.  I did try this at one point but it did not make any difference. 😭  I could have been doing something non-obvious, however, so this is why I am reaching out now to make sure I have all considerations factored.
Thank you for any consideration and/or assistance you can provide. 🙏

You probably don't want to hear this, but "we're working on it" :)
Unfortunately there isn't one thing we can do to fix the issues, but you should know that perf (and reliability) is our top focus, across tooling and compiler teams. I suspect most of the improvements will come from the compiler side of things, and the good news there is that we have some execellent plans (removing design-time/run-time file split, re-using source generator assets, removing the usage of the "dynamic file" system) and even better news is that the compiler is now part of this repo, and owned by the same team that owns the VB/C# compiler (and they seem to know what they're doing). Unfortunately all of this is going to take a long time, and whilst there will definitely be incremental improvements , its impossible to say exactly when your issues in particular will be "fixed".
To share some recent results though, @chsienki has recently been doning on the source generator to improve perf, that has some very exciting results:

Method
Job
BuildConfiguration
Mean
Error
StdDev
Median
P95
Ratio
RatioSD
Gen 0
Gen 1
Gen 2
Allocated

Razor_Add_Independent
Baseline
Release_Nuget
183.429 ms
3.0090 ms
7.8207 ms
181.514 ms
205.752 ms
1.00
0.00
14000.0000
1000.0000
-
62,397 KB

Razor_Add_Independent
Current
Release
4.847 ms
0.0671 ms
0.0594 ms
4.827 ms
4.955 ms
0.03
0.00
312.5000
281.2500
-
1,957 KB

Razor_Edit_Independent
Baseline
Release_Nuget
14.477 ms
0.1900 ms
0.1777 ms
14.412 ms
14.794 ms
1.00
0.00
843.7500
406.2500
-
5,087 KB

Razor_Edit_Independent
Current
Release
4.457 ms
0.0641 ms
0.0568 ms
4.448 ms
4.554 ms
0.31
0.00
328.1250
296.8750
-
1,980 KB

Razor_Remove_Independent
Baseline
Release_Nuget
176.081 ms
3.2093 ms
8.3981 ms
172.740 ms
201.633 ms
1.000
0.00
13000.0000
1000.0000
-
61,412 KB

Razor_Remove_Independent
Current
Release
1.245 ms
0.0152 ms
0.0119 ms
1.244 ms
1.263 ms
0.007
0.00
156.2500
82.0313
-
814 KB

Razor_Edit_DependentIgnorable
Baseline
Release_Nuget
2.747 ms
0.0519 ms
0.0460 ms
2.755 ms
2.796 ms
1.00
0.00
226.5625
156.2500
-
1,146 KB

Razor_Edit_DependentIgnorable
Current
Release
3.129 ms
0.0586 ms
0.1128 ms
3.095 ms
3.349 ms
1.15
0.03
210.9375
187.5000
-
1,292 KB

Razor_Edit_Dependent
Baseline
Release_Nuget
137.331 ms
4.3744 ms
12.8980 ms
130.231 ms
166.848 ms
1.00
0.00
9000.0000
500.0000
-
41,554 KB

Razor_Edit_Dependent
Current
Release
5.938 ms
0.0996 ms
0.0932 ms
5.904 ms
6.091 ms
0.04
0.00
343.7500
312.5000
-
2,148 KB

Razor_Remove_Dependent
Baseline
Release_Nuget
187.709 ms
8.4931 ms
24.9087 ms
172.975 ms
232.938 ms
1.000
0.00
14000.0000
1000.0000
-
61,193 KB

Razor_Remove_Dependent
Current
Release
1.219 ms
0.0144 ms
0.0120 ms
1.213 ms
1.242 ms
0.005
0.00
156.2500
82.0313
-
814 KB

At the moment, these are still being worked on, so aren't in the repo, and unfortunately only help with Hot Reload scenarios, but once we as I mentioned above we have plans and are working towards reusing the source generator assets so these improvements will come to tooling in time. I just don't know what time.
I just wanted to mention all of this so that you know we are working on it, and mostly so that you know we really appreciate the effort you've gone to in sharing your data, and reporting problems, and the patience that you have shown thus far.

You probably don't want to hear this, but "we're working on it" :)

You'd be surprised @davidwengier I very much do want to hear that. 😁  Thank you for the update and insight with your provided metrics.  It would of course be ideal to have a general window of time when this will be completed but it's understandable that these are not available at the moment.  Hoping+wishing for the best here and that you are able to provide some quality relief soon. 👍

Working on some more perf improvements in https://github.com/dotnet/razor/pull/8133, in particular one change has allocations in document mapping down from something that linearly increases with file size, to just a few kb regardless of file size. Some of those improvements only affect light bulbs, but the document mapping in particular will help a lot of things, including even just clicking around in an un-changed document.

Awesome, thank you @davidwengier any bit helps, so any updates like this are very valuable to me.
FWIW please also consider that in my world it's a war on two fronts.  In addition to Visual Studio, I am also running R# which has also been struggling in this area as well for several years now.  Tracking my woes on JetBrains' side here:
https://youtrack.jetbrains.com/issue/RSRP-484052/2021.1-RTM-Razor-Development-R-Doubles-CPU-Utilization
One may ask why I am such a glutton for punishment.  Believe me when I say I ask myself this constantly. 😅😭  To be honest, if the runtime didn't work so freaking well I would have serious thoughts here about my direction.  However, the finished product when deployed works very well to my liking, so that really only leaves the tooling as the biggest risk & consideration in my world.
The deeper issue for me is that I have had my nose in .NET since 2001, and this is by far and away the most successful endeavor I have set out on -- even if it's only a few hundred dollars ATM -- so I would like to protect the 20-plus-year investment, so to speak. :)

Hi, @davidwengier thank you again for looking into this issue.  A thought arose for me over the weekend and I wanted to ensure that I brought it up here.  Can you confirm if the code that I have provided is being used anywhere to validate the performance improvements being made on your side?
The fundamental concern here is that you are addressing one issue while my codebase is facing another.
Primarily, what concerns me is that opening my solution takes many, many minutes (5-10) of CPU churn by devenv.exe and ServiceHub.RoslynCodeAnalysisService.exe and will ultimately take many GBs of memory right out of the gate, around 7-8GB.
Additionally, when I close the solution in Visual Studio and force a GC, all the Razor objects are retained in memory as denoted in this issue.  This has been the case for Razor development since the outset so want to confirm if this is being addressed in your current efforts.
I may be misunderstanding the issue, of course, but my understanding is that such a large amount of memory utilization -- which only increases as development goes on -- is leading to a good amount of GC activity and a lot of the grief that I am experiencing whenever Razor development is encountered.
Thank you for any further update/insight that you can provide. 👍

The fundamental concern here is that you are addressing one issue while my codebase is facing another.

I completely understand the concern, and you are correct to think this. The problem is that your codebase is unlikely to be facing a single issue, and this is especially true when dealing with memory allocation issues as they are (obviously) cummulative. Because of that, there isn't going to be one fix that will magically solve all of the issues, but rather the solutions will be cumulative too. So for example, I made a change that will definitely stop memory bring being consumed on every keystroke by the light bulb feature of VS in Razor files. The investigation of that issue, and the validation of the fix, was done in isolation for the light bulb feature. I cannot say if that will stop memory being consumed in VS as a whole, for your solution, on your machine. I would love it if I could, but its simply impossible to know. All we can do is keep fixing things, and making things better.

Thank you for your reply and thoughtful consideration @davidwengier.  I appreciate the approach of taking things as they arrive.  However, from my perspective, I have been doing this for over two years now and it does not seem to be getting any better.  This is part of my challenge here.
It would seem that at the very least your team would have a codebase that is similar to mine (i.e., 100k+ lines of C#, 25k+ lines of Razor) that would surface these issues, which are very clear, present, and obvious.  In my opinion, these issues should not be making it into Preview/RTM bands but have been doing so for well over a year now.
Right now as it stands, I am basically a full-time QA engineer for your product(s) and this does feel very fair to me as someone trying to build a business on your technology.
Consider that as a result of this, I am battling two fronts:

The inordinate challenge of actually making money with a technology
The unnecessary challenge of tooling used to make the technology that makes the money in the first challenge

So you are aware, the experience is so bad for me that it is starting to impact my estimates now when I plan new features.  Development that should take about 1 day is being given 1.5 days now to account for all the friction that tooling incurs at the moment.  This is neither desired nor long-term viable and leads me toward the desire to start reaching for another technology stack altogether.  Again, I am so invested and therefore committed -- one might say attached 😁 -- at the moment that this is not a serious desire.    However, the longer this continues to persist and feels like a sinister IQ test for how much grief one will endure, the more realistic this desire becomes.
Thank you for any further consideration and discussion around this.

Again, airing out some grief here, but I will have mornings this one where I spend about 10 minutes developing with a new preview, and notice that my CPU is perpetually pegged at 30-35%.  I then need to spend over an hour tracking it down and getting it reported, making sure it's not something that I have already reported as there are already dozens of these it seems I have reported over the past several years:
https://developercommunity.visualstudio.com/t/Perpetual-35-CPU-Utilization/10267474
Again it would seem if someone over there simply took a solution with a good amount of code (100k lines of C#, 25k+ lines of Razor) and did some basic development for 30 minutes or so, they would easily find issues like this with a release candidate before it gets shipped it out to a band.
One final thought here is that when I start looking for investors.  A lot of startups do not use Microsoft tech as you know and I am sure to get questions about this.  I am having trouble seeing if they would actually approve of what I have to go through to make everything work here.  This is a concern emerging in my world.
This is also in addition to finding another developer that would actually be able to tolerate what I have to go through at the moment to get the solution to build+debug, and upon successfully building+debugging, having to deal with issues such as the above.  Even if I was able to manage to get someone to accept this, I am not sure they would last for long and retention+turnover would be a very real challenge. 😭

@Mike-E-angelo have you tried disabling all your third party VS plugins/extensions (if you have any) like ReSharper etc as I wonder if they might be causing the high CPU usage?

This is the first thing I do before reporting @adrianwright109 :)  As I mentioned I have several years of experience now fighting the two fronts and have gotten good at knowing where to report the grief.

Speaking of which, just encountered a new issue in ServiceHub.RoslynCodeAnalysisService.exe:
https://developercommunity.visualstudio.com/t/Perpetual-30-CPU-Utilization-in-Servic/10267489
Yes, seeing double this morning.  30-35% utilization in devenv.exe, followed by another in ServiceHub.RoslynCodeAnalysisService.exe 😭

Thanks for the comments folks, I appreciate the passion and definitely appretiate the frustration at things not working. Having said that, I'm not entirely sure that there is anything I can really say in response. Yes @Mike-E-angelo we have various projects of various sizes and complexities we use, and you've been kind enough to provide yours, and all of the tracing and diagnostics that you record are great. The issue here is not about finding issues - we agree that they exist - the issue is that the fixes are long and complicated. If someone, or something, has given you the impression that anyone is denying you're having issues, or that there are performance improvements that we need to make, then let me be very clear and categorically state that is not true.
I wish we could snap our fingers and fix everything in the next version, but the realities and complexities of the fixes cannot be understated. For example, the thing that causes the most memory issues (at the moment) is in the Razor compiler, with various things like how it uses syntax annotations, conditional weak tables, and how it mixes syntax and semantics etc. The fix here is to overhaul the compiler to bring it in line with the latest architecture from Roslyn, but not only is that a lot of work, it also would break compatibility with every public API the current compiler has, and break 1st and 3rd party consumers. Rolling out these sorts of breaking changes necessarily has to be an effort over a reasonable time, coordinated with lots of partner teams.
Hopefully each month with each preview release there will be more and more fixes and improvements, but we're definitely on a "slowly chip away at things" tragectory. The bad news is we're not finished, the good news is we're not pretending we are.

Thank you very much for your continued dialogue @davidwengier.  I promise that at some point I'll get out of your hair here. :)

If someone, or something, has given you the impression that anyone is denying you're having issues, or that there are performance improvements that we need to make

It's not that I feel ignored.  Quite the opposite actually and it's quite nice on that front. :)  My primary beef is that I will report one issue and another one will pop in its place.  It feels like I have been faced with the same bug wearing different masks now for two years straight.  As I said, it feels like an IQ test of how much grief someone will endure and I am passing it spectacularly -- or failing, depending on your perspective. 🙃
Anyways, I appreciate you being here and letting us know we're heard and you are chipping away at things.  Again, if the final deployed result didn't work so very well in production this would be a no-brainer for me, and I would have switched to another stack long ago.  Having excellent performance in production/runtime is really what's keeping me holding onto hope here. 👍
FWIW, in other news, JetBrains fixed another big nasty in my world, so that will help a bit here with the chips as you chip away over there. 🙏

Not sure if this one is already captured somewhere, but noticed nearly a minute of CPU churn after pressing a few keys in my editor this morning.  This is 2x the usual fare.  Reported here:
https://developercommunity.visualstudio.com/t/Razor-Development:-Several-Keypresses-Ta/10276767?port=1026&fsid=7a71e5fb-e57b-4e75-8bfc-f432324b281a
Also, worth noting that the GitHub issue here is being tracked in my semi-weekly standups in the Friction Points section where I discuss items causing me distress:
https://youtu.be/jSQxW0DJjtI?t=851
I hope I pronounced your name correctly, @davidwengier 😬  I may just call you "Mr. David" from now on. :)

Close enough :)
In other news, another perf improvement just went in, which has some very good results for improving formatting time and memory. Even if you never explicitly format a document, you'll still get some benefit from this PR whenever you paste, as at the moment the editor is always issuing a formatting request when pasting (tracked by #7025)

Another baddie gets tagged: https://github.com/dotnet/razor/issues/8371#issuecomment-1451528446

Thank you for your continued dialog @davidwengier it is appreciated.
I am enduring some grief here this morning, and I wanted to be sure about the ticket I referenced in the opening message of this ticket.  I realize there are several issues that are being tracked.  However, the one that is generally causing me the most harm is high RAM usage that does not appear to be properly swept.  This ultimately leads to high RAM utilization and constantly disruptive GC sweeps.
The IDE does not start out like this, so to me, this seems to imply a memory leak somewhere.
My inquiry with this here is that this seems like a bug that can be addressed sooner rather than later, as opposed to a fundamental issue with tooling that might take a while.  I could be misunderstanding this, of course, so wanted to pipe up here and get clarification if possible.
For additional reference for my morning here, I am facing GC sweeps roughly every 20 seconds ATM with basic typing in my environment, which has 32GB of RAM dedicated to it via Hyper-V Manager. 😬

FWIW JetBrains is looking into some of the issues I am encountering and they are confirming a 1GB memory leak in Visual Studio when opening/closing the v3.zip solution as provided/reported to you:
https://youtrack.jetbrains.com/issue/RSRP-491473/R-Razor-Development-EAP-UI-Freezes#focus=Comments-27-6927940.0-0

I wanted to be sure about the ticket I referenced in the opening message of this ticket

The main culprit in your screenshots seem to be TagHelperDescriptors, which is a bit of a problematic area. Firstly the good news is that the issue is being actively worked on (eg, https://github.com/dotnet/razor/pull/8369) by the best minds in the business, but there is slightly more to it as far as a long lasting solution goes.
The TL;DR is that tag helper descriptors are something of a communication mechanism between tooling and compiler. The compiler "discovers" them by finding all components in the project, referenced projects, and references assemblies, and then sends that list over to the IDE tooling. The IDE tooling caches them, so it can hand the list back to the compiler when it needs to compile a Razor page. The tooling also looks after invalidating that cache (when files/projects change) and asking the compiler for a new list (or rather, a delta) if necessary. At runtime some amount of work has to be re-done because runtime compilation of Razor is slightly different, and driven by a source generator.
All of this communication means lots of object allocations, serialization, deserialization etc. Long term what we're aiming for is to converge on a single compilation of Razor for design time and runtime, and have the source generator running for both scenarios. This would mean the entire compilation model of the IDE is flipped on its head, and instead of the IDE telling the compiler when it needs to do work, the source generator ("the compiler") would be run by Roslyn and it would provide data to the IDE. This should remove a bunch of the communication requirements for tag helpers (in theory, everything except the bare minimum needed for IntelliSense), but is a rather large undertaking with many steps. Rest assured we are actively working on those steps, on both the compiler and tooling teams.

Thank you for the update and for your comments @davidwengier, it is greatly appreciated.  After hitting those GC sweeps, I took a look at my Hyper-V configuration and noticed that while I had 32GB allocated to my development instance, it was a max allocation using dynamic memory.  The instance had 16GB allocated to it from a starting configuration.
I did some housekeeping and reconfigured the utilized RAM from my other instances, and now my development instance has 32GB starting allocation with dynamic going to 40GB.  I developed yesterday and my environment seemed a bit nicer without the constant+obvious sweeping that I was encountering several days prior.  I emphasize seemed as that whenever I mention any success around these issues is when they usually get worse. 😩
For the time being, it looks like I have bought myself some further breathing room and can attend to the next fire here.
Thank you for letting me know about #8369.  It is beyond awesome for me to see the effort. 👍💖🙏😌

So I have been using Visual Studio 17.6.0 Preview 2.0 since its availability last Tuesday and it has been quite remarkable.  I have noticed no GC sweeps and RAM utilization is usually around ~10GB.  Additionally, forcing GC after closing all documents will get RAM utilization to ~5GB which seems new.  Usually, once it gets to 10GB (or higher) it will stick there and forcing GC will not matter.
It's gotten to the point here that I wanted to stick my neck out here and see if #8369 is the reason for this magic. 🤔

@davidwengier / @DustinCampbell / @ryzngard / @sharwell  / @333fred / @jaredpar
I wanted to take some time to ensure things are as they seem before commenting and closing out this issue.  Everything is really amazing now, and it feels like a new IDE for me. I still get 25% CPU utilization for 30-60 seconds at a time with a single keypress, but that pales in comparison of having GC sweeps occur every 30 seconds and freezing my IDE for 30 seconds at a time.
My IDE now reports 5GB of usage via R# memory reporting consistently.  Sometimes even dipping in the high 4GB range.  It will jump up to 10-15GB sometimes but it's brief and drops down to 4-5GB after a while with typical GC. There are no longer frequent+disruptive UI freezes and everything feels/seems like it should be. 😌🙏
I can easily develop in this environment now, and I no longer think of having made the wrong life choice in choosing my current development path.  Everything seems possible again and I am elated with my current orientation. 🙏🙏🙏
What really cements this for me is the great communication and effort here by the team there.  I am beyond grateful that you take the time to listen to your customers and fix the important parts when necessary.  It took a really long time to address this after multiple attempts with many various previous issues, but in the end you finally fixed the root issue ailing me during all this time, and that means everything to me.    I mean, you had me already, but this super cements me in, as I said.
Thank you again.  Please know that you and your efforts are making a difference out there. 👍💖🙏

Thank you for the kind words @Mike-E-angelo, we're glad things have improved, but I thnk I can speak to everyone when I say: I hope you don't mind if we keep working to improve things  😁


lightspark-sdk README example createInvoice error

The example does not work, it gives me this error
LightsparkException [Error]: Request CreateInvoice failed. [{"message":"Variable '$node_id' got invalid value 10; Expected type 'ID'. The id input is not valid.","locations":[{"line":3,"column":5}],"extensions":{"error_name":"InvalidInputException"}},{"message":"Variable '$amount_msats' got invalid value 'Whasssupppp'; Expected type 'Long'. The input value must be an integer.","locations":[{"line":4,"column":5}],"extensions":{"error_name":"InvalidInputException"}}]
It is outdated?

Sorry about that! Looks like the README was incorrect. Could you please try:
const encodedInvoice = await lightsparkClient.createInvoice({
  nodeId: RECEIVING_NODE_ID,
  amountMsats: 100000,
  memo: "Whasssupppp",
  type: InvoiceType.AMP,
});

It didn't work that way, I made it work like this:
const encodedInvoice = await lightsparkClient.createInvoice(
RECEIVING_NODE_ID,
100000,
"Whasssupppp",
InvoiceType.AMP,
);

Thanks!


Cleanup project structure for better package importing

Remove TextMeshPro, remove lighting, rename Plugin to Plugins for it to be detected, place DemoScene and Scripts in Demo subfolder

Hm I've been building with the 3 PRs merged just fine, maybe it's because I have other things in

The .jslib move to Asset/Plugins doesn't work for me. The js functions in the jslib don't get injected when I build. I remember having the same issue initially which is why I moved it to /Plugin instead. Am I missing something?

Hm I've been building with the 3 PRs merged just fine, maybe it's because I have other things in Plugins? It's the folder recognized by Unity in the docs. I'm using v2021.3.11f1, maybe that matters?


pfi-aud-usd-tbdex error: Expected kid in JWS header to dereference to a DID Document Verification Method

Running the pfi-aud-usd-tbdex example unmodified throws an error:

This issue was initially reported on discord and I was able to reproduce it by running the install/first run steps listed in the README.

fixed: https://github.com/TBD54566975/tbd-examples/pull/25

fixed