[SIP-75] Proposal for Embeddable Dashboards

[SIP-75] Proposal for Embeddable Dashboards
📣 Status
Currently in alpha and under active development. You can activate the feature by turning on the EMBEDDED_SUPERSET feature flag. Embed dashboards into your application using the @superset-ui/embedded-sdk package on npm.
It is not advised to use this feature in production at this time! This will be subject to breaking changes from future PRs, and currently lacks important user-facing functionality.
As we continue development, relevant PRs will mention this issue, and be linked automatically by GitHub below. Check back with this page to stay up-to-date with progress. (you can also subscribe in the issue sidebar)
Motivation
Embedding Superset dashboards into a third-party application is currently only possible if the dashboard is made public, or if users authenticate through Superset. We would like to add a feature to integrate dashboards into third-party applications ("Host Apps") with authentication handled by the Host App.
There should be no need to log in to an integrated dashboard if the user is already logged in to the Host App.
Example use cases:

A storefront wants to offer engagement metrics to each of its vendors, integrated into the store management experience.
A bank wants to offer a curated financial wellness dashboard to account holders.

Superset currently allows for a variety of valid strategies for granting API access, which can include various types of API Keys, depending on the deployment. Rather than force a specific API Key system into Superset, we’ve designed this spec to be agnostic to those details and depend on the existing security manager features as much as possible.
Proposed Change

Dashboard creator marks their dashboard as "embedded" in the dashboard UI. This design isn’t done yet, but it’ll likely be a modal or a new panel in the edit mode sidebar. Dashboard creator defines a list of allowed referrer domains. When a creator makes a dashboard embedded, documentation is displayed on how to actually embed the dashboard.

Introduce a new endpoint POST /embedded-resource/, which returns an Embed Token that can be used to view the embedded resource. This token will be a JWT containing user info, row-level-security parameters (could be encoded either as a filter expression or as a RLS rule id), and grants of access to certain dashboards. It expires fairly quickly, so it must be refreshed periodically.

A Superset SDK is available for the Host App frontend to use, which will render the embedded dashboard page in an iframe, and pass the Embed Token through to the iframe using postMessage. The Host App will pass a function to the SDK which fetches a new Embed Token, so when the token is about to expire, the SDK can get a new one.

A new page, /embed/dashboard/:id will be used as the url for the iframe. This page acts as a wrapper around the dashboard. It receives the message from Host App containing the Embed Token, validates the sender of the message, and ensures that all requests use the token as auth.

When a request is made using an Embed Token, security manager will use the information in the token to do its checks rather than looking up user, role and RLS info from the database.

Sequence diagram of the embedded dashboard auth flow:

New or Changed Public Interfaces
POST /embedded-resource/ endpoint
A new frontend SDK that will expose functions to embed a dashboard into the Host App
New dependencies
flask-jwt-extended for handling jwt embed tokens
Migration Plan and Compatibility
Will create new table embedded_dashboards containing a dashboard id reference, an allowlist of referrers, and any additional configurations of the embedded dashboard
Rejected Alternatives
OAuth login - requires users to sign in to the iframe, lower UX
Generating a url or token in the Host App - better to keep generation on Superset side to avoid exposing implementation details, and so that we don’t have to define a specific API key spec
Passing a signed url instead of an embed token - This makes revoking access more complex

This SIP is approved! Thanks for the discussion, all. Closing it now.

This should be considered as part of this work
https://github.com/apache/superset/issues/17187

Thank you @suddjian for the proposal and your commits. It'll be life saver.
There is one thing that I would like to mention before you guys finalizing this feature.
Custom Url Parameters feature is one of the handy functionalities and It is not dispensable. When I check the master branch, for now It seems Embedded SDK only takes dashboardID parameter.
To be able to use {{ url_param('custom_variable') }} and reference It in dataset SQL queries, I think there should be an other SDK parameter like url_params: string as we all used to use a lot of url parameters.
Example from doc:
www.example.com/superset/sqllab?countrycode=ES&language=EN

SELECT count(*)
FROM ORDERS
WHERE country_code = '{{ url_param('countrycode') }}' AND language_code = '{{ url_param('language') }}'

Thanks for the comment, @gokhansari. Rather than using URL parameters, we can use the message channel shared by the SDK and Superset-within-iframe to accomplish this.

Hi @suddjian I'm trying to use the embedded SDK and integration the SDK into my project like this:
embedDashboard({
      id: "3", // given by the Superset embedding UI
      supersetDomain: "http://localhost:62421/superset",
      mountPoint: document.getElementById("superset-container"), // any html element that can contain an iframe
      fetchGuestToken: () => {setTimeout(() => "a token from swagger ui", 10) },
      debug: true,
    });

but the sdk inject an url http://localhost:62421/superset/dashboard/3/embedded in iframe which showing "page not found" am I doing wrong?

I'm using the kubernetes deployment, and already set the EMBEDDED_SUPERSET to True in pythonpath/superset_config.py by modify the configOverrides value in my values.yaml

Updates, I'm now able to use the embedded sdk by these settings:

Updates, I'm now able to use the embedded sdk by these settings:

Where did you put these values, @idears-org? I run superset from docker

@campino2k
I'm using helm to deploy on my cluster, and these setting are placed in my values.yaml
# A dictionary of overrides to append at the end of superset_config.py - the name does not matter
# WARNING: the order is not guaranteed
configOverrides:
  override:: |
    FEATURE_FLAGS = {
      "EMBEDDED_SUPERSET": True
    }
    ENABLE_PROXY_FIX = True
    PUBLIC_ROLE_LIKE = "Gamma"
    WTF_CSRF_ENABLED = False

It will automatically set the superset_config.py
There should be a way to set the same by using docker.

Thanks. I inspected the config.py and found it also uses Environment Variables as overrides, so:

works like a charm.

Heads up for anyone using this feature. Some time soon (after #19364 is merged) there will be an update pushing out a breaking change.
Instead of using the Dashboard id, and any dashboard being embeddable via the sdk, your app will need to use an Embedded Dashboard uuid that can be found through the dashboard options menu. See that PR I mentioned for more info on that.
I'll post here again when that update goes out. I don't anticipate further breaking changes after this update.

Is this feature available now in test?
Or no way to have a try?

@suddjian I am curious about this new embedded feature. Does it require that all users are precreated on the SS application before a dashboard can be accessed from an iframe in the host app?
I have successfully embedded SS reports in our app using standard OAuth SSO. I had to write some custom python code in the values.yaml which the helm install injects into superset_config.py, but after that it works like a charm and does not require the user to be precreated in SS or to login to view reports.
I use a shared OAuth service provider, Keycloak in my case, which our app uses to request the JWT access token and operates no differently than when using Google or some other authentication provider to provide SSO.  I simply have to pass the signed JWT as an access token parameter with each dashboard URL request, which of course contain the user's scoped info and appropriate roles, and our dashboards are configured with using the SS dashboard role-based access feature. New users are created on the fly and SS roles are updated from the JWT token as needed, so very minimal ongoing coordination needed between our app and the SS app as our user base grows.

Could we filter embedded data by userid?
I mean I would like to embed superset into my application and would like to display only data that are relative to the current login user.

Hi, I am able to embed superset into our website. It works fine as read-only dashboard. However, when I click "Enter fullscreen", or "Edit Dashboard" -> "Discard", it shows this error:
{"errors": [{"message": "403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}

"Edit chart" shows a blank page.
"CREATE NEW CHART" gives this error in the browser console:
Blocked opening 'https://superset.example.com/chart/add?dashboard_id=5' in a new window because the request was made in a sandboxed frame whose 'allow-popups' permission is not set.

I have given the guest role "Admin" and the rls is empty when getting the guest token.
Please keep up the great work! Hopefully, these issue can be resolved soon.

Could we filter embedded data by userid? I mean I would like to embed superset into my application and would like to display only data that are relative to the current login user.

I think it should work by using Jinja templating. There is a {{ current_user_id() }} macro available which should work just fine with the Guest Token that is used by the Embedded dashboard. https://superset.apache.org/docs/installation/sql-templating

Hi, I am able to embed superset into our website. It works fine as read-only dashboard. However, when I click "Enter fullscreen", or "Edit Dashboard" -> "Discard", it shows this error:
{"errors": [{"message": "403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}

"Edit chart" shows a blank page.
"CREATE NEW CHART" gives this error in the browser console:
Blocked opening 'https://superset.example.com/chart/add?dashboard_id=5' in a new window because the request was made in a sandboxed frame whose 'allow-popups' permission is not set.

I have given the guest role "Admin" and the rls is empty when getting the guest token. Most other menu links seem to work.
Please keep up the great work! Hopefully, these issue can be resolved soon.
@shenrie Do you mind sharing your changes to superset_config.py? I am interested to see how to hide the superset top navigation menu.

hey, can you provide me documentation to embed an iframe in my web application ? I'm trying to use the jwt from the security api but no result .

@faycal-merouane Here is an online instruction you can try:
<EMAIL_ADDRESS>

I don't see this in my workspace. thanks for any help

I don't see this in my workspace. thanks for any help

I think you need to set ["EMBEDDED_SUPERSET": True] in your configuration file for superset under default feature flags for the embed feature to be shown.

Does anyone know how to control the layout of the embedded dashboard.
I used the following to pick a specific chart on the dashboard since I need to customize it with react components but my react app still shows the entire dashboard.
embedDashboard({
        local:'en',
        id: '28c544a1-c74e-46ef-b466-f2e6e67a6ed3', // Replace with the ID of your dashboard
        supersetDomain: 'http://localhost:8088', // Replace with the URL of your Superset server
        mountPoint: document.getElementById('superset-container'), // Replace with the ID of the DOM element in which you want to render the dashboard
        fetchGuestToken: () => embedToken,
        layout: [
          {
            components:[
              {
                id:'2106',
                type:'slice',
              }
            ]
          }
        ],
        dashboardUiConfig: {
          // Custom options for the dashboard, if needed
        },

      });

I'm also not sure what the type is for a specific chart in a dashboard, so I used 'slice'.

Regarding URL Parameters for template dashboards.
https://github.com/apache/superset/issues/17187#issuecomment-1045047163
Hi @suddjian

Thanks for the comment, @gokhansari. Rather than using URL parameters, we can use the message channel shared by the SDK and Superset-within-iframe to accomplish this.

Could you expand upon this or provide an example of how this might be achieved? I've looked all over for an example and can't find anything of value

Hi. returned {"errors": [{"message": "403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}
EMBEDDED_SUPERSET": True

Hi. returned {"errors": [{"message": "403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}
config on EMBEDDED_SUPERSET": True

I think this discussion will help you. You need to configure a Superset Role on your Superset server that will be used to apply permissions for your embedded Guest Token:
https://github.com/apache/superset/discussions/18814#discussioncomment-2776733

Hi guys HELP ME. 403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server
My backend superset_conifg.py
PUBLIC_ROLE_LIKE_GAMMA = True
GUEST_ROLE_NAME = "Gamma"SQLALCHEMY_ECHO = True
SESSION_COOKIE_SAMESITE = None
ENABLE_PROXY_FIX = True
FEATURE_FLAGS = {
"EMBEDDED_SUPERSET": True
}
I got guest token, My Frontend:

getToken = () => {
$.ajax({
'type': 'POST',
'url': "**************",
'data': {},
'headers': {
'X-CSRFToken': "{{csrf_token}}",
},
'dataType': 'json',
'success': function (data) {
getGuestToken(data.access_token, data.refresh_token)
},
});
}
getGuestToken = (access_token, refresh_token) => {
    $.ajax({
        'type': 'POST',
        'url': "******",
        'data': {
            'access_token': access_token,
            'refresh_token': refresh_token
        },
        'headers': {
            'X-CSRFToken': "{{csrf_token}}",
        },
        'dataType': 'json',
        'success': function (data) {
            supersetEmbeddedSdk.embedDashboard({
                id: "*******************************",
                supersetDomain: "********,
                mountPoint: document.getElementById("superSet"),
                fetchGuestToken: () => data['token'],
            });
        },
    });
}

@suddjian I am curious about this new embedded feature. Does it require that all users are precreated on the SS application before a dashboard can be accessed from an iframe in the host app?
I have successfully embedded SS reports in our app using standard OAuth SSO. I had to write some custom python code in the values.yaml which the helm install injects into superset_config.py, but after that it works like a charm and does not require the user to be precreated in SS or to login to view reports.
I use a shared OAuth service provider, Keycloak in my case, which our app uses to request the JWT access token and operates no differently than when using Google or some other authentication provider for SSO capability. I simply have to pass the signed JWT as an access token parameter with each dashboard URL request, which of course contain the user's scoped info and appropriate roles, and our dashboards are protected using the SS dashboard role-based access feature. New users are created on the fly and SS roles are updated from the JWT token as needed, so very minimal ongoing coordination needed between our app and the SS app as our user base grows, which was an important requirement for our use case. We cannot precreate or have to manage users in SS beyond standard OAuth capacity.
I have attached a screen capture of our app showing an example SS report embedded in an iframe. We are still iterating on the UI and style of the SS reports for a better experience, but this demonstrates the point.

Hi Shenrie ,
could you please provide the custom code that you have developed to use SSO and get JWT token for embedding, from official rest api documentation, login API does not support SSO based authentication.

Any updates on this? Does anyone have a reference on how to get the tokens for embedding a dashboard, when oauth is enabled?
Thanks in advance

Thanks for the comment, @gokhansari. Rather than using URL parameters, we can use the message channel shared by the SDK and Superset-within-iframe to accomplish this.

Could you please share about an introduction how to we use the message channel?

Hi everyone, so I have a different use case, I'm trying to embed a dashboard(lets say it is landing page) which contains link to other dashboards  so the landing page is getting embedded correctly but the issue I faced is when I try to access the other dashboard it is casusing error the error is occuirng at  these api's image attached

https://github.com/apache/superset/assets/116628630/8a72d51c-1bb8-4cbe-9ca9-81e60b648588

Hi all... this SIP was passed and implemented forever ago. It's not really sustainable to pile on with more questions/bugs/requests, so I'm going to (politely as possible, I thank you all!) lock this thread, and encourage anyone seeing this to open either a new issue or discussion. Thanks!


[feature request]: support filter by not only  path but also request method

https://github.com/sphinx-contrib/openapi/pull/78
https://github.com/sphinx-contrib/openapi/pull/64


fix(file-nesting): ensure consistent application of multiple nesting

When multiple file nesting rules were configured, rules were being applied inconsistently. For example, sometimes when both package.json and vite.config.* rules existed, only one set of rules would be applied while the other would be ignored intermittently.
Example configuration that was working inconsistently:
{
  nesting_rules = require('neotree-file-nesting-config').nesting_rules,
}

The fix modifies the nesting logic to:

Collect all matching rules before applying any nesting
Use file IDs to prevent duplicate nesting
Track nested status properly
Apply rules in a deterministic order

This ensures that all configured nesting rules are applied consistently regardless of their order or complexity.
Fixes #1494

It sounds like a good solution, but I'm afraid I don't have the time to properly evaluate it. See https://github.com/nvim-neo-tree/neo-tree.nvim/issues/1606.


TO_REWORD

@guidograzioli That's a fix for jws-dot... The weird part is that I'm pretty I already added the missing folder that a few weeks ago... Weird.

@guidograzioli disregard my previous comment, this is a "new fix".


[#3167] Support for active thread of vert.x

Codecov Report

Merging #3209 into master will increase coverage by 0.03%.
The diff coverage is 70.87%.

@@            Coverage Diff             @@
##           master    #3209      +/-   ##
==========================================
+ Coverage   39.27%   39.31%   +0.03%
==========================================
  Files        2042     2045       +3
  Lines       67188    67198      +10
  Branches     9331     9323       -8
==========================================
+ Hits        26388    26416      +28
+ Misses      38200    38194       -6
+ Partials     2600     2588      -12

Impacted Files
Coverage Δ

...ler/context/active/EmptyActiveTraceRepository.java
0% <0%> (ø)

...int/profiler/context/DefaultAsyncTraceContext.java
7.54% <0%> (ø)
:arrow_up:

...int/profiler/context/active/ActiveTraceHandle.java
100% <100%> (ø)

...vercorp/pinpoint/test/MockTraceContextFactory.java
69.44% <100%> (+0.43%)
:arrow_up:

...ider/stat/response/ResponseTimeMetricProvider.java
100% <100%> (+33.33%)
:arrow_up:

...receiver/service/ActiveThreadLightDumpService.java
81.96% <100%> (ø)
:arrow_up:

...or/metric/response/ReuseResponseTimeCollector.java
76.19% <100%> (+8.62%)
:arrow_up:

...tor/metric/response/DefaultResponseTimeMetric.java
100% <100%> (ø)
:arrow_up:

...ercorp/pinpoint/profiler/context/DisableTrace.java
30.23% <100%> (+0.96%)
:arrow_up:

...nt/profiler/context/active/SampledActiveTrace.java
100% <100%> (ø)

... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 176b9f9...1689a33. Read the comment docs.


[Bug] Chart animation when time series changes

Expected Behavior
An empty chart filled with new data from an ajax call should animate from zero to the actual chart data.
Current Behavior
I have an existing line chart that has a time series on the x axis. Via javascript I am updating the chart values and chart labels, and each update causes the time series to change. The time series could an interval of 30 minutes, 1 hour, 1 day, 1 week, etc.
I am using following code:
myChart.config.data.labels = response.data.chartLabels;
myChart.config.data.datasets[0].data = response.data.chartValues;
myChart.update();

In this case, the chart is updated and there is animation; however, the animation looks strange because of the differing time scale in the new values (the old chart kind of slides horizontally).
I tried first settings all the values to zero, before inserting the new data before the ajax call is made:
myChart.config.data.datasets[0].data = [];
cpuChart.config.data.labels = [];
cpuChart.update();

// ....perform ajax call to get new data

myChart.config.data.labels = response.data.chartLabels;
myChart.config.data.datasets[0].data = response.data.chartValues;
myChart.update();

As you see, once the ajax call has completed, I then set the labels and data and call myChart.update(). However, in this scenario, there is no animation between the empty chart and the newly inserted data and labels. The new line series just appears immediately with no animation.
I have also tried using myChart.update(5000), but it has the same effect.
Environment

Chart.js version: https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.7.1/Chart.bundle.js
Browser name and version: Chrome Version 58.0.3029.110 (64-bit) (Ubuntu 16.04)

@JonoB after setting the new data have you tried .reset() ?
myChart.config.data.labels = response.data.labels;
myChart.config.data.datasets[0].data = response.data.chartValues;
myChart.reset();
myChart.update();

Thanks for the response @etimberg !
I just tried your suggestion, but it looks like the same thing happens: https://youtu.be/S84noJOSZK0

Hmm, looking more at the code of reset, this might work but I don't know about the performance implications
myChart.config.data.labels = response.data.labels;
myChart.config.data.datasets[0].data = response.data.chartValues;
myChart.update();
myChart.reset();
myChart.update();

Do you have a fiddle that reproduces this?

Another option would be to use something like .splice() on the existing array to shift in the new elements (so that the underlying array doesn't change and the models are created) then my original suggestion might work
const len = myChart.config.data.datasets[0].data.length;
myChart.config.data.datasets[0].data.splice(0, len, ...response.data.chartValues)
myChart.reset();
myChart.update();


Nullish coalescing operator makes package incompatible with webpack

The package breaks a build with Webpack 4. This is because of the nullish coalescing ( ?? ) operator. Instead of using ?? use || for backward compatibility.

i guess it doesn't hurt in this case to change them
https://github.com/lekoala/bootstrap5-tags/releases/tag/1.5.11


[Refactor] Refactor MaterializedViewRule to support complex sync mv

What type of PR is this:

[ ] BugFix
[ ] Feature
[ ] Enhancement
[x] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Problem Summary(Required):
Fix https://github.com/StarRocks/starrocks/issues/22401
Decouple the ugly CreateMV's output name must be mapping/associated with the Base table:
TODO:

For sync mv, keep baseColumnName in the Column, so can be used for MV Rewrite
Support output name alias later.
Support complex expression output.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[ ] 3.0
[ ] 2.5
[ ] 2.4
[ ] 2.3

[FE PR Coverage Check]
:heart_eyes: pass : 233 / 249 (93.57%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/optimizer/rule/mv/MaterializedViewRewriter.java
34
44
77.27%
[142, 144, 145, 146, 148, 149, 150, 151, 152, 154]

:large_blue_circle:
com/starrocks/sql/ast/CreateMaterializedViewStmt.java
13
16
81.25%
[249, 257, 261]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/mv/MaterializedViewRule.java
164
167
98.20%
[209, 210, 648]

:large_blue_circle:
com/starrocks/catalog/MaterializedIndexMeta.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/mv/MVUtils.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/sql/ast/MVColumnItem.java
9
9
100.00%
[]

:large_blue_circle:
com/starrocks/catalog/Column.java
6
6
100.00%
[]

:large_blue_circle:
com/starrocks/alter/MaterializedViewHandler.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/sql/InsertPlanner.java
1
1
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 194 / 209 (92.82%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/optimizer/rule/mv/MaterializedViewRewriter.java
32
44
72.73%
[141, 142, 144, 145, 146, 148, 149, 150, 151, 152, 154, 156]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/mv/MaterializedViewRule.java
160
163
98.16%
[208, 209, 647]

:large_blue_circle:
com/starrocks/catalog/MaterializedIndexMeta.java
2
2
100.00%
[]


GPL-693 BUG - MLWH study_user table not updating

When a contact/collaborator is removed from a study in Sequencescape the entry in the mlwh persists
RT Ticket Number
702503
To Reproduce
http://sequencescape.psd.sanger.ac.uk/studies/8/collaborators
add yourself as follower
go to mlwh.study_user
find id_study_temp = 6
note users
in Sequencescape remove yourself
in mlwh.study_user user persists
Expected behaviour
in mlwh.study_user user is removed
Additional context
James G added
So based on past fixes:

Removal of the role_user entry wasn’t updating the study message in the WH. I had to uses touch to ensure that the timestamp on the study was updated, as otherwise the message wouldn’t have a more recent timestamp than the existing one. ( I think this took a couple of passes, as I don’t think I updated the timestamp first time round)

2)I think I might have needed to update the WH itself at some point to ensure existing relationships were purged. Its possible that this was just done on warehouse_two, and the unified WH may still be using the older code. Not sure though, it was a few years ago and is only a vague memory.

Oddly, there are tests on the WH side of things:
https://github.com/sanger/unified_warehouse/blob/master/spec/support/it_behaves_like_associated_with_roles.rb

And this is included in the study spec:
https://github.com/sanger/unified_warehouse/blob/master/spec/models/study_spec.rb

Looks like the study isn't being touched as intended.

Looks like the study isn't being touched as intended.

Odd, I've written a unit tests to cover the particular behaviour that is used, but in isolation its working correctly. I guess an integration test is in order.

Odd, I've written a unit tests to cover the particular behaviour that is used, but in isolation its working correctly. I guess an integration test is in order.

Buh, Integration tests work as well....

Buh, Integration tests work as well....

Bingo! The bug triggers if we have multiple users with the same role.

Bingo! The bug triggers if we have multiple users with the same role.


CLI: add profile in commands suggested by printers

A few of the asset printers provide commands to the user:

a command to display the description of an asset after a get
a command to download an asset after a get
a command to display a leaderboard

These suggested commands currently do not include the profile that was used to run the previous command. This prevent users from simply copy/pasting the suggested commands in their terminal.
Example
Current ouput
1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1
KEY                         1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3
NAME                        Skin Lesion Classification Objective
METRICS                     macro-average recall
TEST DATASET KEY            None
TEST DATA SAMPLE KEYS       None
OWNER                       MyPeer1MSP
PERMISSIONS                 Processable by anyone

Download this objective's metric:
	substra download objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Display this objective's description:
	substra describe objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Display this objective's leaderboard:
	substra leaderboard 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3

Desired output
$ substra get objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1
KEY                         1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3
NAME                        Skin Lesion Classification Objective
METRICS                     macro-average recall
TEST DATASET KEY            None
TEST DATA SAMPLE KEYS       None
OWNER                       MyPeer1MSP
PERMISSIONS                 Processable by anyone

Download this objective's metric:
	substra download objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Display this objective's description:
	substra describe objective 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Display this objective's leaderboard:
	substra leaderboard 1cdafbb018dd195690111d74916b76c96892d897ec3587c814f287946db446c3 --profile node-1

Done!


writeContract  and useReadContract method reports error "Type error: Expected 0 arguments, but got 1." writeContract

Describe the bug
I got
Type error: Expected 0 arguments, but got 1.

  113 |       return
  114 |     }
> 115 |     writeContract({
      |                   ^
  116 |       address: tokenAddress || zeroAddress,
  117 |       abi: erc20Abi,
  118 |       functionName: 'approve',
error Command failed with exit code 1.

this error when I build my next.js project
 const { data: hash, writeContract, isPending, error } = useWriteContract()
  const approve = () => {
    if (!spender) {
      console.error('spender is required')
      return
    }
    writeContract({
      address: tokenAddress || zeroAddress,
      abi: erc20Abi,
      functionName: 'approve',
      args: [spender, requiredTokenBalance || BigInt(0)]
    })
  }

Link to Minimal Reproducible Example
No response
Steps To Reproduce
No response
Wagmi Version
2.5.4
Viem Version
2.5.0
TypeScript Version
4.95
Check existing issues

[X] I checked there isn't already an issue for the bug I encountered.

Anything else?
No response

TypeScript >=5.0.4 required


[v2] ☂ Umbrella issue for v2 docs

Why contribute to Docusaurus v2 docs?

Docusaurus 2 is built with brilliance and so you may want to know how:

your dev hot reload server builds and rebuilds in matters of seconds and milliseconds
the client site will be an SPA, code-split by route, and pages are load lightning fast without even a spinner, how is that even possible 🌚
site works even without javascript

Your very chance of experiencing docs composed from scratch, rephrase, from code, to human language on a popular OS project
Enjoy the pleasure of finding things out, teach and share it with others, practice writing skills
Slash her majesty will be your friend!

List of doc pieces up for grabbing
Docs

[x]  Introduction / Contributing (#1700 @wgao19)
[x]  Getting Started / Configurations → Themes, plugins, and presets configurations (#1929 thanks, @kadair!)
[ ] Getting Started / → themeConfig (relevant comment here)
[x] Getting Started / Initialization – #1655
[ ]  Guides / Styling and Layouts → Styling components with CSS-in-JS frameworks
[x]  Guides / Docs / Sidebar → Subcategories (#1670 @wgao19)
[x]  Guides / Docs / Static Assets (@endiliey)
[x]  Guides / Blogs (@dt97)
[x]  Guides / Using Themes (#1645 @wgao19)
[x]  Guides / Reaching Users → search (#1734 @endiliey)
[x]  Guides / Reaching Users → analytics (#1678 @wgao19)
[x]  Guides / Reaching Users → SEO (@wgao19)
[x]  Guides / Deployment → Deployment with Netlify (#1719 @wgao19)
[x]  Advanced Guides / Plugins → @docusaurus/plugin-google-gtag (@wgao19)
[x]  Advanced Guides / Plugins → @docusaurus/plugin-ideal-image #1665 (@endiliey)
[x]  Advanced Guides / Themes (#1645 @wgao19)
[x]  Advanced Guides / Presets
[x]  API Reference / Lifecycle API (WiP #1724 @wgao19)
API Reference / Themes API
API Reference / Presets API
[x] docusaurus/link, useBaseUrl, head (wip @endiliey)

Miscellaneous

[x] Update all headings to sentence case (@yangshun)

Working on a doc piece
Within the markdown files for each of the doc pieces above, you may find in comment blocks some suggested outlines and reference materials:

Where to look in our code regarding the topic
Our v1 doc on the corresponding feature
Doc pieces from external projects on similar topics

As you work on the docs, use existing and external materials critically as examples or inspirations. You are more than welcome to explain things in your own style.
Also, feel free to ask us if you have any questions. You may tag us directly in your PR.
Adopting a doc piece

Create a PR and reference this issue
Within your PR, mark clearly which piece(s) you are working on
Once we see your PR, we'll mark in this issue that the corresponding piece is under work in progress by you
If you don't feel like continue working on the piece, close the PR so that other people can pick it up

Note that:

Your initial work does not have to include all the content for the piece, put WIP to indicate that you are working on it
You don't need to ask for permission or progression, etc. As long as there is no open PR (we will mark it beside the section), you can pull one up for the piece you are working on

Have you read the Contributing Guidelines on issues?
Yes

@wgao19 I would like to help with some of the parts

@kushthedude sounds great! You may PR directly for the piece you hope to work on. Thank you!

assigning myself for docs on docusaurus/linj, withbaseurl, head, etc

@wgao19 I would like to contribute to some of the docs namely Guides/Blogs for a start. I went through the documentation and the docs folder of the project and went through the guides-blog.md file. But am unable to understand what type of changes to be in that doc piece. I am a beginner by the way so any help on this would be really appreciated.

hey @dt97 thank you for offering to help! Based on the file names you mention, seems like you are looking at v1 docs. To clarify, this issue is for Docusaurus v2, which is a completely new static site generator with focus on doc sites.
If you are hoping to hop on to v2, the toc in this issue are referring to the docs in this directory. You should be able to match the files there. And within the actual files, there are "work in progress" status marks alongside some references.
Would you like to give them a try?
If you prefer to work on v1 docs as you may be using it, that is ok too. Please let me know which part you'd like to work on, and I'll see if I can provide any guidance on that.

Thanks for the help. As suggested by you, I was looking out for the files in the wrong directory. I want to contribute to the docs namely Guides/Blogs for Docusaurus v2. I referred to the corresponding doc for Docusaurus v1 for getting some idea about it, but am unable to get a clear idea as to how to get Docusaurus v2 up and running on my macOS Mojave version 10.14.5 and access the Blogs section for creating Blogs.
Any guidance on getting started with Docusaurus v2 Blogs would be much appreciated. Meanwhile I am learning how to use Docusaurus and trying to understand its features by deploying a demo website going through Docusaurus Installation.

Hey @dt97, once again the Docusaurus Installation docs you are referring to is v1 docs. V2 docs are in this directory, and a website version is in Docusaurus v2 site.

Thanks. Working on it!!!

While going through the Docusaurus Contributing document found that the Discord Invite to join the docusaurus-2-dev channel seems to be invalid due to possible expiration of the invite. Not clear whether an issue needs to be created for this or not.
Kindly look into it @wgao19 if possible, so that other new contributors can use the channel to their aid.

Hey @dt97 can try this link?

Thanks @wgao19. The provided link works fine.
Created a demo Blog markdown file in my system after looking at the sample Blogs present in the my-website folder of the Docusaurus v2 clone made following the Installation Guide.
Exploring the various features involving creation of blogs in v2 release and planning the documentation content side by side. I hope it would be fine to take some inspiration from the v1 documentation for Creating Blogs for the format and content of the v2 documentation.
Kindly correct me if I'm wrong. Planning to raise a PR soon if all goes well.

Hi @dt97 yes it is perfectly ok to use v1 as example. Looking forward to you PR :)

@wgao19 have created a draft markdown file for blog.md for Docusaurus v2, am reviewing it and would raise a PR soon. I have shared the same with you on Discord. Your inputs on any modifications required are most welcome.

@dt97 sure, will do by this week

hey @dt97 i've read through the draft and it's looking great, plz go ahead and PR this in :]

@wgao19 have raised a PR on this.

Currently, it's not so obvious what are the available themeConfig option for @docusaurus/theme-classic. We should document it https://github.com/facebook/docusaurus/tree/master/packages/docusaurus-theme-classic
Example: https://github.com/facebook/docusaurus/pull/1909/files

If I'm not wrong we need to augment the existing documentation for using-themes for v2 to add details about themeConfig option for @docusaurus/theme-classic right?

You can start editing https://v2.docusaurus.io/docs/theme-classic for the theme config. I have provided an example

Sure thing.

From what I'm reading it seems like the Infima framework powering v2 is not yet public? I'm trying to figure out the best way to override globals (fonts), and while I see how to override the coloring scheme, I am unable to find a full list of --ifm- variables. Is there anywhere I could find these? Also, how should I be directly accessing the document  to pull in customizations like Google Fonts? Do I setup my own theme to override the classic-theme?

@garretteklof how about trying https://v2.docusaurus.io/docs/docusaurus.config.js#stylesheets

Yeah that would definitely allow me to import different styles, however that's not really my concern. I could probably hack in some !important statements inside the custom css stylesheet --- just trying to find the most elegant solution. Is there a full list of --ifm- variables anywhere? The Infima doesn't look to have the docs ready. Thanks for the help regardless. There is some pretty cool stuff being done within this project!

Is it possible to support @ material-ui / styles of material-ui？

@vvni mind elaborating more?

@vvni mind elaborating more?

Using @ material-ui/styles on the server side requires additional processing, and the converted style needs to be entered into the html string.

@vvni mind elaborating more?

Thank you for your timely reply. Will you consider doing this support?

Please file a separate issue, this issue is not the right place to discuss about it. We are shorthanded at the moment but maybe external contributors might help out.

Please file a separate issue, this issue is not the right place to discuss about it. We are shorthanded at the moment but maybe external contributors might help out.

Thank you for your reply.😬

I was browsing the pages at https://v2.docusaurus.io and I felt that the docs were very complete and hence I came to look at this issue and just realized that this issue is already 95% done! Amazing work @wgao19 and the others!

Our docs are in a much better place now compared to a year ago. There are some TODOs left in the code, but I'll clear them up. Closing this issue! Thanks again @wgao19! ❤

Hey, I would like to help !!  are there any tasks left I can work on? @wgao19

@restrictedCodex this task is closed


Add support for circular buttons

Hi,
just found the repo and this is exactly what I need, however, I need a circular button with the same effect 🙂

Hey, i just added this feature and submitted a pull request, hope its going to be merged soon.

Thanks to @Saad29 I've merged and updated the library on pub.dev.


Need to trigger deletion many times to clean  up a directory

Say a directory D  is watched by chokidar, D has many subdirectories which have many sub ones, too. (e.g. node_modules directory)
When I try to remove the directory D , I have to trigger deletion many times.
Each time the deletion only cleans parts of the directory, which annoys me.
Is there any way to clean up the directory with one deletion?
I am on win10 x64,  node 4.4.0, chokidar 1.6.0
Thanks for any suggestions or clues.

Not sure what you mean by "trigger deletion".

Let me clarify it.
I have to delete the directory D several times (with Shift ＋ Delete keys on Win10) to clean up the directory and all subdirectories. Each deletion only removed some subdirectories.
Hope to clean up the directory with only one deletion.
Is it clearer?
thanks.

Still don't understand. On a Mac, pressing ⌘⌫ would remove the folder and all subdirectories if the folder is selected right now.
Do you mean that on Windows pressing a combination would only remove some subdirectories?

yes, I mean that.
Pressing Shift + Delete keys on Windows only remove some subdirectories.
Need to press the keys multiple times to remove all subdirectories.

By the way, if the directories were not watched by chokidar, one pressing Shift+Delete keys would remove all subdirectories.  However, once watched by chokidar, multiple pressings are needed.

Interesting.

This is because of strange fs locking on Windows. I really don't think there's anything we can do about it.
Possibly could be reported to libuv in case there is something they can do with the lower-level system APIs, but I wouldn't hold my breath.
Obviously you could just tolerate it as it is, stop watching before attempting a recursive delete, or switch off of Windows for this sort of dev work (a Linux VM running on Windows won't have this problem).
You could also try different approaches at deleting, such as trash-cli

@es128  thanks for response.
I am developing a tool in which chokidar is used to watch file changes and then those changes including deletion are synchronized with the server side's.
Windows OS is not only for dev work, but also a production environment,  I'm afraid I still need to support Windows any way.
Do you know any option to stop such fs locking on Windows?
thanks.

I'm unaware of an easy fix - I would have mentioned it if I were.
You could try toggling the usePolling option, although setting it to true comes with some performance tradeoffs and is not suitable for too large a file tree.
I suggest you also do some googling and look through articles like this to see if you can find any other helpful suggestions that might be tenable for your app/users.

@es128 appreciate  your suggestions. I'll give it a try.

After adding usePolling, the problem was gone.  Close this. Thanks.


Fix typo in NavigationView Sample 5

Description

This fixes a typo in the XAML snippets of NavigationView sample 5
Motivation and Context

This change makes the sample working
How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

@microsoft-github-policy-service agree

/azp run

Good catch and thanks for the fix!

/azp run


add bans command

gets information on <user>'s minecraft bans from fishbans

Since fishbans doesn't really exist anymore, I'm gonna go ahead and close this and maybe it'll be replaced eventually by a different ban command.


zpool: Fix broken Pool.init()

Noticed this constructor is not used anywhere, so it was silently broken. Would probably make sense to test adding/removing handles from a 0 capacity pool as well, but for now I just added a simple test so the init() function is compiled and analyzed by Zig.

Nice find, thanks!


Added myself

Added mine HTML file

Your pull request looks good, and is ready for merging!
Thank you for being a part of Open Source, I hope you continue to help other repositories in the future!
This is an automated response, beep boop.
if you think there is a mistake, contact a maintainer

d8b536f126297d5ec2da2592bb1180b42118ce2d


How do I verify Phone number?

Qmunicate iOS version is containing phone verification system.
But Android version is not having.
I hope you to add this feature and provide us updated result.

Please add phone number or email verification system  in android system.


[contao] update release policy link

Unsure if the new link destination should be

https://contao.org/release-plan or
https://contao.org/en/release-plan

https://contao.org/release-plan is better because the webserver detects the browser language and redirects to language specific url:
de -> https://contao.org/de/release-plan
en -> https://contao.org/en/release-plan
es -> https://contao.org/es/plan-de-publicacion
fr -> https://contao.org/fr/plan-de-publication


Can't get it working on Grav v1.1.5

I don't know if it is because grav version.
But I don't get any pencil on frontend.
Are there any steps I should follow to get it working?
Or is it plug 'n play, because right now it is not working like that.
TY

I Just updated the demosite to Grav 1.1.5 and it is still showing up.
You have to be logged in at the frontend and viewing a contentpage
(eg: http://fred.bughunter2k.de/blog/sunshine-in-the-hills)
But saving isn't working any more as it seems.

Link not work

http://fred.bughunter2k.de/blog/sunshine-in-the-hills
Link not work
Fixed


[!NOTE]
It runs fine on all other architectures!

To reproduce
Steps to reproduce the behavior:

run command /opt/unbound/sbin/unbound-anchor -a /opt/unbound/etc/unbound/var/root.key

Expected behavior
It should run fine as it does on all other architectures except armel/ARMv6
System:

Unbound version: 1.21.0
OS: Debian GNU/Linux 11 (bullseye)
unbound -V output:

Version 1.21.0

Configure line: --disable-dependency-tracking --prefix=/opt/unbound --with-pthreads --with-username=_unbound --with-ssl=/opt/openssl --with-libevent --with-libnghttp2 --enable-dnstap --enable-tfo-server --enable-tfo-client --enable-event-api --enable-subnet
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 3.4.0-dev
Linked modules: dns64 subnetcache respip validator iterator
TCP Fastopen feature available

Additional information
Unbound was started using a unbound.sh script available here. Updated the unbound.sh file to list the opt/unbound/sbin dir and just run the unbound-anchor command without args:
...
drwxr-xr-x 2 root root     4096 Aug 15 15:28 .
drwxrwxr-x 1 root root     4096 Aug 15 09:41 ..
-rwxr-xr-x 1 root root 10668240 Aug 15 15:27 unbound
-rwxr-xr-x 1 root root  4688360 Aug 15 15:27 unbound-anchor
-rwxr-xr-x 1 root root  9197304 Aug 15 15:27 unbound-checkconf
-rwxr-xr-x 1 root root  6178844 Aug 15 15:27 unbound-control
-rwxr-xr-x 1 root root     6143 Aug 15 15:28 unbound-control-setup
-rwxr-xr-x 1 root root  4748440 Aug 15 15:27 unbound-host
  [i] Neutrino emissions detected...
/unbound.sh: line 421:   283 Illegal instruction     (core dumped) /opt/unbound/sbin/unbound-anchor
/unbound.sh: line 427:   296 Illegal instruction     (core dumped) /opt/unbound/sbin/unbound-anchor -a /opt/unbound/etc/unbound/var/root.key
...

Looks to me like building unbound in this way will successfully build, doesn't work on the linux/arm/v6

Other users report issues with 32bit vs 64bit versions for rpi compilation. Perhaps this is a cause of the issue, what was the unbound compiled for, eg. it prints that at the configure stage, and what is the running architecture. It may also be possible to run the other binaries, say with -h, to give a brief attempt to run it.

User that initially reported to origamioffical here.  It was compiled on an x86 machine, but using the v6 platform flag.
docker buildx build --platform linux/arm/v6 -t unbound-pihole

The running arch on the rpi 1 B+:
$ uname -m
armv6l

OS is Raspbian GNU/Linux 12 (bookworm)
Updated the configure to capture output:
    ./configure \
        --disable-dependency-tracking \
        --prefix=/opt/unbound \
        --with-pthreads \
        --with-username=_unbound \
        --with-ssl=/opt/openssl \
        --with-libevent \
        --with-libnghttp2 \
        --enable-dnstap \
        --enable-tfo-server \
        --enable-tfo-client \
        --enable-event-api \
        --enable-subnet > /root/configure-output.log 2>&1 && \
    make install && \

I checked the configure-output.log, but no mention of architecture was made, it's quite a long file, and just restated the above ./configure from what I could tell.
running the unbound command itself also results in the core dump:
/unbound.sh: line 422:   283 Illegal instruction     (core dumped) unbound -v

Possibly related, I did find that the build of openssl (referenced above in the full docker file) only mentioned armv4 and armv7:
        + ./config --prefix=/opt/openssl --openssldir=/opt/openssl no-weak-ssl-ciphers no-ssl3 no-shared -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong
        Configuring OpenSSL version 3.4.0-dev for target linux-armv4

Unsure if it's red herring, so I'm rebuilding the openssl with --target=armv6-linux-gnueabihf, but my system is quite slow and takes ~6h.


Linux

I work normally on Windows
Address already in use on Linux
I took a look and felt that the address was not being used (it should be

That's weird, can you check with some other port? by default it was set to 443 since it was HTTPS

If it is port 443, there is no program on my server using port 443

I am now working normally


Improve input prompt

Currently, just a draft, looking for feedback
Scope
Just general improvements for the input box, like handling special keys and characters.
Plan

[x] Handle special characters like &!* and uppercase (solved by #66)
[x] Handle special keys such as delete, end, home, etc.
[ ] Add a working cursor so the user can edit their text

Also @Elius94 eslint seems to be configured to only allow CRLF endings

But all files are LF for some reason which causes a lot of IDE errors

Is this intentional?

Hi! yes, with clrf it should be ok for windows but I wouldn't want it to make problems on linux, however I can't tell you since I use windows. Otherwise everything is fine, thank you

The above commit fixes the bug where the box "breaks" when pressing delete/backspace/tab when empty.
Also CRLF works fine on Linux in my experience.

Above commit fixes NaN from appearing when backspacing a negative number (such as -12). Before, when you deleted the last digit before the dash, it would change to NaN (another backspace would reset to 0). Now, once the last digit it removed it just sets it back to zero.

Currently working on a placeholder feature, using chalk for colouring. However, chalk.gray results in this happening...

Any idea why @Elius94?

(above commit doesn't fully work, just so you can help test)

Fixed issue but the yellow side line still appears for some reason

Great! you are doing a great job. Thank you.
as soon as you have a final version I will do some testing myself and accept the PR

Figured out the chalk issue. Chalk inserts invisible ANSI characters that don't take up width on a screen but count toward string length.
This line then checks string length rather than visible length and over-compensates
https://github.com/Elius94/console-gui-tools/blob/main/src/components/layout/DoubleLayout.ts#L270

Figured out the chalk issue. Chalk inserts invisible ANSI characters that don't take up width on a screen but count toward string length.
This line then checks string length rather than visible length and over-compensates
https://github.com/Elius94/console-gui-tools/blob/5fceb67daf2b6daff872b967a5aa8936bdcbfe0a/src/components/layout/DoubleLayout.ts#L270

Ok, yes! Chalk style characters are not visible and should not bel counted to check the size of a printable string.
Previously the inputp popup didn't use styled strings and so there wasent this trouble.
You have to escape the style chars from string before count.

Hm, played around and changed a few things
The ANSI characters should now be factored into the calculation (only on DoubleLayout for now thought... perhaps we can look into combining the common functions into a BaseLayout in the future)
However, this weird phenomenon now occurs only when using Chalk:

It appears to draw two lines

I think now we just need to add the cursor (maybe flashing), and after that we can include this PR in the new release! Thanks :D

Yeah, when I have some free time I'll implement a flashing cursor
The extra line issue is still present so I'll exclude placeholders from this PR, and will open another just for that feature when it's ready (to avoid delaying this PR too much).

It's finally finished! Sorry for the delay, been busy with life in general lol.
Flashing cursor and placeholder is also done. Code is slightly jank right now but it should work.


Duplicate comment when converting back from document

The yaml will contain duplicate comment when converting the parsed document by "toString"
Example:
test1:
  foo: #123
    bar: 1

After converting to document and toString
test1:
  foo: #123
    #123
    bar: 1

I'm planning a major refactor of this using a new parser. I'll make sure it is covering this issue as well

Moshen, have you started the rewrite?  I just made a kludgy fix for one of the bugs I was encountering.  It occurred to me the biggest issue is you are not modifying the AST, but instead are modifying the YAML directly, and not updating the AST pointers as text is added and removed.  This causes offset issues into the YAML source.  Add on top of it the NPM packages you are tied too are outdated and you are using deprecated NPM mechanisms.  A rewrite will be a multi-day effort and is probably best done using yaml-ast-parser from Mulesoft.  Even that isn't ideal but it will give you a good starting point.  If you are going to do a rewrite, I will assist.  Let me know.

I have started the refactoring #26
The goal is to never touch the yaml string and alway do the modifications in the AST because the new AST has comments in it too.


problems controlling what ends up in output folder

@jeffkl  hi again,
another issue that i think its related with cetral pacakges.. i have this in central
<PackageReference Update="Tekla.Common.Geometry"                                Version="4.5.9"         ExcludeAssets="runtime" PrivateAssets="all" />

and in a unit test project i have this
    <PackageReference Include="Tekla.Common.Geometry">
      <IncludeAssets>all</IncludeAssets>
      <ExcludeAssets>none</ExcludeAssets>
      <PrivateAssets>none</PrivateAssets> ()
    </PackageReference>

but for some reason the dlls are not being copied over to the test folder. so tests fail.
if i rremove the  ExcludeAssets="runtime" PrivateAssets="all" from the central packages then it works fine.
however this is not what im after, basically there are a few packages that are not supposed to be copied over to our bin folder by any projects... since this dll is put to private GAC folder we usually dont allow duplicated copies of this to bloat our packages
Am i using this correctly? thanks again

MSBuild ordering matters in this situation.  The original PackageReference item is declared in the project as:
<PackageReference Include="Tekla.Common.Geometry" />

But then CentralPackageVersioning is doing an update:
<PackageReference Update="Tekla.Common.Geometry"
   Version="4.5.9"
   ExcludeAssets="runtime"
   PrivateAssets="all" />

The update overrides what you have in your project and ExcludeAssets="runtime" tells the system to not copy the runtime assemblies to your output path.
I do not recommend using any IncludeAssets/ExcludeAssets/PrivateAssets in Packages.props unless that's what you want for every project.  You generally only want to set the version.
Each project should set IncludeAssets/ExcludeAssets/PrivateAssets as necessary.

In this case, i really have no other way around... one of the reasons why
we use this central approach is to ensure all versions are the same acrross
the 100s of projects. One consequence of this is that the output of all
those 100s of projects is also tightly locked to first avoid duplicate dlls
being shipped and the second to redude the overwall build time the
ammountof copying done (this is extremely important in a build farm)
The perfect for these scenarios is that there would be a option to disable
referce copy unless explicitly required. That was the reason ive moded to
central in hope this would do it trick.
I can workaroubd by using the packageversion var and copy those to output
but that is a very ugly solution in my case...
Tests are a special case, since they need the stuff to run and we tipically
have them in seprate folders to avoid copy violations and versions clashes..
Doing what u suggest its going to be loads of maintenance for users since
they again have to be touching the project file over and over again to
exclude those dlls bloathing the project files as in the old format
Do you have any other way for this?
On Thu, 3 Oct 2019, 18:46 Jeff Kluge<EMAIL_ADDRESS>wrote:

MSBuild ordering matters in this situation. The original PackageReference
item is declared in the project as:

But then CentralPackageVersioning is doing an update:

The update overrides what you have in your project and
ExcludeAssets="runtime" tells the system to not copy the runtime
assemblies to your output path.
I do not recommend using any IncludeAssets/ExcludeAssets/PrivateAssets in
Packages.props unless that's what you want for every project. You
generally only want to set the version.
Each project should set IncludeAssets/ExcludeAssets/PrivateAssets as
necessary.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/MSBuildSdks/issues/133?email_source=notifications&email_token=AAH6HSGLI6ITVGUQWSIEH3DQMYHWNA5CNFSM4I5E2BYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIUSVI#issuecomment-538003797,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6HSGORE3HNA564ZUL6QDQMYHWNANCNFSM4I5E2BYA
.

You can have this in Packages.props:
<Project>
  <ItemGroup>
    <PackageReference Update="Tekla.Common.Geometry" Version="4.5.9" />
    <!-- All projects do not copy dependencies to the output folder except test projects -->
    <PackageReference Update="Tekla.Common.Geometry" ExcludeAssets="runtime" PrivateAssets="all"
       Condition="'$(IsTestProject)' != 'true' />
  </ItemGroup>
</Project>

That will only set the ExcludeAssets=Runtime for non test projects.

not very pretty but does the job!

excellent if there was a flag to do this :)
but thanks again.
By the way, any idea when central package management will arrive to nuget/visual studio. Notice thats its still in incubation.

I don't recommend it, but if you really want all PackageReferences to be ExcludeAssets=Runtime except test projects, you can have a single ItemDefinitionGroup instead:
<Project>
  <ItemDefinitionGroup Condition="'$(IsTestProject)' != 'true'">
    <PackageReference>
      <ExcludeAssets>Runtime</ExcludeAssets>
      <PrivateAssets>All</PrivateAssets>
    </PackageReference>
  </ItemDefinitionGroup>
</Project>

This sets default metadata for all <PackageReference /> items and would allow you to remove the PackageReference Update="package" ExcludeAssets="Runtime" for all of the packages.

yes, i will probably not apply the exclusion to all because some projects will need to copy some references! i havent tested this scenario yet since so far i only go failures for files being copy to places where they shoundt be.

@jeffkl the proposed solutions works only in visual studio! if you run from command line with msbuild
IsTestProject is not going to be evaluated correctly. problem seems comming from nuget.exe. When running from visual studio on the left

this are properly marked as runtime assemblies, if running directly from command line those are not evaluated. If we remove  IsTestProject it works as expected. however i endup with loads of dlls in output..

Closing since there's a working solution


Start collection of example playbooks

This patch starts a collection of example auto-heal playbooks. The
initial playbooks in this collection are intended to resolve CPU and
memory issues in OpenShift nodes that run as virtual machines managed by
oVirt.

@nimrodshn @zgalor @moolitayer @elad661 these are the playbooks that I used for the demo that I shared with you recently. Please review.

/lgtm


How to exit the program in kafka/examples/consumergroup/main.go gracefully?

when I test the program , I just use Ctrl + C to  killed it.
Is there a graceful way?
Or, What is the best practice to exit the program ?

1  Ctrl+C
2 kill $pid


.nonadmin files in pre-menuinst v2 conda installations

This is an issue for both constructor and menuinst I guess, but I'll keep it here because I think it has to do more with menuinst (even if the fix is in constructor proper).

The context:

On Windows, menuinst v1 has some super user permissions elevation logic to enable the creation of shortcut items in the Start Menu for "All Users" installations.
The way menuinst anticipates whether it needs superuser permissions or not is by checking the presence of a .nonadmin file in the root conda installation (aka %PREFIX%\.nonadmin). The presence of the file dictates that no extra permisions are needed. The absence means superuser was needed at installation time.

This file is placed there by the constructor-based installer when the user installs Miniconda, Miniforge, etc.
It is primarily there to notify the uninstaller whether superuser is needed or not, but menuinst also uses it because well, it's there.

Again, this is a Windows-only thing! .nonadmin doesn't exist in Linux or macOS, even if no special permissions were needed to install conda.

The problem:

menuinst v2 extends shortcut creation to all platforms, Linux and macOS included.
It still checks for permissions needed or not via the presence of this file.
However, existing Unix installations (all of those provided by a constructor-based installer built with constructor 3.4.5 or earlier), will NOT have the .nonadmin file even if no special permissions were needed.
This will make menuinst believe that, since .nonadmin is absent, it indeed NEEDS superuser permissions, and will request sudo powers, when it doesn't technically need them.

Possible solutions:

Add a post-link.sh script to the menuinst v2 package to add the .nonadmin file to $CONDA_ROOT in Unix systems. The logic would be something like "if it's not there, try to add it; if we could do it, it was a nonadmin install; if we failed, that's ok, it was an admin install".
Change how we approach the "superuser permissions needed" on Unix systems. Maybe we can ignore the .nonadmin file and instead check whether the CONDA_ROOT path is in a system-wide location (/opt, /usr/local...), outside of the current user home (~/), while handling if we are sudo already or not. This might have a few rough edges though; e.g. what if the user installed to a non-standard location like /my-stuff/conda. Is that user-only? All users? Maybe it's world writable but, does that mean that the shortcuts I am processing should be available to all users?

I feel we need to rethink how this is approached outside Windows, specially because HPC users will probably run Linux and in those setups you can find system-wide conda installations with per-user environments that should NOT ask for superuser permissions to process their own shortcuts.
Maybe it's all way simpler: if the installation is running with superuser permissions already, then install to an "All users" location. If it's not, we'll install to current user directories. But we should not depend on the state of the $CONDA_ROOT folder.
Thoughts? Tagging a few folks that have shown interest in the project or the superuser elevation issue before: @aganders3 @mrclary @marcoesters. Thanks!

Add a post-link.sh script to the menuinst v2 package to add the .nonadmin file to $CONDA_ROOT in Unix systems. The logic would be something like "if it's not there, try to add it; if we could do it, it was a nonadmin install; if we failed, that's ok, it was an admin install".

Does any other package use .nonadmin? Or does constructor add this for the sole benefit of menuinst?

Change how we approach the "superuser permissions needed" on Unix systems. Maybe we can ignore the .nonadmin file and instead check whether the CONDA_ROOT path is in a system-wide location (/opt, /usr/local...), outside of the current user home (~/), while handling if we are sudo already or not. This might have a few rough edges though; e.g. what if the user installed to a non-standard location like /my-stuff/conda. Is that user-only? All users? Maybe it's world writable but, does that mean that the shortcuts I am processing should be available to all users?

Maybe we consider two possible solutions:

Provide user input to menuinst specifying user|all user treatment. This would allow users to dictate behavior under in any situation.
Put .nonadmin in the package environment, not necessarily the root environment. This seems more natural to me anyway, since shortcuts are created for a package in a particular environment. This could address circumstances where root environments are "all users" and user environments are not.

I personally dislike the "admin unless a file has been found" approach and would rather see a ".admin" file instead. But that may be too much of a breaking change since it also affects how constructor works.
Adding a user|all option is a good idea in either case.
The tricky part for Linux is indeed an HPC environment where there is an all-users installation but every user has a shortcut that they can remove. I wonder if a check like is_sudo or UserID == FileOwnerID could be a solution here.

Put .nonadmin in the package environment, not necessarily the root environment.

This could be a promising approach, but which package environment? The .nonadmin file is created by constructor, so it needs to be in a place that both constructor and menuinst can easily find.

Provide user input to menuinst specifying user|all user treatment. This would allow users to dictate behavior under in any situation.

The problem is that most of the time menuinst is called by conda behind the scenes; it's not the user dictating that directly. So we need some kind of heuristic to decide without user intervention.

Put .nonadmin in the package environment, not necessarily the root environment.

That's promising, but it's breaking the previous assumptions for Windows (where root env is checked). Maybe we would check package env first and then only base if absent, but since absence could be both meaningful (is admin) and lack of signal (we didn't try to establish whether admin is needed or not), it'll be ambiguous.
The more I think about it the more I believe that in Unix systems we can get away with a different strategy. In Windows, requesting super user permissions is more involved than prepending sudo to the command. It's usually done from the UI, right clicking on the shortcut, or pre-invoking a CMD instance with admin permissions. So it's way more cumbersome and hence why I think previous engineers working on it decided for the file sentinel for auto-elevation.
On Unix, users could just rerun the command with sudo in-front if they reaaally need an all-users shortcut. If they are installing something for all users, that probably requires sudo anyway at that point. So I'd say we check for the user id and if it's root we go for all users; otherwise, we default to the current user location.

I wonder if a check like is_sudo or UserID == FileOwnerID could be a solution here.

Yea, this (or a similar variation) is essentially what I think we should do.

Provide user input to menuinst specifying user|all user treatment. This would allow users to dictate behavior under in any situation.

The problem is that most of the time menuinst is called by conda behind the scenes; it's not the user dictating that directly. So we need some kind of heuristic to decide without user intervention.

Conceded.

Put .nonadmin in the package environment, not necessarily the root environment.

That's promising, but it's breaking the previous assumptions for Windows (where root env is checked). Maybe we would check package env first and then only base if absent, but since absence could be both meaningful (is admin) and lack of signal (we didn't try to establish whether admin is needed or not), it'll be ambiguous.

What if we disambiguated by always creating a file, i.e. .nonadmin and .admin. If there is no file, then check base environment (for old paradigm), otherwise it is affirmatively declaring to create shortcuts for all users or just this user.

What if we disambiguated by always creating a file, i.e. .nonadmin and .admin.

That can work for new environments, but we have to figure out a way for old environments too.
I am going to start prototyping something in a PR and we'll see how ugly that looks :D

#156 is now ready for review implementing the discussed approach (sans .admin markers).


Add a TimeLineview

Mock-up

problem: adding CHorSplitterWindows and CTimeLineViewineview creates problem, the ownerdrawn controls are nolonger updated correctly, and closing the logview windows runs into an assertion.
It looks like some window-messages like WM_CLOSE or WM_DRAW are not delivered to the Logview anymore.
see maybe https://www.microsoft.com/msj/0795/dilascia/dilascia.aspx

Debugview's class MainFrame is a CTabbedFrameImpl<CMainFrame, CDotNetTabCtrl>
The next step to isolate this problem is to make the class CMainFrame from the POC also inherit from that.

Maybe it is a problem related to this:
http://forums.codeguru.com/showthread.php?337200-CComboBox-MeasureItem(-)-is-not-called
although:

it does not explain the assertion on close
we do not explicitly assign ID's anywhere

I integrated the CTimeLine control into Debugview++, it can be enabled at compile time using
using SelectedTabItem = CLogViewTabItem2;
however when it is enabled some window messages cause endless recursion and a stack overflow.
For example adding an 'exclude' process filter but using the context menu should call OnViewProcessExclude, but this now crashes with a stack overflow.


backport: bitcoin#18038, #14193, #17564, #17999, #18740, #19096, #16426, #17737, #18698: deglobalization backports (part 2)

Made possible with significant contributions by @UdjinM6

pls see https://github.com/UdjinM6/dash/commits/pr4801

LGTM but 16426 was built on top of 16034 originally and we backport 16034 in #4679 which should be merged first imo. This PR should be rebased after that.


Modify at VertexAttribute,want more control

Change the private constructors to public.Let users  have more control on VertexAttribute,like to set attribute data type ....
I saw the code of com.badlogic.gdx.graphics.glutils.VertexArray.bind(ShaderProgram, int[]),It seems supported GL_FLOAT and GL_UNSIGNED_BYTE,But when I what to create a VertexAttribute with type GL_UNSIGNED_BYTE,I found the constructors is private,so I think, it may be public to provide more control on Vertex Attribute?
Or,there has some reason must keep this constructors private?
Sorry for my english.
best regards

Maybe to support this,com.badlogic.gdx.graphics.glutils.VertexData have to add method to return the ByteBuffer?If not do that, even I can create a GL_UNSIGNED_BYTE  Vertex Attribute,but I can not set it's value?Because I can only get a FloatBuffer?
Sorry for my english.
Best regards

Theoretically this has quite some uses. But Java doesn't support structs, so without hacking around that (like is done for color), it won't be practical and probably only introduces problems. What is your practical real world use-case?
If we include this then at least it should have sufficient javadocs which includes a warning.

@xoppa
Like you say,Simply make public may just make more work due to support what i want.
make them protected would be nice,since constructors be protected,someone who want more control of VertexAttribute could find a way to do some extensions.
Thank you for think about this!
Sorry for my english.
Best regards


vnetFirewallManagementSubnetAddressPrefix is not changing

Describe the bug
Creating AKS with custom Networking option option for AzureFwManagement subnet greyed out and deployment fails in conflict

To Reproduce
Steps to reproduce the behavior:

Go to  Networking enable like in the screenshot above
Change adressIP space
vnetFirewallManagementSubnetAddressPrefix option is disabled and hardcoded in bicep
Fails in conflict with new IP space entered

The separate Management Subnet is only supported when using the Basic SKU of Azure Firewall (when deploying with AKSC).
You want to leverage it for Standard Firewall SKU @lenisha ?

Attempted to reproduce with this configuration

@lenisha - Can you advise what i've missed to reproduce your problem?

@Gordonby  yes  I picked Standard Firewall and the only change I did change from "<IP_ADDRESS>/16" address space to "<IP_ADDRESS>/16" Ip space, and accordingly AKS Nodes subnet  and other subnets. It then failed as VNET bicep tried to create included FWManagement subnet and it's ip range was "10.240.x.x" , Bicep need to be adjusted not to create it

Thanks....  Ok, trying to repro with this config now :)

Repro'd.
subnet 'AzureFirewallManagementSubnet' is not valid in virtual network 'vnet-fwrepo'


Message in slack notification

It would be really great to get the error message included in the slack notification. (Or even select all the fields you're interested in!)

I just came back looking for this - was about to comment on it and realised I was the one who posted it in the first place!
The start of the message gets added as a link in slack but it gets truncated  - It would be super to see the whole of the error message.


Calling clean_cluster
Cleaning resources in namespace kube-system
No resources found
No resources found
No resources found
configmap "extension-apiserver-authentication" deleted
endpoints "kube-controller-manager" deleted
endpoints "kube-scheduler" deleted
lease.coordination.k8s.io "kube-controller-manager" deleted
lease.coordination.k8s.io "kube-scheduler" deleted
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Cleaning resources in namespace kube-public
No resources found
No resources found
No resources found
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
No resources found
Cleaning resources in namespace kube-node-lease
No resources found
No resources found
No resources found
lease.coordination.k8s.io "dred" deleted
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Cleaning resources in namespace default
No resources found
No resources found
No resources found
endpoints "kubernetes" deleted
persistentvolumeclaim "my-release-moodle" deleted
persistentvolumeclaim "data-my-release-mariadb-0" deleted
service "kubernetes" deleted
endpointslice.discovery.k8s.io "kubernetes" deleted
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

current PR build also fails with similar error https://github.com/ubuntu/microk8s/pull/1616/checks?check_run_id=1193543738

@gliptak thanks for reporting this issue.  It looks like it is stuck in deleting the persistent volume claims.
NAME                        STATUS        VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-release-moodle           Terminating                                                     142m
data-my-release-mariadb-0   Terminating                                                     142m

I usually just do a sudo snap remove microk8s --purge if i want to nuke my cluster. Then followed by sudo snap install microk8s --classic --channel [your preferred channel]
This one does the job cleanly.

@ktsakalozos  While trying to reproduce this last weekend, i found out that in certain cases microk8s reset doesn't restore back the cni.
Because microk8s start returns 1 so the script exists before reaching the point where we apply the cni.yaml.
For some strange reason it looks like it fails here.
https://github.com/ubuntu/microk8s/blob/8c23ed3800608aa10c7b6d6b01ac9c256f382464/microk8s-resources/actions/common/utils.sh#L429
Didn't get the chance to actually print it.  It doesn't happen all the time though.  My guess is it could be related to the "checkpoint in progress" issue.  But i have to try it again.

I just experienced this issue, and I tried several actions to resolve it such as: opening another terminal, and reboot, then running the microk8s reset again, and then I would try opening another terminal and running microk8s status and/or microk8s inspect and would get results back such as microk8s is not running or docker installed, etc. (uninstalling docker didn't make any difference), and I don't recall installing it in the first place, I just installed microk8s and enabled add ons and started creating pods, etc., after much searching and trying several different options. Finally I took the advice of @balchua , and ran the command sudo snap remove microk8s --purge, which seems to be the only way out of this. Thank you @balchua for that suggestion to run the command sudo snap remove microk8s --purge.


InitialFile in ShowSaveFileDialog

InitialFile in Dialog settings is not passed to dialog itself.

Same issue here - InitialFile has no effect. The real propertyname used by Avalonia is InitialFileName.

Actually the Avalonia property is SuggestedFileName. Can apply that one easily.
SuggestedStartLocation isn't being applied either; which takes an IStorageFolder instead of a string. Will need to think of the proper solution to this.

Actually the Avalonia property is SuggestedFileName. Can apply that one easily.

Just to clarify: you mean it can't currently be applied, but you'll make it available in a future release?

Ya. Made most of the code change already; but will need time to sit down properly and think about the SuggestedFolder issue.

Sounds good. Fwiw just SuggestedFile would be extremely useful tho - the issue with it starting blank is that the user has to explicitly type a filename with the correct extension, otherwise they could save in the wrong format. That's mainly what I was trying to preempt.

It's not that simple; I'm asking public feedback and expert review before implementing a solution. It needs to be done right.
https://github.com/AvaloniaUI/Avalonia/discussions/14055

It's now fully implemented, along with a complete overhaul and simplification of FileSystem classes.

Nice! Will keep my eyes peeled for when it hits nuget


Need to setSharedAccessExpiryTime relative to Azure system time

Azure Service:  Azure Blob Storage
Java API:  SharedAccessBlobPolicy.setSharedAccessExpiryTime(Date)
I would like the ability to set the shared access expiry time relative to Azure's system time, but I don't see a way to do this - I can only give it an absolute time via a Date object.  This can cause issues with URIs that have short TTLs.  If the time drift (between the time on the system running my code and whatever time Azure Blob Storage thinks it is) is big enough, at best my URIs may expire sooner than I intended, or the URI may be invalid and return a 403 upon request due to the expiry time occurring before the time the URI was issued.
What would be nice instead would be to be able to request a relative interval beyond Azure's current system time, say via an int parameter coupled with a TimeUnit.  That way if I request a URI with a 60-second TTL, the duration of the TTL matches what I expected and is delivered by the same system that will enforce the expiration.

Hello @mattvryan
Do you still need help on this?
Note: As part of repository cleaning effort, this issue will be closed by Sep 12 if no response is received from submitter and/or not further action is required.

Yes I would still like this to be added to the storage SDK.

@mattvryan Sorry for the delay. This would require a new feature on the service.
@pemari-msft can you give some insight into whether this is likely to be offered by the service and a potential work around in the meantime?

Unfortunately, there is no “system time”. Each azure storage machine has a different system time which is where clock skew comes into play.
However, even if there were only one system time, sas tokens are generated client-side, so The sdk would have no way of telling what the system time of Azure was at that time. Unfortunately this is not feasible technically.

I am going to close this issue as it is not a feature that we can support. Please feel free to reopen it if you feel you need more support on paths forward.


parse request OR response

I'm implementing an ssdp server. This means I am receiving both broadcasted http requests or responses on a udp socket. There is no guaranteed ordering. So when I receive data it can either be a  request or a response.
Is it possible to use beast in this fashion? The read calls seem to require that you specify if you are reading a request or a response.

Currently there is no way to do that, but it is something I can add in the next big update to the parser.

That would be nice! Can you give an expectation of the timeframe of the next big update?

6 weeks give or take. For now, you can try something like this:

Read at least 5 bytes into a streambuf
Check if those bytes match the characters HTTP/
If yes, then its a response, else its a request
Now call beast::http::read with the appropriate message type, with the same streambuf. Beast will use the bytes already in the streambuf before going to the network to get more.

I already have code that uses the node parser, but I'm planning to migrate to beast. So I'll just wait a bit for now. Thanks for the fast response!

Just to let you know, that I have not forgotten about this issue. I plan on addressing it as soon as I have finished the refactoring of the HTTP parser and algorithms. Beast will provide this function:
/** Return `true` if the input sequence indicates a HTTP request.

    This function will return `true` if the bytes in the input sequence form the
    beginning of an HTTP request, or `false` if the bytes in the input sequence
    form the beginning of an HTTP response.

    When something other than an HTTP request or response is detected in
    the input sequence, the return value will be tribool "indeterminate."

    @par Preconditions

    The function requires that `boost::asio::buffer_size(buffers) >= 5`.
*/
boost::tribool
is_request(ConstBufferSequence const& buffers);

This should become an example similar to the detect_ssl and async_detect_ssl example.

There is no real use-case for this in an example program so I think we should leave this to another library.

There is no real use-case for this in an example program
What exactly do you mean with this?

What exactly do you mean with this?

I mean that any example program which would use this feature would be completely contrived.
I can help you write the code if you still need this feature though, would you be interested in that? One thing I should point out though, Beast HTTP stream operations don't work on UDP sockets. So I'm not sure how you plan to get around that.

I'm using regular asio for receiving the messages, but it would be nice if I could use beast for decoding the http messages. Currently I'm using both beast and also have a copy of the nodejs http parser in my project.
I would like to get rid of the nodejs parser, so it would be nice if I could use the beast parser, but is seems to require the isRequest template argument which I do not know.

I explained in an earlier reply how to do this. Check the first 5 characters of the input buffer and see if they match "HTTP/". If yes, then you have a response, pass isRequest=false. Otherwise, pass isRequest=true. Is this not sufficient? Example:
template<class ConstBufferSequence>
boost::tribool
is_request(ConstBufferSequence const& buffers)
{
    if(boost::asio::buffer_size(buffers) < 5)
        return boost::indeterminate;
    char buf[5];
    boost::asio::buffer_copy(boost::asio::buffer(buf), buffers);
    return std::memcmp(buf, "HTTP/", 5) != 0;
}


Increased CPU usage after updating to Win10 2004

After updating Windows 10 to version 2004 (OS Build 19041.329) Process Hacker idles at a constant ~5% CPU usage, decreased to ~3% if run as Administrator.
Scrolling the main process list is also weird: I made a short gif and as you can see every item gets duplicated and overall the scrolling does not look right. Also PH's CPU usage increases significantly as shown in this image taken right after recording the gif. This also applies when dragging any of the PH's windows other than scrolling. The Threads tab while doing any of these looks like this.
Other than that, the services.exe process also idles at a constant ~5% (image), with occasional spikes, though I'm not sure how to reproduce them. Here is services.exe's Threads tab.
Last thing: using Task Manager you can see Process Hacker and services.exe are top 2 in CPU usage, BUT closing Process Hacker completely puts services.exe at a complete rest!

Try changing the EnableServiceNonPoll setting to 1

That fixes the issue with services.exe, but PH still takes a lot of CPU when idling/moving windows/scrolling.

Can you also try disabling the "Check images for digital signatures" and "Check services for digital signatures" options in the general settings window?

Sorry, I should have mentioned earlier that I already disabled both options after reading https://github.com/processhacker/processhacker/issues/595#issuecomment-610927929.

you need to compile process hacker on your own to get your PDBs, start processhacker, now run cmd.exe as admin and run this:
wpr.exe -start  CPU && timeout 30 && wpr.exe -stop C:\CPUUsage.etl
after capturing 30s of cpu usage open the generated ETL in Analyzer (WPA.exe), configure symbols to use Microsoft symbol server and add an entry to your local ProcessHacker PDBs.
Now load symbols, drag&drop the CPU Usage (sampled) from left side of WPA to right pane, select processhacker.exe, rightlick and filter for it. Now expand the stacks in the table and look what processhacker does.

you need to compile process hacker on your own to get your PDBs, start processhacker

well I only got so far because my own build of Process Hacker doesn't have issues. Is this normal?

this is strange. Capture the trace, zip it, upload it and send the link to @dmex (look at his profile for email). He said he has the PDBs and can analyze it.

I tried to build my own version again, and this time it lags. I'm not sure what went different. I tried a couple of times again, making sure it's not the official version. Sorry for the confusion.
That said, with my own laggy build I went ahead with the analysis. I know what this is doing but I don't have the experience to make guesses nor assumptions.

The "big" spikes are when I tried to use Process Hacker (like scrolling, highlighting some processes so the tooltip would pop up etc.). I don't know what the other little spikes are.
In the second screen I highlighted every single CI.dll!SymCryptSha256AppendBlocks_ul1 function call. There's roughly ~4k as you can see, and they correspond to each spike.

 Finding the exact binary causing the issue would be very handy so we could sent it to Microsoft and get the issue fixed.

In the second screen I highlighted every single CI.dll!SymCryptSha256AppendBlocks_ul1 function call

Your stack trace is showing an application trying to inject a binary into Process Hacker using either the SetWindowsHookEx function  and/or AppInitDlls registry keys but we're using the Windows extension point mitigation policy which disables both of those injection methods for the current process. Process Hacker also uses the Windows signed only mitigation policy which prevents loading/injecting binaries not signed by Microsoft.
lines 14-21 on your stack trace here:

The injection would be returning an error code and the calling process is likely immediately trying to inject the binary again and again which is causing the random CPU usage usages you've been seeing.
Try closing/terminating all other programs on your machine until the CPU usage issue disappears and it'll probably be the application causing the issue. If that doesn't work there are some tools available for checking which applications have registered these hooks.

Easier than I thought, the culprit is Overwolf. Closing it makes everything go back to normal.
I suppose it would be worth to contact their support so they can handle this specific case better.
Either way thanks for your help.

the culprit is Overwolf.

Win10-2004 added an undocumented override exemption to the ProcessExtensionPointDisablePolicy security mitigation which was previously used to disable unauthorised third party window hooks being added to Chrome, Edge, Process Hacker and other software using this mitigation but these new override exemptions now defeat the entire purpose of using that mitigation on Win10-2004 and above.
The major performance issues with Process Hacker are a direct result of these changes. We also use the MicrosoftSignedOnly security mitigation to prevent loading of non-Microsoft binaries but win32k.sys is using the newly added exception to the ProcessExtensionPoint mitigation policy in an attempt to inject window hook DLLs from third party software into Process Hacker which then faults due to our binary also using the MicrosoftSignedOnly mitigation policy which triggers an infinite loop within win32k.sys and the kernel executive.

win32k.sys overrides the ProcessExtensionPoint security mitigation (newly added backdoor exemption for UIAccess=true).
win32k.sys queues an APC into Process Hacker for injecting the window hook DLL with LoadLibraryExW
LoadLibraryExW returns an error while loading the injected DLL due to the MicrosoftSignedOnly security mitigation.
win32k.sys ignores the failure and repeats the entire procedure over and over again in an infinite loop.

The APCs queued to Process Hacker by win32k.sys are handled internally by the GetMessage (NtUserGetMessage) function and cause the process to utilize 100% of a single processor core (you'll see 50% of a dual core machine, 25% of a quad core etc...) due LoadLibraryExW being executed by these APC requests within our process thousands of times every second.
I've tried raising the issue with Microsoft but have not received a response yet.

I've disabled mitigation polices for users running Win10-20H1 and above since Microsoft still hasn't fixed the crippling performance issues: https://github.com/processhacker/processhacker/commit/edf229a3e9e54490f231366e2e460d647e07b805
It's extremely sad that some random process on the machine calling SetWindowsHookEx can make Process Hacker completely  unusable because of that stupid backdoor policy bypass Microsoft added for UIAccess. 🤮
The only other way to fix the issue - without disabling security like commit edf229a - would be splitting the project into pieces and launching those parts on a separate desktops since window hook DLL injection using SetWindowsHookEx doesn't work across desktop sessions.... That will take quite some time to implement so unless Microsoft fixes these issues the security mitigations will now remain disabled. 😕
You can also expect attackers will now exploit those changes (great). So I've updated the policy descriptions when viewing the process mitigations so that you're (at least) aware of those changes (since they're not documented anywhere yet): https://github.com/processhacker/processhacker/commit/e856774d8e4583d46f478f56537cfc2d7bc684d0
😕

So is this also the cause for all the Event View Error spam that generates when using Process Hacker? For example "Applications and Services Logs" > "Microsoft" > "Windows" > "CodeIntegrity"

all the Event View Error spam

Yes. The events are generated by LoadLibraryEx when the injected window hook DLL failed to load because of the MicrosoftSignedOnly policy. The window hook injection is stuck in an infinite loop so you'll get an infinite amount of log messages 😂

That sucks, but good to know where and why they come from. By chance I've noticed "Kernel-EventTracing" Errors as well, and they seem to pop up (at least more often) while using Process Hacker, do you know if PH is also a cause? I cannot tell honestly, it might just be coincidence too.

I've noticed "Kernel-EventTracing" Errors as well, and they seem to pop up (at least more often) while using Process Hacker

The error code should be when the name already exists.

I just learned that applications with UIAccess=true need to be in program files or windows to run correctly.
How do you run the app that triggers it? From such system folders or from a different location?

How do you run the app that triggers it?

You can just install a third party application to easily demonstrate the issue.
Download Overwolf from here:
https://www.overwolf.com/
https://download.overwolf.com/install/Download?Channel=web_dl_btn2

Install Overwolf and then execute it.
Start Process Hacker.

You will see thousands of events in the event log caused by the infinate loop in the kernel and if you're running sysinternals debugview you'll see thousands of debug messages related to DLL injection failing because of the security policy.
You can also view the threads of Process Hacker while this is happening and see user32_ClientLoadLibrary being called and generating 100% utilization on some machines loading/unloading DLLs thousands of times a second and moving the mouse over Process Hacker windows or doing anything that generates a window message causes user32_ClientLoadLibrary to be called 10x more often than normal causing even more performance issues.
I've been forced to disable Windows mitigation policies by default because the performance impact of this issue on some machines is insane and Microsoft has so far refused to fix the issue. So you will now have to manually enable the Windows mitigation policies by selecting the 'Enable mitigation policies' from the Process Hacker options window:


Rewrite about screen to move away from Qt Designer

This is the last form that was built with the designer once some form of #29 goes in.

There's also the evidence editor form. I'll work on that in the same PR.

Addressed in PR #36


Update to Babel 6

This fixes #1. In Babel 6 the default export is no longer incorrectly exported as module.exports.
Babel users of this library will not notice this change, because Babel let's you import the module the same way. But for users of ES5 and TypeScript this is a breaking change:
ES5
// before
const wrap = require('express-async-wrap')
// now
const wrap = require('express-async-wrap').default

TypeScript
// before
import wrap = require('express-async-wrap')
// after
import wrap from 'express-async-wrap'

Have you tried this? https://www.npmjs.com/package/babel-plugin-add-module-exports
Might help you here 👍

That reverts to the old behaviour 😄

It does maybe I misunderstood the requirement :)

The whole point of this PR is to have a proper default export 🙂

@domadams What is the status of this? Babel 5 is incredibly outdated by now.

@felixfbecker I think the before for TypeScript should be:
const wrap = require("express-async-wrap");

Using the import-equals syntax does not work for me (typescript-2 with express-async-wrap-1.0.0).

It depends on your target setting. But I simply switched to npmjs.com/package/async-middleware