CLI should globally support JSON output

There should be a top-level flag that is supplied (possibly --json or -j) that switches between user-friendly output and automation-friendly (JSON) output. This lets the CLI be scriptable in many different ways, not the least of which allows the output to be further refined or interrogated with tools like jq.

Big fan of this. As an addition, if we're working on scripting, it would be valuable to also accept the json output (for keys at least) as input for token creation and actor signing.

Let's just make sure we don't hand-code something that can be done with xargs redirection. For example, you can call wash twice and use xargs to pump the output into another invocation's input

Good point. I have a feeling some of these improvements are going to show up when we start some of this scripting


added  an excluded repositry from hacktoberfest

Contributing
added  an excluded repo from hacktoberfest into excluded repo list  and this is valid excluded repo you can check
If you want to add a project to this list, please make sure that:

[x] You read the contributing guide
[x] The repository has hacktoberfest as topic and counts for hacktoberfest event
[x] The project can't be just an "add your name" or simply + 1 contribution to hacktoberfest
[x] Place them in alphabetical order
[x] One repository per PR/contribution - avoid conflicts and simplify review
[x] Your pull request is not spammy just adding an empty line, otherwise it will be marked as invalid and not count to hacktoberfest.

If they meet the requirements above, feel free to create a PR. Happy Hacking! :sparkles:

@OtacilioN      hello sir  merge this pull request because many people going in wrong repo


Fixes lint in validate target

For some reason, the golint check in make validate does nothing, and thus, we missed some lint warnings 🐙.
This add a lint target and make the validate target depends on and make it pass.
/cc @calavera @MHBauer @mrunalp (for the comment on CgroupSpec
🐸
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>

LGTM.
lol-branch-name
No comment on CgroupSpec comment.

@vdemeester LGTM. Thanks!

LGTM

LGTM


Professores Facid

@reinaldosaraiva Thank you for your request. Please provide us with:

the school official website URL
the school official full name ("unifacid wyden" seems to be either incomplete, or incorrectly lower-cased, or shortened, or all of the above)
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course

Also, please provide us with a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.
You can provide us with any of the following:

A copy of instruction (a hard-copy or electronic one) given to you by the school that tells you to use an email address in this email domain
An URL of a page at the official school website that says that students use this email domain
A reference to a recognized journal where an article was published by someone from the school, with explicit affiliation to the school, and e-mail address from the email domain
A WHOIS data that link the email domain to the school somehow
A copy of official letter from the school representative saying that the school provides e-mail addresses in this domain to the students
Anything else that can prove this fact in similar way, i.e. can be verified independently.

@reinaldosaraiva You might miss my previous comment. Please react.

@reinaldosaraiva I am sorry, this request remains unanswered for a while. I have to close it without merging. Please reopen and comment if you have an update regarding the aforementioned issue. Thank you for your understanding!


Handlamp Command Conflict

When used in conjunction with https://thunderstore.io/c/lethal-company/p/Mellowdy/HandheldMap/
Typing in hand-held map into the terminal to buy it instead prompts for buying the handheld lamp instead, even when headlamp is loaded and headlamp is a store item are both set to false
Is there a way to make the command matching for it less aggressive?

There was a mod called terminal command fixes or something like that. That will fix it. I don't know why the terminal is so blunt in vanilla. I may change the name a bit but we'll see.

Ah, yeah, if it's a vanilla bug, it can't be helped
Thank you for the suggestion,. I'll give it a a try!

I'm glad, hope you enjoy.


Make Folded transparent as well if transparency is enabled

The title says it all. It doesn't look consistent if everything has a transparent background except the folds.
Would be a nice extra to have a consistent appearance 😏

Since even I could resolve that with a simple change I can provide a patch 😏
diff --git a/lua/zenbones/specs/dark.lua b/lua/zenbones/specs/dark.lua
index 00d7dc2..226d66b 100644
--- a/lua/zenbones/specs/dark.lua
+++ b/lua/zenbones/specs/dark.lua
@@ -68,7 +68,7 @@ local function generate(p, opt)
      LineNr          { fg = p1.bg.li(opt.lighten_line_nr or 35), bg = opt.solid_line_nr and p1.bg.li(4) or "NONE" }, -- Line number for ":number" and ":#" commands, and when 'number' or 'relativenumber' option is set.
      SignColumn      { LineNr }, -- column where |signs| are displayed
      FoldColumn      { LineNr, gui = "bold" }, -- 'foldcolumn'
-     Folded          { bg = p1.bg.li(14), fg = p1.bg.li(64) }, -- line used for closed folds
+     Folded          { bg = not opt.transparent_background and p1.bg.li(14) or "NONE", fg = p1.bg.li(64) }, -- line used for closed folds
      CursorLineNr    { LineNr, fg = p.fg, gui = "bold" }, -- Like LineNr when 'cursorline' or 'relativenumber' is set for the cursor line.

      -- ModeMsg      { }, -- 'showmode' message (e.g., "-- INSERT -- ")
diff --git a/lua/zenbones/specs/light.lua b/lua/zenbones/specs/light.lua
index 8318ef0..579d8da 100644
--- a/lua/zenbones/specs/light.lua
+++ b/lua/zenbones/specs/light.lua
@@ -68,7 +68,7 @@ local function generate(p, opt)
      LineNr          { fg = p1.bg.da(opt.darken_line_nr or 33), bg = opt.solid_line_nr and p1.bg.da(4) or "NONE" }, -- Line number for ":number" and ":#" commands, and when 'number' or 'relativenumber' option is set.
      SignColumn      { LineNr }, -- column where |signs| are displayed
      FoldColumn      { LineNr, gui = "bold" }, -- 'foldcolumn'
-     Folded          { bg = p1.bg.da(16), fg = p1.bg.da(64) }, -- line used for closed folds
+     Folded          { bg = not opt.transparent_background and p1.bg.da(16) or "NONE", fg = p1.bg.da(64) }, -- line used for closed folds
      CursorLineNr    { LineNr, fg = p.fg, gui = "bold" }, -- Like LineNr when 'cursorline' or 'relativenumber' is set for the cursor line.

      -- ModeMsg      { }, -- 'showmode' message (e.g., "-- INSERT -- ")

Yes, please create a PR 😄
Would also be great to see a before/after comparison.

Thank you so much!!! 😃


ReferenceError: check/esjon is not defined when using meteor-client-side

When using meteor-client-side (not meteor's cli) I'm receiving the following error:
`ReferenceError: check/esjon is not defined`

It seems to be caused by angular2-meteor which is relying on packages like check and ejson to be exposed, but apparently it ain't so good to rely on that.
The issue occurs since the following commit:
https://github.com/Urigo/angular2-meteor/commit/f05a40b3251cd72f84c23a260174f9bff97faa2f
To fix it, I use the following work around:
import checkPack from 'meteor/check';
import ejsonPack from 'meteor/ejson';

Object.assign(window,
  checkPack,
  ejsonPack
);

Reverting this commit might solve this issue...

Removing meteor namespaces were necessary since they are not available on the server.
But I'll look in a way to use it as Package.ejson and Package.check

Unfortunately this problem is partially fixed, as for now I'm receiving the following error:
EXCEPTION: Error: Uncaught (in promise): EXCEPTION: Error in :0:0
ORIGINAL EXCEPTION: ReferenceError: Match is not defined
ORIGINAL STACKTRACE:
ReferenceError: Match is not defined
    at Function.DataObserver.pushCb (http://localhost:8100/build/js/app.bundle.js:93391:35)
    at ChatsPage.MeteorComponent._prepArgs (http://localhost:8100/build/js/app.bundle.js:93361:67)
    at ChatsPage.MeteorComponent.subscribe (http://localhost:8100/build/js/app.bundle.js:93298:24)
    at new ChatsPage (http://localhost:8100/build/js/app.bundle.js:118976:15)
    at DebugAppView._View_ChatsPage_Host0.createInternal (ChatsPage_Host.template.js:15:25)
    at DebugAppView.AppView.create (http://localhost:8100/build/js/app.bundle.js:15000:22)
    at DebugAppView.create (http://localhost:8100/build/js/app.bundle.js:15212:45)
    at ComponentFactory.create (http://localhost:8100/build/js/app.bundle.js:9882:37)
    at http://localhost:8100/build/js/app.bundle.js:58194:50
    at http://localhost:8100/build/js/app.bundle.js:98:195
ERROR CONTEXT:
[object Object]

Can you please re-open the issue?

@DAB0mB update the title of the issue


put method - how to ?

var input = document.getElementById("app-logo");
var file = input.files[0];
var fd = new FormData();
fd.append("file", file);
function responseHandler(e) {
console.log("readystate change", e);
}
var xhr = new XMLHttpRequest();
xhr.open("PUT", "http://localhost:8080/uploads/logo", true);
xhr.onreadystatechange = responseHandler;
xhr.send(fd);
I had tried above to create logo image inside uploads/logo - shows strict-origin-when-cross-origin

Is cors enabled?

Hi Ethan, I did not find any checkbox both from master or new-ui brach. please let know as It needs fix.

As far as I know, this isn't implemented yet (in the ui). You must enable it you must

Go to %USERPROFILE%\AppData\Roaming\Simple Web Server (in file explorer)
Find the server you want
add the option upload and set it to true
close and re-open the server

This is using the current master branch

The upload option is in the new UI. It's called "Allow file upload" in the Advanced Rules section.
@ethanaobrien We don't currently have an option for CORS. What did you mean when you asked if cors was enabled?

We don't currently have an option for CORS. What did you mean when you asked if cors was enabled?

Do we not? I have it implemented (line 16)

I don't think it came up in #1. What does it do?

@terreng
Oh, sorry. All cors does is set headers that allows websites to perform a request to it when it is outside of the domain (allows cross origin requests)

Does it just set Access-Control-Allow-Origin to * or what?

@terreng Correct

I've added the missing CORS option to the new-ui branch.
@WebScaffolder Please try enabling the new CORS option.

Hi, CORS issue I just found while testing with PUT method for uploading file but as 'Access-Control-Allow-Origin' is a response header I need to build by editing WSC handler.js where it is missing from new ui brach and after/before build it shows entry not found even after SPA enabled.

I've added the missing CORS option to the new-ui branch.
@WebScaffolder Please try enabling the new CORS option.

ok let me test.

Hi, I can't test CORS problem from my vue SPA for PUT method while upload as you need to fix 404 - Not Found entry not found: /login First. But I had a test using postman and works like a charm. So only problem now SPA shows Not Found entry. Please let me know for new ui.

could you post a picture of the code you are using to perform a PUT request. If I am correct (and I am (I wrote the handler)) the 404 - Not Found entry not found: /login is only given off by a GET request

Hi,
404 - Not Found entry not found: /login - yes you are right, this problem is with get request with github repository only. Soon i will send you a screenshot.
Note that relese source has no problem with spa but i can not find WSC folder there.

Note that relese source has no problem with spa but i can not find WSC folder there.

The version 1.0.0 release uses completely different internals to run the web server, and that's why there isn't a WSC folder.

Hi, Can you then just fix the old repository (where WSC present) for SPA support. Or update PUT method for version 1.0.0 so that as a developer we can use it for file uploading too.

Hi, Can you then just fix the old repository (where WSC present) for SPA support. Or update PUT method for version 1.0.0 so that as a developer we can use it for file uploading too.

We are not supporting the old version anymore. If there is an issue with the latest source, we'll fix it.

just update the spa support for latest release I mean for new ui branch and we developer can use it. you already fixed put cors issue. thanks for that. but we developer need SPA support as well.

Hi, any luck for improving new ui with SPA support as you packed WSC files  might for commercial release but without SPA it is not usable. please let me know.

@terreng Would it be possible that we need the other spa options? (optModRewriteNegate and optModRewriteTo)
line 270

@WebScaffolder Could you please clarify what exactly the issue is? Do you have a file at /index.html?

hi, here I have attached the problem from new ui....but it dose not happen by release version.

I think the problem is that we need to add the optModRewriteNegate option. Not sure what it does but I'm pretty sure this is it

optModRewriteNegate - no idea but I did not see any config present either new ui or release 1.0.1, just to clarify to find out the problem.

@WebScaffolder Would you mind opening google chrome and installing this chrome extension. Then could you set the mod rewrite options to whatever and send a screenshot when it works?

ok, sure

I think you are right. enabling negate it works with chrome

extension

Awesome, thank you. @terreng What do you think?

Can you just apply it for new ui branch. I will test and let you know if works.

@WebScaffolder I've fixed the code on the new ui branch. It should work now. Thanks for bringing this to our attention. Please let us know if you have any other trouble, otherwise I'll close this issue.
@ethanaobrien The way the regex is written, it should always be negated. I'm going to add an option to choose which page to rewrite to, called rewriteTo.

it doesn't work with new changes. and i tried to write below code but no luck -
if (this.app.opts.spa) {
var matches = request.uri.match('.\.[\d\w]+$')
var negate = true
if (matches === null && negate||
matches !== null && ! negate
) {
console.log("Mod rewrite rule matched", matches, '.\.[\d\w]+$', this.request.uri)
var handler = new DirectoryEntryHandler(this.fs, this.request)
handler.rewrite_to = '/index.html'
this.rewrite_to = handler.rewrite_to
}
}

That's the old code. Please use the latest.

hi, I tried below code too but no luck -
if (this.app.opts.spa) {
if (!this.request.uri.match(/.*.[\d\w]+$/)) {
console.log("Single page rewrite rule matched", this.request.uri);
this.rewrite_to = this.app.opts.rewriteTo || "/index.html";
}
}

Please confirm that you are no longer running the chrome extension web server. I tested the code and it should work. If not, could you please describe the issue?

Hi, it works by your given instruction. if any other issue found I will let you know. thanks for your effort and make it more nicer than old extension.

Great, I'm glad you figured it out! Thanks again for letting us know about the issue. We'll be releasing the new version of this app soon, so look out for that. Let us know if you find anything else, thanks!


router filter: implement hedge_on_per_try_timeout.

Implements the hedge_on_per_try_timeout option in HedgePolicy config
which instructs the router filter to not cancel requests that hit the
per try timeout before sending a retry. This means the router must be
able to manage multiple simultaneous upstream requests and handle
deciding which response "wins" and canceling any other in flight
requests.
Finishes #5841.
Description: Implements request hedging (racing) in response to a per try timeout in the router filter.
Risk Level: High
Testing: Unit/Integration tests
Docs Changes: Added a request hedging section to the HTTP routing arch overview in addition to proto-level docs.
Release Notes: updated
Fixes #5841

I'd like some integration tests for this as well, but will probably wait until Matt is done with the one he's writing so I can use it as a guide.

working through the build failures and will add more tests today

This is no longer WIP and ready for comment

tried this build out again, and hit an assert in addDownstreamWatermarkCallbacks because it gets called multiple times. will need to take a look at that

@mattklein123 are you able to take a look to make sure you like the direction of this change? If we agree on the behavior here I can work on adding integration tests

@mpuncel sorry we raced comments. At a high level it looks sane to me, but it's a super complicated change and extremely high risk so I need time to look through. I think it's fine to start with integration tests, since I doubt the user visible behavior will change? WDYT?

sounds good to me! will address your comments from today and work on integration tests

Gonna mark this as no stalebot since Mike is out this and next week

Please merge master and move release note to 1.11.0 and let's get this reviewed and merged!
/wait

merged master and moved release note, will start working through Alyssa's feedback

added some integration tests, and a x-envoy-hedge-on-per-try-timeout header (mostly to make the integration test easier to set up but is also a useful feature).
I expect the test I added to fail until https://github.com/envoyproxy/envoy/pull/6526 is merged in because the last retry is the one that succeeds, but currently we see a retry overflow value.

I think this is in a pretty good state. I didn't write any unit tests for the conn manager changes because I think the integration tests cover it pretty well. I have cases there for large request with small upstream buffer size, and large response with small downstream buffer size. There is no retrying if the full request can't be buffered so that means there is no hedging either.

/retest

/retest

/retest

/retest

/retest

oops github was telling me my comments were failing but they were actually going through

Will give this a detailed review once we merge the breakout PR.
/wait

unfortunately I can't seem to reproduce that ipv6 flake anywhere but on circleci so i'm not sure what's going on

I merged master and pushed that commit up, but I still need to do some more work on the watermark stuff, so no need to review yet.

I definitely think we should sort out the watermark issues (as a separate PR!) and land them before we unhide any hedge config.  Happy to chat about them with both of you now that I'm back.

@alyssawilk I think the PR is in a good spot with the watermark stuff (although I believe i'm still missing some of the regression tests you asked for w.r.t. the for loop over the list of registered callbacks).
There are a couple of observations that made things simpler:

for the "request too big" case, there should only be one in flight upstream request. This assumption holds true because a) we can only have multiple upstream requests if there was a per try timeout and b) (after https://github.com/envoyproxy/envoy/pull/6643/files) a per try timeout can only occur after we've seen the whole downstream request. That means I can comfortably pause downstream reads and know I'm not slowing down other requests.

for the "response too big" case, I'm either actively writing data downstream after having selected a "winning" upstream request and resetting the others, or there is no "winning" upstream request at all and i'm not writing data downstream. In either case it's okay for me to read disable upstream requests and wait for a below watermark callback.

you're totally right that #6643 helps you out here, insofar as you simply won't retry or hedge at all in cases where the backend is behaving badly and doesn't drain the tcp socket buffer.  It's kind of a bummer, because it means that both hedging and retries may simply not work for large requests and new connections because if the TCP window is not warmed, the buffer will fill, reading will stop, and the data will simply never be read.  Looks like setUpRetry already fast failed in that case so it's no worse, it just means there' a whole class of failures we don't retry or hedge for.  The other thing required for this to work was correct unwinding, which landed in  #6578.  Now the codec will which will readEnable for subsequent HTTP/1 requests after the hedge is successful. For H2 that's already sorted out onStreamClose.

for 2) I think you're good given you reset the other upstreams once you have headers. I think you can update hedge docs a bit to make it more clear (rather than good response, good response headers etc.) but watermarks shouldn't be a problem here.  yay!

looks like my PR made cluster memory size go above the test threshold?
[ RUN      ] IpVersions/ClusterMemoryTestRunner.MemoryLargeClusterSizeWithStats/IPv4
test/integration/stats_integration_test.cc:199: Failure
Expected: (m_per_cluster) < (59600), actual: 59848 vs 59600

Could that be because of the stat I added?

yeah, sounds plausible.  if I recall correctly @cmluciano added this test but @jmarantz put a bunch of work in to making the stats scale better memory wise.  Unfortunately I don't think we have a great way of handling stats for off-by-default features, where we want the stat if it's configured and we don't want to use memory for all the stats for all the features for all the clusters.  Maybe Josh has thoughts here?

@mpuncel @alyssawilk sorry catching up. Is this ready to review again or are we still waiting on changes?

There's at least one other case I reviewed (forgot exactly where) of having some extra stats enabled by flag. In that case the stats made sense by default, but you only get them if you ask for them by turning on a feature.
In this case it seems easy enough to refrain from initializing a stat when the  feature it's measuring is turned off.

BTW the stat memory-scaling work is still ongoing; the biggest gains are pending (including one waiting for review in #6161).

@mattklein123 it's ready for a look (aside from this cluster memory issue)!

I'm not really sure what my options here are for selectively enabling the stat. Any HTTP request could enable the feature with headers.

@mattklein123 @alyssawilk ping for review and advice on what to do about cluster memory. One option would be to remove the new stat for now and deal with it in another PR

@mattklein123 @alyssawilk ping for review and advice on what to do about cluster memory. One option would be to remove the new stat for now and deal with it in another PR

If you want to add the stats in a follow up that's fine with me, but I do think we should add them (and I mentioned another one we might want to add).

RE "I'm not really sure what my options here are for selectively enabling the stat. Any HTTP request could enable the feature with headers." -- one option is that you could only create that stat when someone enables the feature. Another option is to determine that the stat/feature is sufficiently important that we should always have the stat for it even if the feature is disabled. I'm OK with any of that; my position is that I just want to be conscious of the memory consumed.

RE "I'm not really sure what my options here are for selectively enabling the stat. Any HTTP request could enable the feature with headers." -- one option is that you could only create that stat when someone enables the feature. Another option is to determine that the stat/feature is sufficiently important that we should always have the stat for it even if the feature is disabled. I'm OK with any of that; my position is that I just want to be conscious of the memory consumed.

It might be sufficient in this case to make it a router stat vs. cluster stat. The other option is to only allow header override if the feature is configured in some way, then create the stats in a new sub-scope. Personally I think I prefer the latter option?

pushed a new commit addressing most of the last round of feedback. Will mark the relevant items resolved, there are a couple that will require more thought on my end

FYI I'm going to do a wider internal release of this branch to Square today

I am going to do another pass to make sure the coverage is good, probably I can test more permutations of bad headers, resets, and per try timeouts in the integration tests

By the way this is what the upstream access logs look like when making a call that takes 2s in the server with a per try timeout of 100ms:
[2019-04-26T19:27:16.298Z] "GET /sleep/2s HTTP/2" 200 -/- UT 0 0 2002 2002 "curl/7.29.0" "3280f316-acc0-43bc-abff-7895bda99574" "envoy-test.slug.gns.square" "-" "<IP_ADDRESS>:32351"
[2019-04-26T19:27:16.619Z] "GET /sleep/2s HTTP/2" 0 -/- UT 0 0 1681 - "curl/7.29.0" "3280f316-acc0-43bc-abff-7895bda99574" "envoy-test.slug.gns.square" "-" "<IP_ADDRESS>:32351"
[2019-04-26T19:27:16.957Z] "GET /sleep/2s HTTP/2" 0 -/- UT 0 0 1344 - "curl/7.29.0" "3280f316-acc0-43bc-abff-7895bda99574" "envoy-test.slug.gns.square" "-" "<IP_ADDRESS>:32351"
[2019-04-26T19:27:17.331Z] "GET /sleep/2s HTTP/2" 0 -/- UT 0 0 969 - "curl/7.29.0" "3280f316-acc0-43bc-abff-7895bda99574" "envoy-test.slug.gns.square" "-" "<IP_ADDRESS>:32351"
[2019-04-26T19:27:17.711Z] "GET /sleep/2s HTTP/2" 0 -/- UT 0 0 590 - "curl/7.29.0" "3280f316-acc0-43bc-abff-7895bda99574" "envoy-test.slug.gns.square" "-" "<IP_ADDRESS>:32351"

I've removed the hedging stats for now because I'm wary of adding to this PR. Will create an issue to follow up and add the stats in a way that doesn't increase cluster memory without being enabled explicitly.

@mpuncel what's the status here, are you still working on changes or ready for another pass?

@mattklein123 I'm ready for another pass. Since you last looked the big changes are:

removed the stats to avoid the memory issues. I looked into adding a sub scope but it seems complex enough that I'd rather do it in a followup PR if that's alright with you.
grpc-timeout and x-envoy-expected now get set to the global timeout instead of per try timeout when hedge_on_per_try_timeout is enabled. Without this change upstream servers might time out and negate the feature.
Added more cases to the integration tests, now we try 1st request succeeding and 2nd request succeeding for each test case.
there's now a cluster option allow_request_hedging that needs to be on for hedging to be enabled. Without it the route, virtual host, and header configs are ignored. This is mostly in preparation for the stats followup later to avoid backwards compatibility issues where someone can enable hedging without stats

@mpuncel can you merge master to pick up various CI fixes and I will take a final pass? Thank you.
/wait

just merged master again and addressed PR feedback. I also got a screenshot of tracing showing hedging working for a real gRPC request at Square.
It shows an app calling itself with a 100ms per try timeout and the server takes longer than that. It looks a bit odd because the 2 spans have the same parent but don't start at the same time, and then that one server span is long because cancellation wasn't handled (it just does a sleep).

Looks like CI issues. Note also that docs is broken on master right now so watch for a fix for that.
/wait


WTD: projected volumes

Add projected volume type to the volumes concept page:
I've kept this light since I don't fully understand the feature,
but since this is a distinct volume type, it seems helpful to
list it here. Also, this lets us keep concept details out of the
related task, and it gives us a good place to link to the origin
design document.

Apply template to projected volume task:
This involves incorporating an actual scenario and series
of steps. I don't think the configuration is correct,
but I'm hoping the general trajectory is moving in the right
direction.

This change is

I'll have an engineer to a tech review.

@saad-ali, just a few extra details:

The pod config I used is a combination of the one in the basic volume tutorial here and configs in the projected volume design doc here. I doubt this is the right approach, but I just wanted to get a basic example put together.

Also, given that the example projects secrets, I'm not sure how realistic it is to tell users to exec into the container and list the contents of the projected-volume directory.

@jpeeler and/or @pmorie as well, since you are listed as assignees in the projected volumes file.

@jseldess @jpeeler Can you please let me know when this document is complete so I can bring it downstream into the OpenShift docs.

Sure, @mburke5678. Hoping to get back to this PR soon.

@jpeeler, @chenopis, I think I've address all issues, but please let me know. Unfortunately, when fetching updates from upstream, I seem to have added all those commits to this PR. Happy to clean things up, if you could give me some guidance.

Ugh. I didn't set myself up properly. Working off of my forked master branch. I'm going to close this PR and open a new PR with just the relevant updates. Apologies for my git stumbles.

FWIW, you can utilize force push (via git push --force-with-lease) to not have to open up a new PR.


Check Plagiarism Similiarity Range [0-100]

Is your feature request related to a problem?

In the past we encountered plagiarism with a score less then 50%.
Therefore, it would be good, if the range of the "Similarity Threshold in %" could be set to a value within [0, 100] instead of [50,100] for the Check Plagiarism View.
Describe the solution you'd like

The value for "Similarity Threshold in %" should be configurable as [0, 100].

Yes, we can do that. We have limited the results to 500, so the previous performance bottleneck (that led to the decision to 50%) should not be problematic any more


How to match /{suffix...} in go-zero?

Can you describe it in more detail?

Can you describe it in more detail?

Issue Description
I am currently developing a gateway application that requires dynamic routing based on the suffix of the URL path. I am considering migrating our existing system to the go-zero framework to leverage its built-in features such as load balancing and distributed tracing. However, I’ve encountered a limitation in the go-zero routing system that may impede this migration.
Expected Behavior
I would like to confirm whether the go-zero routing system supports a pattern that allows for wildcard suffix matching, similar to /service/{suffix...}. This pattern should match any request path that starts with /service/ and is followed by any suffix. The gateway should be capable of capturing the entire suffix and routing it to the appropriate downstream service.
Use Case
The gateway should be able to route requests of a project to different module and API versions based on the suffix. For example, /awesome/api/v1/resource/... would direct requests to module-api, version-1 of the project-awesome.
The gateway is intended to aggregate multiple microservices behind a single endpoint, with the suffix determining which specific service the request should be routed to, and apply queuing, lots of monitoring and functions.

func TestSearchTreeMatch(t *testing.T) { 	tree := search.NewTree() 	handler := func() { 		print("match!") 	} 	err := tree.Add("/a/:path", handler) 	if err != nil { 		t.Error(err) 	} 	res, matched := tree.Search("/a/b") 	if !matched { 		t.Error("not matched /a/b") 	} 	assert.Equal(t, res.Params["path"], "b") 	res, matched = tree.Search("/a/c/d/e") // 不能匹配 can not match 	if !matched { 		t.Error("not matched /a/c/d/e") 	} 	assert.Equal(t, res.Params["path"], "c/d/e")  	res, matched = tree.Search("/a") 	if matched { 		t.Error("matched /a") 	} else { 		t.Log("not matched /a") 	} 	res, matched = tree.Search("/a/") 	if matched { 		t.Log("matched /a/") 	} 	res, matched = tree.Search("/a/c?q=1") 	if !matched { 		t.Error("not matched /a/c?q=1") 	} 	assert.Equal(t, res.Params["path"], "c?q=1") }
here is it


Full support for custom timeout in testcloud reboot

Currently setting the maximum time for a reboot in the testcloud plugin works only when a custom reboot command is used, for example like this:
tmt-reboot -t 60 -c reboot

It would be nice if we support timeout without the custom command as well:
tmt-reboot -t 60

Related code is here:
https://github.com/teemtee/tmt/blob/6e4742b93a2afbfcee7bb237af2251665f828884/tmt/steps/provision/testcloud.py#L1078-L1083
Ideally testcloud would provide reboot parameter in instance.reboot() so that we can hand over the value.

@frantisekz, what do you think?


created new term, fixed some logical definitions Fixes #14391

Hi Chris,
I dont know that we have a design pattern for this one. All I did was to standardize what we have around that branch:
The parent "cellular detoxification" has EA:
detoxification
and ('occurs in' some cell)
Some children such as 'cellular detoxification of cadmium ion' had
detoxification
and ('occurs in' some cell)
and ('has input' some 'cadmium cation')
while others such as 'cellular detoxification of methylglyoxal'
had
'cellular detoxification'
and ('has input' some methylglyoxal)
Since that gave the same inferences (according to protégé) I made the latter the same as the former.
The other two terms, 'nitrogen compound' and the new 'fluoride', are not in the ChEBI imports. Should this be part of the same ticket or can the logical definitions be created separately?
Thanks,
Pascale

Hi Chris,
Let me know if it's OK to merge.
Thanks, Pascale

OK to merge yes but we need to address the cellular-X DP soon


accept hashes when pulling from containerd

When pulling packages from a registry, it is standard to check the hashes when they are pulled to make sure they are valid and unchanged. The hashing is expensive but necessary.
When pulling from a local containerd, it is an unnecessary expensive process.
This tells the puller - when creating VM volumes from locally stored VM images, or when creating a new baseos image on a device - to just trust containerd's hashes and not recalculate them. This cuts the processing time for larger images by a factor of 4-6 or more.

@rvs do you understand the error from eden? It is here, with the relevant output:
        # Send command to update eveimage (4.696s)
        > message 'EVE update request'
        message: EVE update request
        > eden -t 10m controller edge-node eveimage-update file:///home/runner/work/eve/eve/dist/default-eve/rootfs-5.12.4-kvm-amd64.squashfs -m adam://
        [stdout]
        time="2020-11-09T21:47:37Z" level=info msg="new datastore created a7f27bdf-a63e-4c40-be5e-115275f5f9d6"
        time="2020-11-09T21:47:37Z" level=info msg="Start uploading into eserver of file:///home/runner/work/eve/eve/dist/default-eve/rootfs-5.12.4-kvm-amd64.squashfs"
        time="2020-11-09T21:47:41Z" level=info msg="Waiting for SHA256 calculation"
        time="2020-11-09T21:47:41Z" level=info msg="Image uploaded with size 234 MB and sha256 d2c9c683bfa946486b489cd0fccd60ac8db1ad3c04bc70d37ef89954dd551497"
        time="2020-11-09T21:47:41Z" level=info msg="new image created dc77a107-c8a1-47ba-9955-d1e0bba6ab4f"
        time="2020-11-09T21:47:41Z" level=info msg="new base os created dc77a107-c8a1-47ba-9955-d1e0bba6ab4f"
        time="2020-11-09T21:47:42Z" level=info msg="config updated /home/runner/.eden/devUUID-5961b1b8-11aa-4817-8069-4678f6d4cbf6.json"
        time="2020-11-09T21:47:42Z" level=info msg="state updated /home/runner/.eden/state-ed953dbb-13d2-48dc-874e-8be6f1325046.yml"
        # Check stderr, it must be empty (0.000s)
        > ! stderr .
        # Run monitoring of Info messages to obtain info with PartitionState inprogress or active and previously defined ShortVersion (1201.206s)
        > message 'Waiting for EVE update...'
        message: Waiting for EVE update...
        > test eden.lim.test -test.v -timewait 1200 -test.run TestInfo -out InfoContent.dinfo.SwList[0].ShortVersion 'InfoContent.dinfo.SwList[0].PartitionState:inprogress|active InfoContent.dinfo.SwList[0].ShortVersion:5.12.4-kvm-amd64'
        testProg: /home/runner/work/eve/eve/dist/bin/eden.lim.test
        [stdout]
        Log/Info/Metric Test
        === RUN   TestInfo
            lim_test.go:184: Wait for info of default number=1 timewait=1200
            testContext.go:240: WaitForProc terminated by timeout 20m0s
        --- FAIL: TestInfo (1200.00s)
        FAIL
        [stderr]
        time="2020-11-09T21:47:43Z" level=info msg="config updated /home/runner/.eden/devUUID-5961b1b8-11aa-4817-8069-4678f6d4cbf6.json"
        time="2020-11-09T21:47:43Z" level=info msg="state updated /home/runner/.eden/state-ed953dbb-13d2-48dc-874e-8be6f1325046.yml"
        [exit status 1]
        FAIL: ../update_eve_image/testdata/update_eve_image.txt:40: unexpected command failure

--- FAIL: TestEdenScripts (0.00s)
    --- FAIL: TestEdenScripts/update_eve_image (1230.74s)

I just pushed a minor change. There was a case where you could get a zero-size buffer passed to io.CopyBuffer, so I made sure that doesn't happen. Let's let the tests run, and I am testing that specific case as well.

Nice! All CI clean.


Won't you upload it on F-Droid?

Yes? No? Shrug?

No answer, @franciscofranco? Also, code for the last version is missing here. Those of us who don't use Play Store would certainly appreciate getting the app via F-Droid – or at least finding (up-to-date) code & APK here. Any chance?


fix: adds aria label

Overview
Adds aria-stark-labels to text labels when now text is passed.
PR Checklist

[?] I have run this code to verify it works

📬Published Alpha Packages:@codecademy/gamut-labs@1.7.27-alpha.152563.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://5ee93d7105dc830234594d9e--codecademy-styleguide.netlify.appDeploy Logs

Closing for housekeeping now that #844 is in 👍


[Question]. Reset database from asset.

I have a database that is loaded from the assets folder on first run and injected to the rest of the app with provider. I want to enable the reset by discarding the in-memory database and reloading from the assets but I can't wrap my head around it. Tried recreating the class and rewriting the bytes but got the warning that I'm creating the class twice.
class AppDataBaseModel with ChangeNotifier {
  AppDatabase appDatabase = AppDatabase();

  void resetApp() async {
    await appDatabase.close().then((value) {
      log("message");
      appDatabase = null;
      appDatabase = AppDatabase.resetDatabase();
      notifyListeners();
      return;
    });
  }
}

I'm not sure what resetDatabase is doing, but imagine that you could simply delete the local database file and re-open. Are you using a FlutterQueryExecutor or VmDatabase?

I'm not sure what resetDatabase is doing, but imagine that you could simply delete the local database file and re-open. Are you using a FlutterQueryExecutor or VmDatabase?

I'm using a VmDatabase
this is what resetDatabase is doing
LazyDatabase _resetDatabase() {
  // after
  return LazyDatabase(() async {
    var databasesPath = await getApplicationDocumentsDirectory();
    var path = join(databasesPath.path, 'confession.db');

    final file = File(path);

    // copy the file from an asset, or network, or any other source
    // Should happen only the first time you launch your application
    print('Creating new copy from asset');

    // Make sure the parent directory exists
    try {
      await Directory(dirname(path)).create(recursive: true);
    } catch (_) {
      log("exception");
    }

    // Copy from asset
    var data =
        await rootBundle.load(join('assets', 'database', 'confession.db'));
    List<int> bytes =
        data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);

    await file.delete().then((value) => file.writeAsBytes(bytes, flush: true));
    // Write and flush the bytes written
    //await file.writeAsBytes(bytes, flush: true);

    return VmDatabase(file);
  });
}

That code looks alright to me. Are you using the result of _resetDatabase inside the database constructor of the second instance? I just saw the public resetDatabase method from your original snippet.
The warning should only appear when using multiple database instances at the same time. Since you're calling .close before creating the new database, that shouldn't be an issue. Do you see any way that multiple instances of AppDatabase could be active at the same time in your code?


mixin: alerts and rules using same groupname

Thanos, Prometheus and Golang version used:
mixin folder coming from the master branch.
What happened:
Prometheus is complaining thanos-query.rules, thanos-receive.rules, thanos-store.rules and thanos-bucket-replicate.rules group names are repeated in the same file.
What you expected to happen:
That alerts and rules mixins can be imported with prometheus-operator/kube-prometheus.
How to reproduce it (as minimally and precisely as possible):
Import thanos rules and alerts mixins with prometheus-operator/kube-prometheus deployment.
jsonnetfile.json

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/kube-prometheus.git",
          "subdir": "jsonnet/kube-prometheus"
        }
      },
      "version": "master"
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/thanos-io/thanos.git",
          "subdir": "mixin"
        }
      },
      "version": "master",
      "name": "thanos-mixin"
    }
  ],
  "legacyImports": true
}

kube-prometheus.jsonnet

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
  // Uncomment the following imports to enable its patches
  (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  // prometheusAlertsFilter + prometheusAlertsUpdate +
  {
    _config+:: {
      namespace: 'monitoring',

      versions+:: {
        alertmanager: "v0.21.0",
        nodeExporter: "v1.0.1",
        kubeStateMetrics: "1.9.7",
        prometheusOperator: "v0.42.1",
        prometheus: "v2.22.0",
        grafana: '7.2.1',
        thanos: 'v0.16.0-rc.1',
      },
      ...
    },
    ...
    prometheusRules+::
      (import 'thanos-mixin/rules.jsonnet') +
      {
        // groups+: [],
      },
    prometheusAlerts+::
      (import 'thanos-mixin/alerts.jsonnet') +
      {
        // cat existingrule.yaml | gojsontoyaml -yamltojson > existingrule.json
        // groups+: (import 'existingrule.json').groups,
      },
  };

Full logs to relevant components:
Logs

level=error ts=2020-10-21T10:50:54.930Z caller=manager.go:946 component="rule manager" msg="loading groups failed" err="/etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-prometheus-k8s-rules.yaml: 2142:3: groupname: \"thanos-query.rules\" is repeated in the same file"
level=error ts=2020-10-21T10:50:54.930Z caller=manager.go:946 component="rule manager" msg="loading groups failed" err="/etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-prometheus-k8s-rules.yaml: 2236:3: groupname: \"thanos-receive.rules\" is repeated in the same file"
level=error ts=2020-10-21T10:50:54.930Z caller=manager.go:946 component="rule manager" msg="loading groups failed" err="/etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-prometheus-k8s-rules.yaml: 2352:3: groupname: \"thanos-store.rules\" is repeated in the same file"
level=error ts=2020-10-21T10:50:54.930Z caller=manager.go:946 component="rule manager" msg="loading groups failed" err="/etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-prometheus-k8s-rules.yaml: 2592:3: groupname: \"thanos-bucket-replicate.rules\" is repeated in the same file"
level=error ts=2020-10-21T10:50:54.930Z caller=main.go:881 msg="Failed to apply configuration" err="error loading rules, previous rule set restored"

Anything else we need to know:

@metalmatze @paulfantom Any ideas here?

Not really sure what's happening here. Maybe the mixin is somehow deployed twice?

It's not deployed twice, I'm only importing thanos-mixin/rules.jsonnet in prometheusRules and thanos-mixin/alerts.jsonnet in prometheusAlerts.
Prometheus will process those alerts/rules as separated groups, the issue is that some of them are using the same groupname, which cannot be used more than once as said by the Prometheus error is saying.
For example:

alerts/store.libsonnet
mixin/rules/store.libsonnet

It should be an easy fix. Help wanted!


Replace exhaustive match with a lookup table in hex_escape

Replace the 250+ line exhaustive match expression in Literal::hex_escape with an exhaustive [&str; 256] const lookup table. This should result in similar codegen, but is much easier to read. This also removes an allow(clippy::to_many_lines).

I first discovered this approach by peeking at the bstr source for ByteSlice::find_not_byteset and have used this same approach in boba in https://github.com/artichoke/boba/pull/138.


pip Pyfhel went wrong

python 3.11
OS:window 11
I have already install Microsoft C++(does it need to add to path)
I got below when i pip install Pyfthe
C:\Users\chieh>pip install Pyfhel
Collecting Pyfhel
Downloading Pyfhel-3.4.2.tar.gz (1.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 5.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [61 lines of output]
Compiling Pyfhel\Pyfhel.pyx because it changed.
Compiling Pyfhel\PyCtxt.pyx because it changed.
Compiling Pyfhel\PyPtxt.pyx because it changed.
Compiling Pyfhel\PyPoly.pyx because it changed.
[1/4] Cythonizing Pyfhel\PyCtxt.pyx
[2/4] Cythonizing Pyfhel\PyPoly.pyx
[3/4] Cythonizing Pyfhel\PyPtxt.pyx
[4/4] Cythonizing Pyfhel\Pyfhel.pyx
running egg_info
writing Pyfhel.egg-info\PKG-INFO
writing dependency_links to Pyfhel.egg-info\dependency_links.txt
writing requirements to Pyfhel.egg-info\requires.txt
writing top-level names to Pyfhel.egg-info\top_level.txt
cimporting numpy version '1.26.4'
reading manifest file 'Pyfhel.egg-info\SOURCES.txt'
Traceback (most recent call last):
File "C:\Users\chieh\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\chieh\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Programs\Python\Python311\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\build_meta.py", line 177, in get_requires_for_build_wheel
return self._get_build_requires(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\build_meta.py", line 159, in get_build_requires
self.run_setup()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
exec(compile(code, file, 'exec'), locals())
File "setup.py", line 666, in
setup(
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_init.py", line 155, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_distutils\core.py", line 148, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands
dist.run_commands()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands
self.run_command(cmd)
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command
cmd_obj.run()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 298, in run
self.find_sources()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 305, in find_sources
mm.run()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 540, in run
self.add_defaults()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 584, in add_defaults
self.read_manifest()
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\sdist.py", line 195, in read_manifest
self.filelist.append(line)
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools\command\egg_info.py", line 482, in append
path = convert_path(item)
^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-vtqepg6i\overlay\Lib\site-packages\setuptools_distutils\util.py", line 128, in convert_path
raise ValueError("path '%s' cannot be absolute" % pathname)
ValueError: path '/Users/alberto.ibarrondo/Technocracy/workspace/Pyfhel/Pyfhel/Afhel/Afseal.cpp' cannot be absolute
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

I upgrade my python to 3.12
but still get the following problem
C:\Users\chieh>pip install pyfhel
Collecting pyfhel
Using cached Pyfhel-3.4.2.tar.gz (1.0 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [33 lines of output]
Traceback (most recent call last):
File "C:\Users\chieh\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\chieh\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 112, in get_requires_for_build_wheel
backend = _build_backend()
^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chieh\AppData\Local\Programs\Python\Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in load_unlocked
File "", line 995, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-j9_n7y75\overlay\Lib\site-packages\setuptools_init.py", line 16, in
import setuptools.version
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-j9_n7y75\overlay\Lib\site-packages\setuptools\version.py", line 1, in
import pkg_resources
File "C:\Users\chieh\AppData\Local\Temp\pip-build-env-j9_n7y75\overlay\Lib\site-packages\pkg_resources_init.py", line 2178, in
register_finder(pkgutil.ImpImporter, find_on_path)
^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

I solve the problem by using GIT BASH, run:
git clone --recursive https://github.com/ibarrond/Pyfhel.git
pip install .

Hope this can help you.

From the looks of it, they stored a dependency in their own user directory and then told the package to look in THEIR user directory for the dependency. How did that pass the unit tests?
all they needed to do is say the dependency exits in ./someDirectory instead of /Users/alberto,ibarrondo/[...]


Add mixed domain host excludes for WF 10.0 and 10.1

Currently an experiment to see how the testsuite reacts.

retest this please

I removed the 'hold' label as I see no negative effects in the testsuite.

Why is this needed? mixed-domain tests only check EAP releases not intermittent WildFly ones....

This will be needed for just using core & old slaves, when we figure out how to add elytron to the core feature pack.


[bug] Extremely long compilation times of the final binary crate

Describe the bug
While compiling a Tauri app, a lot of time[^1] is spent compiling the final binary. This also applies for hot/incremental builds using yarn tauri build (including ones with no real changes to the code, like white space changes), with several users reporting compilation times also taking up to several minutes.
When profiling rustc (using RUSTC_BOOTSTRAP=1 cargo rustc -p theseus_gui -- -Z self-profile -Z self-profile-events=default,args), I noticed that most of the compilation time is spend compiling #[tauri::command] macros, in codegen_select_candidate (~52% of compile time). Most of the queries seem related to tauri::InvokeResolver::respond_async_serialized and tokio::runtime::task::harness.
More notably, almost half of the compilation time is spent compiling two small functions in the (barely 19 LOC long!) pack.rs file.
Attached is a Chrome Profiler file, converted from the original ~600MB .mm_profdata file using crox, with the minimum duration of 1ms: chrome_profiler-gui.zip[^2]
[^1]: Up to 10 minutes when compiling the binary itself, with dependencies precompiled to reduce noise.
[^2]: Can recommend ui.perfetto.dev for inspecting the file.
Reproduction
I could use some help isolating the issue to a minimal reproduction example, primarily because I'm not entirely sure what is the real cause of the compilation issues. In the meantime, I invite you to take a look at pack.rs in the Theseus project.
Expected behavior
No response
Platform and versions
Environment
  › OS: Red Hat Enterprise Linux 9.2.0 X64 (EDITOR'S NOTE: It's actually AlmaLinux... which is basically RHEL. Reproducible by other users on different platforms.)
  › Node.js: 20.2.0
  › npm: 9.6.7
  › pnpm: Not installed!
  › yarn: 1.22.19
  › rustup: 1.26.0
  › rustc: 1.69.0
  › cargo: 1.69.0
  › Rust toolchain: stable-x86_64-unknown-linux-gnu

Packages
  › @tauri-apps/cli [NPM]: 1.2.3
  › @tauri-apps/api [NPM]: 1.2.0
  › tauri [RUST]: 1.2.4,
  › tauri-build [RUST]: 1.2.1,
  › tao [RUST]: 0.15.8,
  › wry [RUST]: 0.23.4,

App
  › build-type: bundle
  › CSP: default-src 'self'; connect-src https://modrinth.com https://*.modrinth.com; font-src https://cdn-raw.modrinth.com/fonts/inter/; img-src tauri: https: data: blob: 'unsafe-inline' asset: https://asset.localhost
  › distDir: ../dist
  › devPath: http://localhost:1420/
  › framework: Vue.js
  › bundler: Rollup

App directory structure
  ├─ .vscode
  ├─ src-tauri
  ├─ src
  ├─ node_modules
  └─ dist

Stack trace
Example `codegen_select_candidate` argument (took 3s 931ms 854us):

(ParamEnv { caller_bounds: [], reveal: All, constness: NotConst }, Binder(<std::panic::AssertUnwindSafe<[closure@tokio::runtime::task::harness::poll_future<std::pin::Pin<std::boxed::Box<[async block@tauri::InvokeResolver::respond_async_serialized<[async block@theseus_gui/src-tauri/src/api/pack.rs:5:1: 5:18]>::{closure#0}]>>, std::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::{closure#1}]> as std::ops::FnOnce<()>>, []))

Additional context
See: https://discord.com/channels/616186924390023171/1110350612345917532 (links to Tauri Discord server)
Credits to @Geometrically for the initial investigation.

cc @chippers

https://github.com/thesuzerain/tauri-compilation-slow-example
I have a more specific recreation of this. It should be easier to recreate


Training Time

Hi Team,
Thank you for sharing the great work. I wanted to ask if it is possible for you to provide the training time for each of the task in the paper, on a particular dataset? For example, how much time the pretraining will take on 32 A100 40GB GPUs? How about the other down stream tasks? Thank you and looking forward to your response.

Hi @mmaaz60 ,
Thank you for your interests in our work.
For the downstream tasks, we've attached the training log at the MODEL_ZOO from which you can probably read out the training time.
For the pre-training task, it takes ~5hr/epoch to train a dual-encoder baseline (i.e. ground-truth narrations only). We train it for 5 epochs. For LaViLa-style Dual-Encoder, the training time is 2x since we alternately sample text between Narrator and Rephraser.
Note that the hardwares where we perform pre-training/fine-tuning are 32x V100 32GB GPUs (unless otherwise specified). I haven't experimented with A100 but from some external comparison, the speedup is approximately 2~3x. You might reduce your ETA accordingly.
Hope this answers your question.
Best,
Yue

Closing this issue due to inactivity... Feel free to re-open the issue if needed.


Constant Error: Cannot find module 'typescript'

Environment
Operating System: Linux

Node Version:     v21.6.1
Nuxt Version:     3.11.2
CLI Version:      3.11.1
Nitro Version:    2.9.6
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      devtools, modules
Runtime Modules<EMAIL_ADDRESS>Build Modules:    -

Package
@nuxt/eslint
Reproduction
Fresh install of nuxt3 and nuxt-eslint module, just followed the tutorial
Describe the bug
Once i finished the tutorial on how to use:
https://eslint.nuxt.com/packages/module#quick-setup
with this command:
npx nuxi module add eslint
and try to yarn lint
i got this error:
Oops! Something went wrong! :(
ESLint: 9.3.0
Error: Cannot find module 'typescript'
Require stack:
Additional context
No response
Logs
No response

Just had the same issue.
Not sure this is the best approach, but I installed typescript as a dev dependency then it worked.

Hi @speechless2 , @mstflotfy !
I think this problem is specific to yarn, but what do you think?
I conducted experiments with pnpm and npm, but I could not confirm any similar problems.
However, a similar problem was confirmed when performing operational experiments with yarn.
Therefore, I don't think it's a yarn issue rather than a Nuxt ESLint issue.

Hey @shinGangan,
I just installed the module using: npx nuxi module add eslint in a fresh bunand npm projects, and it worked without issues. It does seem that this problem is specific to yarn.

Hi @mstflotfy , thanks for your comment!
I'm glad the experiment results were the same. 💚
If there is a problem on yarn side, it may be okay to close this issue. 🤔

Hi @speechless2 , has the problem been resolved?
Or is it still happening?

Still happening

What's the yarn version you are having? Could anyone share a minimal reproduction with the yarn version and lockfile? Thanks

I have used yarn 4.3.1. Here's steps to reproduce:

Node 22.1.0
npx nuxi@latest init eslint-issue
cd eslint-issue
rm -rf node_modules
rm -yarn.lock
yarn set version berry
yarn config set nodeLinker node-modules
npx nuxi module add eslint
I added a couple more ESLint related dependencies:

Package.json
{
  "name": "nuxt-app",
  "private": true,
  "type": "module",
  "scripts": {
    "build": "nuxt build",
    "dev": "nuxt dev",
    "generate": "nuxt generate",
    "preview": "nuxt preview",
    "postinstall": "nuxt prepare"
  },
  "dependencies": {
    "@nuxt/eslint": "^0.3.13",
    "nuxt": "^3.12.2"
  },
  "devDependencies": {
    "@eslint/js": "^9.5.0",
    "eslint": "^9.5.0",
    "eslint-plugin-vue": "^9.26.0"
  },
  "packageManager"<EMAIL_ADDRESS>}

And then create this eslint.config.mjs file in your project root:
import js from '@eslint/js'
import vue from 'eslint-plugin-vue'
import withNuxt from './.nuxt/eslint.config.mjs'

export default withNuxt(
  js.configs.recommended,
  ...vue.configs['flat/recommended'],
  {
    ignores: [
      '*.ts',
      'node_modules',
      'public',
    ],
    languageOptions: {
      ecmaVersion: 'latest',
      sourceType: 'module',
    },
    rules: {
      //Because auto imports -.-
      'no-undef': 'off',
    },
  }
)

Run yarn (and note warning about ESLint version)
eslint is listed by your project with version 9.5.0 (p49b75), which doesn't satisfy what @nuxt/eslint and other dependencies request (^8.57.0).

Restart ESLint server in VSCode, and observe output:
Uncaught exception received.
Error: Cannot find module 'typescript'
Require stack:
-<EMAIL_ADDRESS>
Unfortunately I can't attach the yarn.lock file as it is too long. But the above steps are just a clean nuxt install and ESLint module addition.
As I don't use TypeScript in the project at all, it's a bit strange that I have to explicitly add it to my package.json, and that there is no easy configuration option to simply disable it so I don't need to use all these related dependencies. How can we opt-out of Typescript?

I will see what I can do

Here you go @antfu , let me know if this works for you:
https://github.com/adamreisnz/nuxt-eslint-reproduction/tree/issue-437

I've also put together a reproduction in #461 for the other issue I'm experiencing. In that branch you can see that the linter works once you manually add typescript to the package.json file

Any update on no longer requiring TypeScript when using this module?

Thank you @antfu !

I'm still having this issue with v0.5.1. Even setting features.typescript explicitly to false does not fix it. The only way is to install typescript as dependency.

Set to false? Not true?

We are not using typescript.

Here is the error:
[Error - 3:31:03 PM] Calculating config file for file:///home/myproject/eslint.config.js) failed.
Error: Cannot find module 'typescript'
Require stack:
-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>    at Module._resolveFilename (node:internal/modules/cjs/loader:1152:15)
    at Module._load (node:internal/modules/cjs/loader:993:27)
    at c._load (node:electron/js2c/node_init:2:13801)
    at Module.require (node:internal/modules/cjs/loader:1240:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/myproject/node_modules/@typescript-eslint/typescript-estree/dist/convert.js:30:25)
    at Module._compile (node:internal/modules/cjs/loader:1373:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1432:10)
    at Module.load (node:internal/modules/cjs/loader:1215:32)
    at Module._load (node:internal/modules/cjs/loader:1031:12)

Our dev dependencies:
  "devDependencies": {
    "@date-io/dayjs": "^3.0.0",
    "@nuxt/devtools": "^1.0.6",
    "@nuxt/eslint": "^0.5.1",
    "@nuxt/types": "^2.17.2",
    "@nuxtjs/google-fonts": "^3.2.0",
    "dotenv": "^16.3.1",
    "eslint": "^9.9.0",
    "eslint-config-vuetify": "^1.0.0",
    "eslint-plugin-vue": "^9.27.0",
    "firebase-tools": "^13.0.1",
    "html-to-vue": "^1.4.0",
    "nuxt": "^3.8.2",
    "sass": "^1.69.5",
    "vite-plugin-vuetify": "^2.0.1",
    "vue": "^3.3.10",
    "vue-eslint-parser": "^9.4.0",
    "vue-router": "^4.2.5",
    "vuetify": "^3.4.7"
  },

Our eslint config:
import js from '@eslint/js'
import vue from 'eslint-plugin-vue'
import vuetify from 'eslint-config-vuetify'
import vueEslintParser from 'vue-eslint-parser'
import withNuxt from './.nuxt/eslint.config.mjs'

export default withNuxt(
  js.configs.recommended,
  ...vue.configs['flat/recommended'],
  {
    files: ['**/*.js', '**/*.vue'],
    languageOptions: {
      parser: vueEslintParser
    },
    ignores: ['dist/*'],
    plugins: { vue },
    rules: {
...

@antfu
Here is the error:
[Error - 3:31:03 PM] Calculating config file for file:///home/myproject/eslint.config.js) failed.
Error: Cannot find module 'typescript'
Require stack:
-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>    at Module._resolveFilename (node:internal/modules/cjs/loader:1152:15)
    at Module._load (node:internal/modules/cjs/loader:993:27)
    at c._load (node:electron/js2c/node_init:2:13801)
    at Module.require (node:internal/modules/cjs/loader:1240:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/myproject/node_modules/@typescript-eslint/typescript-estree/dist/convert.js:30:25)
    at Module._compile (node:internal/modules/cjs/loader:1373:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1432:10)
    at Module.load (node:internal/modules/cjs/loader:1215:32)
    at Module._load (node:internal/modules/cjs/loader:1031:12)

Our dev dependencies:
  "devDependencies": {
    "@date-io/dayjs": "^3.0.0",
    "@nuxt/devtools": "^1.0.6",
    "@nuxt/eslint": "^0.5.1",
    "@nuxt/types": "^2.17.2",
    "@nuxtjs/google-fonts": "^3.2.0",
    "dotenv": "^16.3.1",
    "eslint": "^9.9.0",
    "eslint-config-vuetify": "^1.0.0",
    "eslint-plugin-vue": "^9.27.0",
    "firebase-tools": "^13.0.1",
    "html-to-vue": "^1.4.0",
    "nuxt": "^3.8.2",
    "sass": "^1.69.5",
    "vite-plugin-vuetify": "^2.0.1",
    "vue": "^3.3.10",
    "vue-eslint-parser": "^9.4.0",
    "vue-router": "^4.2.5",
    "vuetify": "^3.4.7"
  },

Our eslint config:
import js from '@eslint/js'
import vue from 'eslint-plugin-vue'
import vuetify from 'eslint-config-vuetify'
import vueEslintParser from 'vue-eslint-parser'
import withNuxt from './.nuxt/eslint.config.mjs'

export default withNuxt(
  js.configs.recommended,
  ...vue.configs['flat/recommended'],
  {
    files: ['**/*.js', '**/*.vue'],
    languageOptions: {
      parser: vueEslintParser
    },
    ignores: ['dist/*'],
    plugins: { vue },
    rules: {
...

Please create a new issue with proper minimal reproduction. Thanks


fix bad camera reference by using instance variable instead of parameter

Camera reference was wrong with some constructor, it uses the camera from parameter which is no longer valid when changing camera later.
note that It's not possible to reference instance variable when calling another constructor, so i had to move it to the final constructor.

I'm not familiar with the CameraSorter but I understand the new default CameraGroupStrategy (Camera camera) constructor won't set the sorter, can this break backwards compatibility? The constructors seem very simple, would it make sense in this case to just duplicate the logic and avoid calling one from the other?

@obigu you're right, i think it's more clean now, thank you.


⚠ Bofrost.at has degraded performance

In 9ef6123, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1614 ms

Resolved: Bofrost.at performance has improved in 1f510c5 after 28 minutes.


Pull request with type [CREATE] for user [sihlam] and object type [ConsoleExtension] - titled [test]

Testing PR, ignore


Data frame editing example doesn't work correctly

From https://shiny.posit.co/py/api/express/express.render.data_frame.html (and the core version), the third example app doesn't work correctly. It is meant to save the changes, but it only saves the most recent change to the data, instead of all the cumulative changes.
The examples are at:

https://github.com/posit-dev/py-shiny/blob/f732222ad981490ecaa698e7078e93c0ae6c6bd4/shiny/api-examples/data_frame_set_patches/app-express.py
https://github.com/posit-dev/py-shiny/blob/f732222ad981490ecaa698e7078e93c0ae6c6bd4/shiny/api-examples/data_frame_set_patches/app-core.py

@schloerke I changed the example in #1348 so that it works, but I feel like there could be a better way to do it. I'll merge and deploy now but we should revisit it when you are available.


remove azure-core-management from jacoco

These are fixes which caused auto increment PRs to malfunction after the beta release of schemaregistry.

Remove the commented out azure-core-management as the version tag was out of whack with the way management released and didn't update the versions.
Fix the incorrect version in the sdk/spring/azure-spring-boot-samples/azure-cloud-foundry-service-sample/pom.xml

/check-enforcer override

/check-enforcer override


koord-scheduler: add ElasticQuotaIgnoreOverhead feature-gate

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it
Ⅳ. Special notes for reviews
V. Checklist

[ ] I have written necessary docs and comments
[ ] I have added necessary unit tests and integration tests
[ ] All checks passed in make test

/lgtm
/approve


Bug: Eth appears in Send Assets when the DAO has none

Overview
This was something I had to fix in the Treasury assets page as well.
Gnosis's API returns plain Eth in the response, whether it has any or not, so a DAO with zero Eth will display that option in the Send Assets modal.
We should disregard any asset in the Send Assets modal that has a balance of zero, and not display it.
Steps to reproduce

View a DAO that has zero Eth
Click Send Assets
See that Eth is an option to select

https://app.dev.fractalframework.xyz/#/daos/0xbFC43fba83971596709cA0f66B1C54B89aad7551/treasury
Screenshots / Video

Version
Dev (app.dev.fractalframework.xyz/)
Browser
Firefox
Operating System
Mac
web3 wallet
None

@mobiusTripper-crypto is this something you could fix while you're doing the Eth transfer?

That sounds pretty easy to fix, and if so, i'll bake it into my next Send Assets PR


Swift 3.0 support

Updates to Swift 3.0
Update failing snapshot tests for iOS 10
Update travis for Xcode 8

Current coverage is 76.54% (diff: 82.60%)

No coverage report found for master at d541bb4.

Powered by Codecov. Last update d541bb4...161e944

merged, closing


Provide generalized version of getFilteredGames

For now the function getFilteredGames uses an API endpoint, but does not allow to use it to its full potential. This should be consistent and therefore change.

Extended it. Good enough for now I guess :smile:


Updated ONNX Runtime to 1.16.3

Updated ONNX Runtime to 1.16.3.
Not sure if I should be the one to update the change log. Let me know if I should do this differently. Thank you!

Hi @pjmelling, thanks for the PR. Looks good overall, but please use unreleased for the release date.

Updated the release date to unreleased.

Great, thanks!


Cannot open in classroom

I was doing this series without an issue but upon completion it says it was rejected and says, "We were unable to find a Classroom assignment for your account."
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

Now is it a solution again


RAND_DRBG uses uninitialized urandom read on linux

This message can be seen when openssl s_server is started early after reboot:
./openssl s_server
random: openssl: uninitialized urandom read (32 bytes read)
Using default temp DH parameters
ACCEPT

I have seen this with linux v4.18, and v4.14-stable.
Previous linux versions are also affected, but do not report the error.

This is not an error of openssl. Basically the kernel signals that the in-kernel CRNG was not yet fully initialized with sufficient entropy.

The point is, if we use /dev/urandom to initialize the DRBG the kernel tells us that the
returned random data is not good enough.
I think we should probably use /dev/random, and wait for enough entropy.

I assume this is done when we call the getrandom syscall.  According to the documentation I'm reading, the call we make should block until there's enough entropy, since we don't set the GRND_NONBLOCK flag:

       GRND_NONBLOCK
              By default, when reading from  the  random  source,  getrandom()
              blocks  if  no random bytes are available, and when reading from
              the urandom source, it blocks if the entropy pool  has  not  yet
              been  initialized.   If  the GRND_NONBLOCK flag is set, then ge‐
              trandom() does not block in these cases, but instead immediately
              returns -1 with errno set to EAGAIN.

Ahm, well....
Actually the used cross-compiler comes with a linux headers from 3.2.14, and glibc headers 2.15.
So if it does not define GRND_NONBLOCK and no getrandom syscall, what happens then?

My understanding is that the getrandom() system call is more or less equivalent to reading from /dev/urandom, whereas getentropy() corresponds to /dev/random.
However, there seems to be an important difference between getrandom() and /dev/urandom during early boot time: While getrandom() blocks until the initial seeding of the pool is complete, '/dev/urandom' according to RANDOM(4) never blocks.
   Usage
       The  /dev/random interface is considered a legacy interface, and /dev/urandom is preferred and sufficient in all use cases, with the exception
       of applications which require randomness during early boot time; for these applications, getrandom(2) must be used instead,  because  it  will
       block until the entropy pool is initialized.

       If  a seed file is saved across reboots as recommended below (all major Linux distributions have done this since 2000 at least), the output is
       cryptographically secure against attackers without local root access as soon as it is reloaded in the boot sequence,  and  perfectly  adequate
       for  network encryption session keys.  Since reads from /dev/random may block, users will usually want to open it in nonblocking mode (or per‐
       form a read with timeout), and provide some sort of user notification if the desired entropy is not immediately available.

So I would guess that OpenSSL was seeding from /dev/urandom, not from getrandom().

Okay, if the getrandom syscall is not available we seed from /dev/urandom and only if that
is not available use /dev/random, but it should be other way round.

So, how about that?
diff --git a/e_os.h b/e_os.h
index 9c0888e..461015a 100644
--- a/e_os.h
+++ b/e_os.h
@@ -28,9 +28,9 @@
  * default, we will try to read at least one of these files
  */
 #  if defined(__s390__)
-#   define DEVRANDOM "/dev/prandom","/dev/urandom","/dev/hwrng","/dev/random"
+#   define DEVRANDOM "/dev/random","/dev/prandom","/dev/urandom","/dev/hwrng"
 #  else
-#   define DEVRANDOM "/dev/urandom","/dev/random","/dev/srandom"
+#   define DEVRANDOM "/dev/random","/dev/urandom","/dev/srandom"
 #  endif
 # endif
 # if !defined(OPENSSL_NO_EGD) && !defined(DEVRANDOM_EGD)

Okay, if the getrandom syscall is not available we seed from /dev/urandom and only if that is not available use /dev/random, but it should be other way round.

I still don't think we should do this, we would hog the entropy pool by doing it. Just think of the OpenSSL testsuite where a lot of openssl instances are launched. If they all would be seeding from /dev/random (or getentropy(), you would be in trouble.
For the current Linux kernels the fact that getrandom() blocks until the initial seeding has completed should be sufficient. After that, the kernel CSPRNG is cryptographically secure, which is good enough, unless we are running in FIPS 3.0 mode (t.b.d.) and it is either the first seeding of the <master> DRBG, or prediction resistance was requested. (It is clear that the entropy gathering needs to be improved for FIPS and that we need distinction between blocking sources with entropy guarantuee and nonblocking sources).
As for older kernels without getrandom(): IMHO it is the job of the init system to save and restore a seed file across reboots, as recommended in the citation from RANDOM(4). So I wouldn't consider it an OpenSSL bug if this message occurs.

@kroeckx might know more about kernel CSPRNG details than I do.

How does this affect the answer I gave
to https://github.com/openssl/openssl/pull/8023#discussion_r256318746
I said: "If we let it fail here, the chances are that the caller of RAND_bytes does not check the
return code, or it does at least not have no good strategy to handle the error.
I would prefer to wait, unless it is a permanent error."
but I think VxWorks Randlib is working like /dev/random, just that it fails when no entropy
is immediatly available.

How does this affect the answer I gave to #8023 (comment)

It would be good to have more detailed information about the VxWorks randlib in general and the randBytes() call in particular, before answering this question. Is there any  official documentation available online or offline?
cc @klotzt-draeger

.. the chances are that the caller of RAND_bytes does not check the return code, or it does at least not have no good strategy to handle the error.

This is a really good question and I remember having discussed it with @kroeckx (IIRC) at some time:
Currently the DRBG refuses to generate random bytes if it is in the error state(*), so it is crucial to check the return value of RAND_bytes(). I asked Kurt whether it wouldn't be better to return an error and generate the requested random bytes regardless of the error state, just to be safe in the case where the application wouldn't check the error. But the conclusion was that it would be even worse, because we would provide fake security and the chance of detecting the failure would be less than if the RNG would'nt refuse to generate the output.
(*) The DRBG attempts an automatic error recovery by reseeding, but if that fails, it will remain in the error state.

Citing myself from RAND(7):
       As a normal application developer, you do not have to worry about any details,
       just use RAND_bytes(3) to obtain random data.  Having said that, there is one
       important rule to obey: Always check the error return value of RAND_bytes(3)
       and do not take randomness for granted.

getrandom() and getentropy() will always block until the kernel
CSPRNG has been initialized. If you have a kernel that supports
it, so >= 3.17, we should always use it. Only with an older kernel
should be fall back to using /dev/urandom.
I've actually had people report that the boot now hangs because of
this for some people, for instance openssh blocking until it can
get random data.
I don't think we should be using /dev/random at all. If
getrandom()/getentropy() is not available, /dev/random will
probably also do the wrong thing and cause things to hang, but
then hang all the time.
Reading 1 byte from /dev/random should also not be done by each
process. People that want to make sure that no application reads
from /dev/urandom before it's initialized should prevent those
applications from starting instead, by for instance reading 1 byte
during the boot process.

To actually be able to use getrandom on Linux if it's not provided
by glibc, you need to compile against recent enough kernel headers
so that SYS_getrandom is defined. Note that SYS_getrandom is
architecture specific.

For the current Linux kernels the fact that getrandom() blocks until the initial seeding has completed should be sufficient. After that, the kernel CSPRNG is cryptographically secure, which is good enough, unless we are running in FIPS 3.0 mode (t.b.d.) and it is either the first seeding of the <master> DRBG, or prediction resistance was requested. (It is clear that the entropy gathering needs to be improved for FIPS and that we need distinction between blocking sources with entropy guarantee and nonblocking sources).

According to the discussions I've had with the FIPS lab about entropy sources, this isn't correct.  Calling a blocking getrandom() system call is good enough for FIPS and is straightforward to justify.
If we need to use /dev/urandom, the only method I can think of to guarantee that it is properly seeded is via waiting for something like dd if=/dev/random of=/dev/urandom BS=32 count=1 iflag=fullblock  I.e. read bytes from the blocking source and feed them back into the RNG.
We should avoid /dev/random is we can, but we might not be able to do so.

+#   define DEVRANDOM "/dev/random","/dev/prandom","/dev/urandom","/dev/hwrng"

I'd put /dev/hwrng before /dev/prandom.  /dev/prandom is a pseudo random generator like /dev/urandom and I don't believe claims about made about it being properly seeded.

+#   define DEVRANDOM "/dev/random","/dev/urandom","/dev/srandom"

/dev/srandom should be before /dev/urandom and perhaps even before /dev/random.  It is a good source.

According to the discussions I've had with the FIPS lab about entropy sources, this isn't correct. Calling a blocking getrandom() system call is good enough for FIPS and is straightforward to justify.

That's good news. It seems that I was too pessimistic. :-)

Calling a blocking getrandom() system call is good enough for FIPS and is straightforward to justify.

What about prediction resistance? It was always Kurt's opinion (not mine) that a prediction resistance request needs to be propagated upwards. This would imply that in this case, getentropy() would be required to reseed.

Please do not ever use /dev/random in openssl by default. That would be simply not usable in any normal scenario where openssl is used throughout the system.

It is OS business to ensure that the kernel RNG is properly seeded with entropy on system boot, openssl has no way to influence/enforce that. To summarize - use getrandom/getentropy (without GRND_RANDOM flag of course) and if it is not available, use /dev/urandom. That's it.

@t8m I quite agree.

getrandom() should stir in more entropy if any is available, i.e. it isn't a pure PRNG.  Of course, it would always be better to guarantee new entropy has been added.
The problem with /dev/urandom, in the context of the FIPS project, is the justification that it has been adequately seeded.  Once it has been seeded, it's fine but until you can prove that, it isn't acceptable.  In FIPS mode we might be forced to used /dev/random in the absence of getrandom() and getentropy() system calls.  Outside of FIPS mode, I can't see this ever being mandated.

I think this does probably not do what was intended:
openssl-1.1.1a/crypto/rand/rand_unix.c:#  if defined(__linux) && defined(SYS_getrandom)
openssl-1.1.1a/crypto/rand/rand_unix.c:    return syscall(SYS_getrandom, buf, buflen, 0);

... because linux defines __NR_getrandom and glibc defines SYS_getrandom.
So glibc before 2.16 will not define SYS_getrandom regardless of the linux version.

Do not ever use /dev/random ... that is unacceptable

@t8m please explain this?

... because linux defines __NR_getrandom and glibc defines SYS_getrandom.
So glibc before 2.16 will not define SYS_getrandom regardless of the linux version.

Update:
The glibc 2.25 was the first to implement the getrandom function, but I tried to build
glibc 2.24 against the current linux headers, and it did define the SYS_getrandom.
But this define will not be created when the linux headers are updated afterwards.
It would still make sense to use the linux define __NR_getrandom directly, IMHO.

Do not ever use /dev/random ... that is unacceptable

@t8m please explain this?

Apart from the obvious blocking?

Yes, we should use __NR_getrandom.

If my memory is correct, FIPS says that using the OS provided
entropy is fine, it does not talk about which interface you talk
to the OS. I don't see why FIPS should care about /dev/urandom vs
getrandom(), other than that Linux's /dev/urandom is broken and
returns data before it has been initialized. But I think making
sure that is initialized is up to the OS, not OpenSSL.

Apart from the obvious blocking?

Why is that a problem?   The default configuration, since most users are not knowledgeable, should be to wait.

Why is that a problem? You either wait until there's enough entropy,...

The real problem is not that your process needs to wait longer, it's that it slows down all other processes, too, which do the same. Just assume that libcrypto would seed from /dev/random by default. Then all processes using OpenSSL would seed from /dev/random and draining the entropy pool. In the case of a web server, which forks for incoming connections, I guess you would notice the slow down a lot.
You can simulate this draining effect using dd. Just open one terminal running
watch -n 1 cat /proc/sys/kernel/random/entropy_avail

and several terminals running
dd if=/dev/random bs=1 count=32768 | hexdump

You will notice that the performance gets worse and worse if you add more and more of these.
Finally, compare it to the performance of reading from /dev/urandom:
dd if=/dev/urandom bs=1 count=32768 | hexdump

And again I ask, what is the alternative?  Competition for a scarce resource, in a land of naive programmers.

Just wait until the initial seeding has completed and then simply use it's output without blocking any further (that's what getrandom() does).
In this context you might find the following blog post interesting:
https://www.2uo.de/myths-about-urandom
The author's opinion might not be authoritative, but he seems to be supported by D. J. Bernstein's himself. And if I understand Pauli's comment correctly, even the FIPS lab members agree.

Yes, but and that is only after the getentropy syscall which should have been used,
but maybe the fallback after the mis-configuration is wrong, and should use /dev/random
even if it waits.

It would be good to have more detailed information about the VxWorks randlib in general and the randBytes() call in particular, before answering this question. Is there any official documentation available online or offline?

According to https://github.com/openssl/openssl/pull/8023#discussion_r256318746
You may assume that randBytes is returning error when not enough entropy, and when you read from
the randBytes the pool starts to get empty again, it depends on the hardware how long
this takes, can be seconds or minutes.
Is it better to wait or to fail if this happens?

use /dev/random even if it waits.

MAYBE we should only read 1 byte from there and after that continue with /dev/urandom.
I observe that once one byte is available several (~8) one byte reads succeed immediately
untit the /dev/random starts to block again.  As a compromise ?

Guys, some of you seem a bit confused about what getentropy() and getrandom() are and how they relate.

getrandom() is a syscall
getentropy() is a library routine that uses getrandom(), you can see it for yourself in the source

FIPS says that using the OS provided entropy is fine, it does not talk about which interface you talk to the OS.

FIPS 140-2 Implementation Guidance 7.15 Entropy Assessment makes a clear distinction:

In case of the /dev/urandom request, the call to this OS entropy generator is non-blocking. The data obtained from the non-blocking call is not guaranteed to possess the desired amount of entropy. How can the vendor provide the assurance that the requirements of the FIPS 140-2 AS.07.13 assertion are satisfied?
To meet these requirements, the vendor must first demonstrate that the initial call (that is, the first call after the module has been powered up or instantiated) to /dev/urandom returns the claimed amount of entropy.

There is quite a bit more wordage around this but it is the non-blocking nature before initial seeding that is pointed to as the problem.

MAYBE we should only read 1 byte from there and after that continue with /dev/urandom.
I observe that once one byte is available several (~8) one byte reads succeed immediately
untit the /dev/random starts to block again.

I'm pretty sure that this won't guarantee that /dev/urandom has been properly seeded.
NIST doesn't say anything about getrandom() yet.  However, after discussions with our FIPS lab, it appears that it would be a justifiable source and it is guaranteed to be seeded properly.

@paulidale

I'd put /dev/hwrng before /dev/prandom. /dev/prandom is a pseudo random generator like /dev/urandom and I don't believe claims about made about it being properly seeded.

I agree about /dev/prandom, and would actually remove it, it is seeded from /dev/urandom and
uses triple-DES, it only exists because of the hardware support, which might offer some speed-up.
However the /dev/hwrng is not limited to s390, any hardware device can register to be a TRNG device.
I guess if it is available it will be the seed /dev/random and /dev/urandom, so I would keep it
after /dev/urandom.

/dev/srandom should be before /dev/urandom and perhaps even before /dev/random. It is a good source.

I don't know anything about srandom, is this a BSD thing?  Is it similar to /dev/hwrng, the seed
of /dev/urandom ?

getentropy() is a library routine that uses getrandom(), you can see it for yourself in the source

I think you meant to provide a link to getentropy.c, not getrandom.c, but thanks for the link. Indeed, I was not aware that getentropy() is implemented in glibc only, based on the getrandom system call.
(But note that the two links above also show that you are you are just almost right: both functions are library routines:  while getrandom() is a thin wrapper around the getrandom system call, getentropy() adds more logic. ;-)  )

I'm pretty sure that this won't guarantee that /dev/urandom has been properly seeded.

At least with recent linux kernel that is the case.
The getrandom syscall unblocks exactly at the same time where /dev/random is able
to reading the first byte after system boot.

I'm pretty sure that this won't guarantee that /dev/urandom has been properly seeded.

At least with recent linux kernel that is the case.
The getrandom syscall unblocks exactly at the same time where /dev/random is able
to reading the first byte after system boot.

Theodore Ts'o explains in the commit message of commit c6e9d6f388947986 that the behaviour of /dev/urandom was not changed  to block initially (like getrandom) for compatibility reasons:
   This is because changing /dev/urandom reads to block represents an
   interface change that could potentially break userspace which is not
   acceptable.

Idea:
instead of actually reading 1 byte from /dev/random,
we could select /dev/random for readable, and once it is readable, read /dev/urandom.
That would work more or less around the issues with concurrent access.

A promising idea, indeed. I'd be interested to learn whether this trick helps to get rid of the "uninitialized urandom read" warning.

If you use getentropy() or getrandom(), the interface actually
blocks until the it has been initialized, on all OSs. It starts to
return data the same time /dev/random starts to return data the
first time.
If you use /dev/urandom on Linux, it reads wether it's initialized
or not. If you read /dev/random on Linux, it blocks until it's
initialiazed, and then says you remove entropy once you read
anything, which doesn't make sense with the current PRNG in the
kernel (it might have with the old one). Both /dev/random and
/dev/urandom on Linux really are broken. And they don't fix
/dev/urandom because it caused a hang in some applications (which
now hang anyway because they switched to getrandom()).
If you can't use getrandom()/getentropy(), you can work around
/dev/urandom's broken behaviour by reading 1 byte from
/dev/random. This 1 byte only needs to be read by 1 application
during the boot process, not every application should be doing
that. Linux distro's could fix this by adding that 1 byte read
somewhere in the boot sequence, but none do because the boot would
hang. We could do this in OpenSSL, or for FIPS we could just
document that it's up to the OS to make sure this has been done.
This is the only problem FIPS has with the use of /dev/urandom,
just like everybody else.
Recent Linux kernels actually are able to initialize the RNG within
a few seconds of booting.
On some other OSs, /dev/random and /dev/urandom are actually sane,
there is no difference between them, and only return data after
it's been initialized.
In short, just switch to getrandom()/getentropy().

Sure, this is just a fallback, when that is not available, but from the emotional reaction
on my proposal to move /dev/random before /dev/urandom, I concluded that there
must be a real problem somewhere.
Anyway, select for readable ought to guarantee that at least one byte can be
read without blocking.

Hmm, this does not really work as expected, (using v4.14-stable with RT branch):
the select waits a few seconds, and then the  /dev/urandom is still complaining
also reading 1 byte with dd is not waiting long enough,
acutally /dev/random seems to be returning bytes before it is supposed to be initialized:
root@socfpga_cyclone5:~# dd if=/dev/random bs=1 count=1 | hexdump -C ; dd if=/dev/urandom bs=1 count=1 | hexdump -C
1+0 records in
1+0 records out
00000000  f7                                                |.|
00000001
random: dd: uninitialized urandom read (1 bytes read)
1+0 records in
1+0 records out
00000000  96                                                |.|
00000001
root@socfpga_cyclone5:~# dd if=/dev/random bs=1 count=1 | hexdump -C ; dd if=/dev/urandom bs=1 count=1 | hexdump -C
1+0 records in
1+0 records out
00000000  76                                                |v|
00000001
random: dd: uninitialized urandom read (1 bytes read)
1+0 records in
1+0 records out
00000000  a3                                                |.|
00000001
root@socfpga_cyclone5:~# dd if=/dev/random bs=1 count=1 | hexdump -C ; dd if=/dev/urandom bs=1 count=1 | hexdump -C
1+0 records in
1+0 records out
00000000  d2                                                |.|
00000001
random: dd: uninitialized urandom read (1 bytes read)
1+0 records in
1+0 records out
00000000  5b                                                |[|
00000001
root@socfpga_cyclone5:~#

I don't know anything about srandom, is this a BSD thing?

Yes, it is a BSD device.  From the man page:

Strong random data.  This device returns reliable random data.  If sufficient entropy is not currently available (i.e., the entropy pool quality starts to run low), the driver pauses while more of such data is collected.  The entropy pool data is converted into output data using MD5.

MD5 ?

That's what the man page I found said.  I wouldn't be surprised if the algorithm had been changed since.  I wouldn't consider MD5 as being broken as an entropy whitening function -- collisions are very remote from reversibility.  I'd still advocate moving to something more recent of course.

Do not ever use /dev/random ... that is unacceptable

@t8m please explain this?

Apart from the obvious blocking?

Yes, that is the main reason. And for the reason that always-blocking /dev/random fighting for entropy is a broken concept on multi-process multi-user system it is often also blocked by security mechanisms such as SELinux. Basically /dev/random could be accepted as a special choice for user but it should never be used by default in general purpose crypto library.

Hmm, this does not really work as expected, (using v4.14-stable with RT patch):
the select waits a few seconds, and then the /dev/urandom is still complaining
also reading 1 byte with dd is not waiting long enough,
acutally /dev/random seems to be returning bytes before it is supposed to be initialized:

Because the entropy gathered is directed to various pools and never re-used. Roughly said, first the CRNG (which feeds /dev/urandom and getrandom()) is half-initialized with 64 bits of entropy (or so), then the entropy gathered goes to initialize the /dev/random entropy pool and only after that one is initialized, it goes to complete the CRNG initialization. It might not be exactly like that and it changed a little over various kernel versions but you can get the picture.
Basically: Please leave it on the OS to ensure /dev/urandom is properly initialized, do not try to outsmart it in the library code. It would have to be patched out anyway for general use in Linux OSes.

The idea is to wait on /dev/random using select(), but to never
actually read from it.
I'm sure opening /dev/random will fail in some cases, and we
should then just continue like we did previously.

If my understanding of the process is current, this will have a race condition, whereby something else could consume the collection pool's entropy before it is sent to urandom pool. Still, it is a better approach than doing nothing.


Connects auth-service with client

Client sends a request for signing in to auth-service and caches received access-token.
Client sends a request for signing out to auth-service and successfully redirects to '/login'.
Client sends a request for refreshing an access-token, when refreshing a page.
Every request from client contains the access-token as http-header.

I have left the old outcommented lines of code as they were before.

@GabrielInTheWorld I'm currently working on this. I took the opportinity to redo the operator once again to adjust for the multi-meeting part in OS4. I'll ping you, when I'm done.

NOTE: Needs https://github.com/OpenSlides/openslides-auth-service/pull/18 to work...

@GabrielInTheWorld Can you provide new tokens with PR https://github.com/OpenSlides/openslides-auth-service/pull/24 ?

Wait for https://github.com/OpenSlides/openslides-auth-service/pull/19

@GabrielInTheWorld I have good feelings about this! Looks good. What are next steps to take?

@GabrielInTheWorld should we merge it before making more changes?

I'll merge this for now. You can work in a new PR, if you currently have additional things to implement.


Allow edit built-in network

Close #2347

How about adding a Reset button to restore the default value of a network?

How about adding a Reset button to restore the default value of a network?

Yes, that would be perfect. People could accidentally change the values, or want to try a different RPC, but then think it's better with the built-in ones. They'd have no way to do this other than re-installing.

They'd have no way to do this other than re-installing.

Got it.


Loading keys

KeyType implements Serialize which offres a to_bytes function, but there doesn't appear to be a function to load such serialization.

there are various ways to import keys, for example from a string like here: https://github.com/rpgp/rpgp/blob/master/tests/key_test.rs#L143

Are they compatible with the format on KeyType or SecretKey? The various key generation and serialization methods are confusing.