SQLite errors after a while

⚠ Please verify that this bug has NOT been raised before.

[X] I checked and didn't find similar issue

🛡 Security Policy

[X] I agree to have read this project Security Policy

Description
After a while of running Uptime Kuma (on a Synology diskstation as a docker container), it becomes extremely slow, and I start seeing the following error messages:
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?     at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:305:26)     at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:259:28)     at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)     at async RedBeanNode.storeCore (/app/node_modules/redbean-node/dist/redbean-node.js:166:26)     at async RedBeanNode.store (/app/node_modules/redbean-node/dist/redbean-node.js:126:20)     at async beat (/app/server/model/monitor.js:639:13)     at async Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:658:17) {   sql: undefined,   bindings: undefined }     at consoleCall (<anonymous>)     at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:660:25) 2022-11-11T03:15:34.556Z [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues 2022-11-11T03:15:34.556Z [MONITOR] INFO: Try to restart the monitor Trace: [Error: insert into heartbeat (down_count, duration, important, monitor_id, msg, ping, status, time) values (0, 20, false, 10, 'Records: <IP_ADDRESS>', 5, 1, '2022-11-11 03:14:34.065') - SQLITE_BUSY: database is locked] {   errno: 5,   code: 'SQLITE_BUSY' }     at consoleCall (<anonymous>)     at Timeout.safeBeat [as _onTimeout] (/app/server/model/monitor.js:660:25) 2022-11-11T03:15:36.157Z [MONITOR] ERROR: Please report to https://github.com/louislam/uptime-kuma/issues 2022-11-11T03:15:36.157Z [MONITOR] INFO: Try to restart the monitor Trace: [Error: UPDATE monitor SET dns_last_result = 'Records: <IP_ADDRESS>' WHERE id = 6  - SQLITE_BUSY: database is locked] {   errno: 5,   code: 'SQLITE_BUSY' }     at process.<anonymous> (/app/server/server.js:1728:13)     at process.emit (node:events:390:28)     at emit (node:internal/process/promises:136:22)     at processPromiseRejections (node:internal/process/promises:242:25)     at processTicksAndRejections (node:internal/process/task_queues:97:32) If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
👟 Reproduction steps
There is no reproduction step, this just happens after a while. The only way to resolve this is to clear the history of all monitors.
👀 Expected behavior
These errors don't appear automatically.
😓 Actual Behavior
Errors appear.
🐻 Uptime-Kuma Version
uptime-kuma:1.18.5-alpine
💻 Operating System and Arch
Alpine Docker container
🌐 Browser
Chrome
🐋 Docker Version
No response
🟩 NodeJS Version
No response
📝 Relevant log output
No response

I see this is a duplicate of https://github.com/louislam/uptime-kuma/issues/1397


Performance of extendOverride

Our unit tests at work were starting to get pretty slow, so I kicked on the profiler. I found about half (!) of the total time spent when running tests was coming from one method in restmod: Utils.extendOverriden.
Here's a profile of running 20% of our tests: http://screencast.com/t/Bwhk3Iili5
I'm not sure if this is something you were already aware of or not, or if possibly there's something strange about our environment which is a contributing factor.
Thanks,
Chris

I think this has to do with angular tearing things down between tests, so it's forced to rebuild the models after each test file. We have quite a few models, so the time adds up.

I'm closing this issue, we are planning on adding model mocks and helpers in the "future", maybe that will help on improving performance.


fix wrong matching (unsupported CSI and extraordinary text present)

Test Code:
//                Unsupported CSI      Pure Text
//                vvvvvvvvv            vvvvvv
var txt = "ABC" + "\033[1@" + "FOO" + "[1;30m" + "BAD" + "\033[0m";
var html = ansi_up.ansi_to_html(txt);
console.log(html + "\n");

Current ansi_up Output:
ABCBAD

PuTTY Display (for reference, printed same txt byte sequence by other program):
ABCFOO[1;30mBAD

In this test case, [1;30m is a pure text. It is not an escape sequence.
It caused by this regexp matching:
https://github.com/drudru/ansi_up/blob/8d120f8cd02ff6a83249dc0140c72c747454e322/ansi_up.js#L98
In this case, this string has been stored in the text variable: [1@FOO[1;30mBAD
This extraordinary string causes wrong matching and lost "FOO".
If replace text.match(/([\d;]*)m([\s\S]*)/m); with text.match(/^([\d;]*)m([\s\S]*)/m);, this issue will assuage. (But it's not perfect, because unknown escape sequence still display)
Output If this change applied:
ABC1@FOO[1;30mBAD

Sorry my poor English. Thank you.

Awesome - will review today!

Additinal Info: For reproduce this issue, [ (in text part) is not required.
"ABC" + "\033[1@" + "FOO" + "1;30m" + "BAD" + "\033[0m" also reproduce.
https://wandbox.fetus.jp/permlink/GoikPtKrYTDeJOfj (Output area use ansi_up)

...looking at this now. thanks for your patience.

OK - @fetus-hina - good stuff.
Here is what I would like to do.
1st - please create some tests that would fail for the old behaviour.
2nd - I would like to fix the other bug. I want to ignore sequences that are not correct.
If you want to do this, let me know. Otherwise, I can take your code, add tests, and come up with a fix for the unknown escape sequence.

... oh yeah - and if you do it... please add your info as a contributor to the Readme.

OK, I'll fix it.
One moment please.

Pull-request updated.

AWESOME work. I really like the tests.
I'm testing this now. Should be merged in a few and then a new npm pushed in 10 mins.
Thank you!


Sample Code is wrong

Mat src = new Mat("lenna.png", ImreadModes.GrayScale);
should be
Mat src = new Mat("lenna.png", LoadMode.GrayScale);

Thanks! This is correct in the 3.x wrapper. The name of the enum was changed. I will change the sample code as below
Mat src = new Mat("lenna.png", ImreadModes.GrayScale); // OpenCvSharp 3.x
Mat src = new Mat("lenna.png", LoadMode.GrayScale);  // OpenCvSharp 2.4.x


feat: handling UTM, affiliate tracking and GCLid in P2P standalone [WIP]

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

55 of 86 (63.95%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 85.229%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/utils/cookie.ts
53
84
63.1%

Totals

Change from base Build<PHONE_NUMBER>1:
-0.4%

Covered Lines:
5623

Relevant Lines:
6134

💛  - Coveralls


Fixes for Task Agent for release91

Hi @nigurr, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Nitin Gurram). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

Signed off


Assigning To Class/Record Fails to Query Properties

Describe the bug
Following the tutorial while implementing my own data, I'm not able to query and place the result in a new class or record, but the query works fine when I place it in an anonymous object.
How to Reproduce
Working Example
var response = await client.Query(
    o => o.SearchProfileByUsername(first: 1, username: username, selector: pi => pi.Nodes(profile => new {
        Id = profile.Id,
        Name = profile.Name
    }))
);

This results in the following query with an appropriate response:
{"variables":{"username":"test"},"query":"query ($username: String!) { searchProfileByUsername(first: 1, username: $username) { nodes { id name __typename }  } }"}

Note how we are querying in the nodes for id, name, along with __typename.
Not Working Example
Now let's say I have this record:
public record Profile(string Id, string Name);

Note: the same result can be obtained in a class instead of a record
And then I attempt to create and inject from ZeroQL:
var response = await client.Query(
    o => o.SearchProfileByUsername(first: 1, username: username, selector: pi => pi.Nodes(profile => new Models.Profile(
        Id: profile.Id,
        Name: profile.Name
    )))
);

This is now what is sent:
{"variables":{"username":"nblew"},"query":"query ($username: String!) { searchProfileByUsername(first: 1, username: $username) { nodes {  __typename }  } }"}

Note that the only property being queried by ZeroQL in the "node" is __typename, but not id or name. This results in those properties always being null.

Expected behavior
I would expect that accessing a property will always request it in the graphql query regardless if the target object is an anonymous object or a well-defined class
Environment (please complete the following information):

Nuget package version [6.1.1]
IDE: [VS 2022 17.8.6, VS Code]
.Net Version [8.0]
App Type: Console

I have tried replicating your example, and it works fine for me.
Can you create a small repository with the issue so I can pull and check it?
Here is my query:
namespace expZeroQL
{
    public record User(ID Id, string FirstName);
}

// ...

var response = await client
.Query(q => q
    .Me(o => new expZeroQL.User(Id: o.Id, FirstName: o.FirstName)));

Ok update - After some digging it looks like the example I gave didn't give the issue in question, and I was on the wrong track with thinking it had to do with classes/record (although an anonymous type throws an exception while a record just gives a warning when the issue happens). The issue has to do with attempting to transform a portion of the query into a property.
I opened a new issue with the problem in question.


Add Tag-Engage Workflow Operation Handler

This adds a new workflow operation handler with which you can update tags and flavors of media package elements directly in the engage publication. :tada: This means that for such changes, it's no longer necessary to pull the files from the publication into the workspace, saving lots of time. The operation works almost exactly the same as the tag WOH except that it looks at published elements and does not support copy. The publication is updated in the archive as well as the search service.
This could probably be generalized to work for other publications, but this is currently out of scope.

This should go into 13.

This should go into 13.

Any particular reason for this? There's not a rule against features going into a legacy release, is there? One could argue for develop, maybe, but I think it's unlikely this will affect anything else.
I would rather this go into 12 because this will mean less custom code for us sooner. ;)

mmh in my head, I always thought that legacy releases should not get new features. But looking at the mailing list's last messages on that topic, I don't see that anymore. BTW our dev docs still say that features should only go into develop 🙈.
Regarding the operation: I'm not a fan of highly specialized operations, but I think I have kind of given up on that.

Regarding the operation: I'm not a fan of highly specialized operations, but I think I have kind of given up on that.

I agree with you there, I think it would be better if this operation could handle arbitrary publications.
But since I wrote this in one day to quickly fix thousands of published videos with the wrong tag, I'm kind of happy with it anyway. ;D
Maybe in the future...


Error: "Frame window timed out" in signinSilent

Hi,
although see it in many posts, looks like my case don't fit any of the existing. all configured and working, and i'm using implicit flow.
Work well on local machine under localhost, however it doesn't work on the server.
always get "Frame window timed out" error from "this._userManager.signinSilent()"
With fiddler I can see no errors in the response, It does return new token Exp to the required renew redirect url. (can see it in fiddler).
Background, both the SPA site and the Ids on same OWIN host process, and we have sub domain in the url
https://xxx-yy.xxx-xxx-xxx-xxx.dev.xxx.com:44330/ids/
https://xxx-yy.xxx-xxx-xxx-xxx.dev.xxx.com:44330/app/silent-renew.html
Any idea where should I look?

So there's not error in the token server?

No errors, I can see the right response coming on the network with Fiddler.
I get a token with new exp date. Also working under localhost. Could it be
browser security somehow b/c the sub domain?
On Apr 15, 2017 09:39, "Brock Allen"<EMAIL_ADDRESS>wrote:

So there's not error in the token server?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IdentityModel/oidc-client-js/issues/311#issuecomment-294293978,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQnOurmp8xgyqsCEv4KViuyde3A0yZjks5rwMiPgaJpZM4M-H7_
.

Could it be browser security somehow b/c the sub domain?

Possibly -- is it IE?

We see the same problem with IE and Chrome
On Apr 15, 2017 14:32, "Brock Allen"<EMAIL_ADDRESS>wrote:

Could it be browser security somehow b/c the sub domain?
Possibly -- is it IE?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IdentityModel/oidc-client-js/issues/311#issuecomment-294310407,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQnOpRlp3ld_gcbMYHTdgb5DeFkZo5xks5rwQ1HgaJpZM4M-H7_
.

Then not sure. I'd need to you debug it since there are no consistent repro steps.

No update?

Hi, it looks like an internal problem in the network, with company security
protection tool websense proxy setting. Looks like it is not an application
problem. Once this is solve I'll double check.
FYI
On May 25, 2017 9:24 AM, "Brock Allen"<EMAIL_ADDRESS>wrote:

No update?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IdentityModel/oidc-client-js/issues/311#issuecomment-304009143,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQnOg-ybkOiL048mcOP_BP77nd9OD2hks5r9YELgaJpZM4M-H7_
.

Hi,
I have the same problem but only in Firefox (in Chrome the signinSilent method is working fine). In my case I receive the correct response from the server but two errors are showed in the console: "Frame window timed out" and "TypeError: getComputedStyle(...) is null". This is too strange because the response is fine, so I think this could be related only with the browser.

Hi,
I have great news. I have found what is the problem related with firefox. As you can see for instance here: https://bugzilla.mozilla.org/show_bug.cgi?format=default&id=548397 there is a bug in firefox related with the hidden iframes that are created with: this._frame.style.display = "none"; In order to avoid this error, it is recomended that instead of create the hidden iframe with the display = "none", to use
this._frame.style.visibility = "hidden";
this._frame.style.position = "absolute";

I have modified the IFrameWindow.js class in my local and it works!
So please, if you think that this is not breaking anything, include it in your solution (or if you want I can do a pull request).
Could you please inform me if you are going to include it?
Thank you so much!

Sure, except how would that work with CSP enabled?

I have done a search in google and I think this._frame.style.visibility = "hidden"; is completely compatible with the CSP. No issues related with that have been opened.

Hi! Any update?
Thanks!

Did you ever submit a PR for this?

Ok, I made the changes you suggest and they seem to be working. Thanks.

@brockallen removing display: none introduced an issue in chrome 60 where the iframe now has a computed height. If creating the iframe with a display property causes issues in FF, could you add an id/class to the iframe for easy targeting?

That stinks. Yea, I can look at something...

I just pushed 1.4.1-beta.1 -- please test. Also, @franpsif, can you check this in whatever version of FF you were seeing the problem? My version seems to work fine.

Fixed: #430

I am still getting this error in chrome and IE.
Identity server CSP
               var csp = "style-src 'self' 'unsafe-inline'; object-src 'none'; frame-ancestors 'none'; sandbox allow-forms allow-same-origin allow-scripts; base-uri 'self';";

Error: Uncaught (in promise): Error: Frame window timed out
Error: Frame window timed out
    at t [as _error] (oidc-client.min.js:47)
    at t [as _timeout] (oidc-client.min.js:47)
    at ZoneDelegate.invokeTask (zone.js:431)
    at Object.onInvokeTask (core.js:27769)
    at ZoneDelegate.invokeTask (zone.js:430)
    at Zone.runTask (zone.js:198)
    at ZoneTask.invokeTask (zone.js:513)
    at ZoneTask.invoke (zone.js:502)
    at timer (zone.js:3041)
    at resolvePromise (zone.js:836)
    at resolvePromise (zone.js:795)
    at zone.js:897
    at ZoneDelegate.invokeTask (zone.js:431)
    at Object.onInvokeTask (core.js:27769)
    at ZoneDelegate.invokeTask (zone.js:430)
    at Zone.runTask (zone.js:198)
    at drainMicroTaskQueue (zone.js:611)
    at ZoneTask.invokeTask (zone.js:517)
    at ZoneTask.invoke (zone.js:502)

Same here:
Refused to frame 'https://sts-identity.2020.onelife.eu/' because an ancestor violates the following Content Security Policy directive: "frame-ancestors 'none'".
We have the client set on 2 apps - the first one do not have this problem, the second one has it. So it is may be some network problem


Create a community panel to list buttons for Youtube / Feature request / Slack invite / etc

Linear: https://linear.app/intuita/issue/INT-957/create-a-community-panel-to-list-buttons-for-youtube-feature-request
Claap: https://app.claap.io/intuita/community-panel-c-BT2DRbCjzO-TvDllpEFoMjv

Final


Add native SteamVR support - including Steam Actions

Does this affect the legacy HoloToolkit (master) or the Mixed Reality Toolkit (mrtk_release)?
Mixed Reality Toolkit
Is your feature request related to a problem? Please describe.
Community members have stated they would also like to see native SteamVR support instead of using Unity's OpenVR implementation (I'd have to agree :D )
Describe the solution you'd like
SteamVR implemented as a native Platform
Describe alternatives you've considered
Currently Steam VR is implemented using OpenVR

MRTK currently has a SteamVR implementation for the 2018 (and legacy 2019) XR platform. On 2019, it will be utilizing the XR Management platform as available.


Update EN_std_lev_bkg_and_buddy_check.py

Fixes a bug described in #206.

Excellent, glad we caught this. Thanks, @s-good!


Monitor Manager Activity Logs only returns a maximum of 200 events

Although it is still in beta, I have tried to use the MonitorManager and also the MonitorClientImpl to retrieve events from Azure Activity Logs.
When trying to retrieve the next page or to load all events, I only get a maximum of 200, and the next page comes as null. Have tried the ActivityLogsInner with both list() and listAsync() methods in the Azure Java SDK 1.2.1

@RikkiGibson could you please confirm if that is a known issue?

Thanks for letting us know about this. I'll try and figure out where the root cause of this lies.

Hello @rodrigofraxino. It looks like there's a bug in the generated code which is preventing nextLink from deserializing correctly. I'll update this issue once I fix it.

Any update about this issue?

hi @weidongxu-microsoft could you pls help to check if this is still valid issue in latest monitor sdk? thanks

Already fixed.
            PagedList<EventData> logs = azure.activityLogs().defineQuery()
                    .startingFrom(DateTime.now().minusDays(60))
                    .endsBefore(DateTime.now())
                    .withAllPropertiesInResponse()
                    .filterByResource(...)
                    .execute();

            int count = logs.size();

 size = 221


SystemKeyboardButton should allow content modification

When you create a SystemKeyboardButton, the standard action content or the custom text or image, is wrapped in a button without any modifier in between. This makes it impossible to control things like padding, content size etc.

This is fixed in 4.6.4.


add tests for jax.random shape semantics

add tests for the logic in #1515

Hi @mattjj
I can see that the PR #1515 has been merged with the required tests are being added.
Thank you

Thanks for checking! Actually it looks like I made this issue because #1515 did not include enough testing, so the fact that #1515 was merged doesn't by itself mean this issue is complete.
That said, this is a 4.5 year old issue requesting more tests on one specific feature, and we could probably use more tests on many features. So I'll close this issue as not very useful.
Thanks again!


Integrating yt-dlp with a media library (player / downloader)

Checklist

[X] I'm asking a question and not reporting a bug/feature request
[X] I've looked through the README
[X] I've read the guidelines for opening an issue
[X] I've searched the bugtracker for similar questions including closed ones

Question
Hi Guys and thanks for your work!
I'm developing a media library Flyleaf. I have already integrated yt-dlp with a plugin.
Currently, I'm using --write-info-json and -o options to export .json file and read all the available formats which then I expose to the users. My questions are the following:

Is this the ideal -performance wise- way to extract url formats? (eg. I have notice -j/-J might be better?). Minimal bandwidth/requests required (any options to disable default requests that in my case are not required?)

--write-info-json option doesn't respect playlists? I have noticed however if I give a -o filename with an extension it will create two files and one of them will contain the 1st video in playlist json output

Json url vs manifest_url attributes.  I'm currently using URLs (url attribute from json) from each format and open two different ffmpeg demuxers if required (eg different url for audio and different for video) which makes the audio/video sync more unstable and sometimes urls are not working properly.  I would prefer to have a master m3u8 with all the available audio/video/subtitle streams  parsed to ffmpeg directly. Is this possible or it's based on the extractor?

Is this the ideal -performance wise- way to extract url formats? (eg. I have notice -j/-J might be better?). Minimal bandwidth/requests required (any options to disable default requests that my case don't required?)

--write-info-json writes the data to disk, -j writes to stdout. That is the only difference performance wise (assuming you are using -s)

--write-info-json option doesn't respect playlists? I have noticed however if I give a -o filename with an extension it will create two files and one of them will contain the 1st video in playlist json output

It does. Show your verbose logs. You are probably passing some wrong options

Json url vs manifest_url attributes. I'm currently using URLs (url attribute from json) from each format and open two different ffmpeg demuxers if required (eg different url for audio and different for video) which makes the audio/video sync more unstable and sometimes urls are not working properly. I would prefer to have a master m3u8 with all the available audio/video/subtitle streams parsed to ffmpeg directly. Is this possible or it's based on the extractor?

It depends on the extractor. You can take a look at the downloader code (yt_dlp/downloader/external.py/FfmpegFD) to see how we process the URLs. Instead of passing to ffmpeg, I suggest passing it back to yt-dlp with --load-info-json path/to/file.info.json -f yourformat. You can force ffmpeg using --downloader ffmpeg or stream to stdout if needed using -o -

Thanks for you quick response @pukkandan
Can you tell me what is wrong with the following command and it overwrites the json file when using a playlist?
yt-dlp.exe --no-check-certificate --skip-download --youtube-skip-dash-manifest --write-info-json -o c:\root\t01 "https://www.youtube.com/watch?v=XzLGcLw3cHc&list=RDQM_xhIudLKsjM&start_radio=1"

[youtube:tab] Downloading playlist RDQM_xhIudLKsjM; add --no-playlist to just download video XzLGcLw3cHc
[youtube:tab] RDQM_xhIudLKsjM: Downloading webpage
[download] Downloading playlist: Mix - Deep house
[youtube:tab] RDQM_xhIudLKsjM page 1: Downloading API JSON
[youtube:tab] RDQM_xhIudLKsjM page 2: Downloading API JSON
[youtube:tab] RDQM_xhIudLKsjM page 3: Downloading API JSON
[youtube:tab] First video XzLGcLw3cHc found again; Assuming end of Mix
[info] Writing playlist metadata as JSON to: c:\root\t01.info.json
[youtube:tab] playlist Mix - Deep house: Downloading 79 videos
[download] Downloading video 1 of 79
[youtube] XzLGcLw3cHc: Downloading webpage
[youtube] XzLGcLw3cHc: Downloading android player API JSON
[info] XzLGcLw3cHc: Downloading 1 format(s): 22
[info] Writing video metadata as JSON to: c:\root\t01.info.json
[download] Downloading video 2 of 79
[youtube] pUjE9H8QlA4: Downloading webpage
[youtube] pUjE9H8QlA4: Downloading android player API JSON
[info] pUjE9H8QlA4: Downloading 1 format(s): 22
[info] Writing video metadata as JSON to: c:\root\t01.info.json
[download] Downloading video 3 of 79

I will check the code you suggest for the downloader even if I cannot use yt-dlp generally as I'm using ffmpeg api to have full control.

-o c:\root\t01

You are asking all infojsons to be written to the same path. use -P C:\root\to1 instead to just set the folder


v1.7.0

These changes look fantastic! Excellent work.

Any possibility of getting a pre-release NuGet package to play around with these changes?

Any possibility of getting a pre-release NuGet package to play around with these changes?

I generally don't do pre-release NuGet's but I can release it today!


add new example notebook for knowledge base chat with automatic data filtering

Summary
This notebook illustrates a workflow for knowledge-based chat interactions that utilize automatic data filtering through function calls.
Motivation
It's common to target specific data subsets; for example, in an Earnings Call Transcript dataset (with multiple companies and many calls across whole year), one might only interested in Nvidia's Q3 performance. However, a simple semantic search, even when enriched with date augmentation, often isn't comprehensive enough. This notebook provides a method for precise question-answering using contextually filtered data.
For instance, asking question like "whats Microsoft says about Azure in Q3 that year?" triggers call like {'query': 'Azure in Q3', 'company_names': ['Microsoft Corporation (MSFT)'], 'start_date': '2020-07-01', 'end_date': '2020-09-30'} to first filter dataset for more relevant ones. Before going into the RAG and chat process.
Additionally,  feels we can use more examples on function callings.

Hi @AlexZhangji! Thank you for your contribution. Since this is a new content addition, it'll take us some time to review it. We'll get to it soon!

Hi @AlexZhangji! Thank you for your contribution. Since this is a new content addition, it'll take us some time to review it. We'll get to it soon!

Thank you Simon! no problem at all, and happy to answer any questions.

Hi @AlexZhangji! We just wrote a new set of contribution guidelines for the cookbook, and updated our PR template. I'll review your notebook tomorrow, but in the meantime feel free to look at those resources if you're curious. I'll be referring to them as I review your PR.

Hi @AlexZhangji! We just wrote a new set of contribution guidelines for the cookbook, and updated our PR template. I'll review your notebook early next week (apologies for the delay), but in the meantime feel free to look at those resources if you're curious. I'll be referring to them as I review your PR.

Thank you!

Hi @AlexZhangji, thank you for your contribution! While the topic you've chosen seems relevant and promising, this guide needs improvement around its structure and language. I will be closing this PR for now.

Criteria
Score
Description

Relevance
3
Relevant, but the impact needs to be articulated more clearly.

Uniqueness
3
Unique with minor overlaps.

Clarity
1
Large sections of code are not explained; language is unclear in many cases; content is hard to read and follow.

Correctness
3
Code works correctly, minor improvements needed.

Grammar
1
Multiple grammatical errors, starting with the title, and numerous typos throughout.

Refer to our contribution guidelines for more details on this assessment.

Hi @AlexZhangji, thank you for your contribution! While the topic you've chosen seems relevant and promising, this guide needs improvement around its structure and language. I will be closing this PR for now.
Criteria	Score	Description
Relevance	3	Relevant, but the impact needs to be articulated more clearly.
Uniqueness	3	Unique with minor overlaps.
Clarity	1	Large sections of code are not explained; language is unclear in many cases; content is hard to read and follow.
Correctness	3	Code works correctly, minor improvements needed.
Grammar	1	Multiple grammatical errors, starting with the title, and numerous typos throughout.
Refer to our contribution guidelines for more details on this assessment.

Thank you, Simon, for your in-depth feedback.
I understand the current content may not stand alone as its own notebook, but I think automatic data filtering through function calls presents a feature with broad applicability.
Wonder would it add value to integrate this as an additional section within the existing notebook on function calling with knowledge retrieval? (I can streamline the more extensive portions of my current notebook to focus only on the essential aspects.)
I envision an added segment in the existing notebook that expand upon original content. It will not only illustrate interaction with a knowledge base but also delve into the application of an automated filter.
This would allow for nuanced searches, such as querying the CLIP model,  allow to search CLIP model released this year, or even specifics like CLIP papers published after June in the primary category of computer vision —offering a layer of automated metadata filtering atop natural language queries.

I’m open to seeing a PR for that! If things are more streamlined and
better explained, it could be helpful as a minimal section on that
notebook.
Simón Fishman
Applied AI | OpenAI
On Wed, Oct 18 2023 at 2:37 PM, Ji @.***> wrote:

Hi @AlexZhangji https://github.com/AlexZhangji, thank you for your
contribution! While the topic you've chosen seems relevant and promising,
this guide needs improvement around its structure and language. I will be
closing this PR for now.
Criteria Score Description
Relevance 3 Relevant, but the impact needs to be articulated more clearly.
Uniqueness 3 Unique with minor overlaps.
Clarity 1 Large sections of code are not explained; language is unclear in
many cases; content is hard to read and follow.
Correctness 3 Code works correctly, minor improvements needed.
Grammar 1 Multiple grammatical errors, starting with the title, and
numerous typos throughout.
Refer to our contribution guidelines
https://github.com/openai/openai-cookbook/blob/main/CONTRIBUTING.md for
more details on this assessment.
Thank you, Simon, for your in-depth feedback.
I understand the current content may not stand alone as its own notebook,
but I think automatic data filtering through function calls presents a
feature with broad applicability.
Wonder would it add value to integrate this as an additional section
within the existing notebook on function calling with knowledge retrieval
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb?
(I can streamline the more extensive portions of my current notebook to
focus only on the essential aspects.)
I envision an added segment in the existing notebook that expand upon
original content. It will not only illustrate interaction with a knowledge
base but also delve into the application of an automated filter.
This would allow for nuanced searches, such as querying the CLIP model,
allow to search CLIP model released this year, or even specifics like CLIP
papers published after June in the primary category of computer vision
—offering a layer of automated metadata filtering atop natural language
queries.
—
Reply to this email directly, view it on GitHub
https://github.com/openai/openai-cookbook/pull/753#issuecomment-1769359698,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACCQFFXL2PVXY6NVWJQ7XF3YABDYHAVCNFSM6AAAAAA5QIOMQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRZGM2TSNRZHA
.
You are receiving this because you modified the open/close state.Message
ID: @.***>


Scroller in scrollable is no longer visible when hovering its content

Is there an existing issue for this?

[X] I have searched the existing issues.

Is this issue related to iced?

[X] My hardware is compatible and my graphics drivers are up-to-date.

What happened?
In iced 0.4 the scroller within the scrollbar is only visible when mousing over the content - in 0.5 the scroller is permanently visible with seemingly no way to recreate the behavior of 0.4. This is demonstrable by running the scrollable example for both versions.
What is the expected behavior?
Either the default behavior of 0.4 is restored, or it is possible to optionally specify this behavior for a scrollable.
Version
master
Operative System
Windows
Do you have any log output?
No response

Thanks for the quick fix!

I just want to add that now the scroller is not visible at all when the scrollable area is not hovered (I'm using v0.6.0).
Could this be solved? I feel it's useful to see the scroller even when not hovering the scrollable content.


restyled: break README formatting [test]

per https://github.com/iterative/dvc.org/pull/3543#issuecomment-1125195529 cc @julieg18

OK Restyled isn't failing here either... Although it's also not making any changes? Weird

Can't check what's happening rn...


DEBUG: Run some stress tests

Stress test the following

[ ] Number of Bullets (this will test both moving objects and collision detection system)
[ ] Number of peers that can connect until the game starts to get unplayable
[ ] Bullet Damage, how many bullets can hit the player at the same time without causing any issues
[x] Max ping (input delay) until game becomes unplayable

The max ping is somewhere around 100ms. Technically, there the limit is much higher (you can also play with 800ms ping), but it is already unplayable for >100ms ping.


weird error message, if the output directory doesn't exist

hi, thanks for this cool tool.
when runnning windows executable against existing input directory (but without existing output one), it fails with the message: invalid outputpath extension type. it's not a big deal to create output directory beforehand manually, but it really took a while to understand first time what was the actual problem.
it would be very helpful either to change that message to something more meaningful, or (better) to add output directory auto-creation.

Thanks for your feedback. I will fix it later. 😄

@eps696
I have fixed it. The new release will create the folder if it does exist.
See https://github.com/xinntao/Real-ESRGAN/releases/tag/v0.1.2


[store] Error in the Store UI

When signing in the store, even without providing the admin-admin credentials I was automatically logged in as Carbon super Admin.
Screenshot:

Were you logged in to the publisher app on the same browser? This is the behavior of SSO


Using genymotion instead of AVD

Hi
How can I use Genymotion instead of AVD? While Genymotion is runing but AVD not runing I get build error but while AVD is runing all thnigs is OK!
I want to replace Genymotion with AVD completly, Is it possible?
I'm usnig Win 8

in the Genymotion:
Settings->ADB->Use custom Android SDK tools, choose the adb.exe in your Win8 folder
or you could override the adb.exe with the Genymotion's, that may works for you.

@saubcy Thanks, It works for me!


clean unused trace json

i think these 2 traces are not maintained by anybody?
If you two both approve, i can delete them

I think we can remove legacy.
But maybe we can still keep masterchef_0?

masterchef_0  is the first step of staking.
masterchef is about cacalutaing rewards and withdrawing

I think we can remove legacy.
But maybe we can still keep masterchef_0? (It's not being used/tested, and it's already out-of-date though)

ok. I will update masterchef_0 to a new one


docs(MAINTAINERS): add Kent Rancourt as a core maintainer

LGTM.

LGTM :smile:

LGTM

Technically that doesn't count until this PR is merged... ;)

I killed the build so we're not wasting resources to test this beyond docs.

Since it's a markdown file, it's not even touched by docs tests. So let's merge this.


VPC-SC scan (feature request)

This is a feature request for having a scanner that checks VPC Service Controls to ensure the configuration is correct/appropriate.
Some basic checks like:

only whitelisted services are included in the perimeter
appropriate Access Levels are included in the perimeter
only projects from our organization are included in the perimeter

It may make sense to have this scan work in conjunction with an Access Context Manager scan, since Access Levels and VPC-SC are tightly coupled (and their union defines the actual security controls on the organization).

@blueandgold @ryanismert Can you please review this feature request. I believe we need a high level 1 pager that describes the overall scanner behavior as to when to assert a match.

Draft 1 pager with details to scope this work: https://docs.google.com/document/d/14wYLG1IkLXUPf6t9txhdCFiVwXtDnVjN_8QOC9IPw_o/edit


Set the blob name in sas uri

When the sas uri is created for the container the blob name is not specified. By setting the blob name, the sas uri can still be used to copy the blob.
#patch
resolves #116

Tested by creating a SAS url from the Azure portal for the container and importing it in the devTools.

Why do we assume the SAS URL is created for the container, and not the blob? I would assume that we should just create a SAS URL directly for the blob?

There is a Geneva Action creating a SAS token for an environment, and the generated token points at the container and not the blob. But if you enter a sas for the blob it would still work.

There is a Geneva Action creating a SAS token for an environment, and the generated token points at the container and not the blob. But if you enter a sas for the blob it would still work.

Ah, so that's how customers/devs will get the workload instances for now, we pass a read-only SAS URL for the contanier wherein the current.json lives of the Hub environment?


⚠ GitLab Server has degraded performance

In 4a45603, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11019 ms

Resolved: GitLab Server performance has improved in d92705f.


[web] Enable SelectableText on web [WIP]

Description
Enables SelectableText widget on web by setting the user-select and pointer-events on the generated picture layers where-in exists the paragraph tags. This works by applying the change as a hint. The notion here is to eventually enable whole layers to have selectability.

Engine PR required for this change: https://github.com/flutter/engine/pull/17888 - [web] Enable selectable text in PictureLayer.

Related Issues

https://github.com/flutter/flutter/issues/47234 - SelectableText can't be copied on web

Tests

Tests to follow if PR approach is approved.

Checklist

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I signed the [CLA].
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[ ] All existing and new tests are passing.
[ ] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read [Handling breaking changes].

[x] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

Any updates on this?

I know this is much less than ideal, but I made my own workaround instead, since this doesn't allow natively copying the selected text. Similar to above, I noticed that the only thing blocking native HTML from doing its thing is user-select: "none" and pointer-events: "none" being set in ` tag.
So I created a simple enable_selection.dart file that removes those lines from the generated main.dart.js file. I'm sure there is probably a reason for it being there, but this works just fine for my use case. Hopefully this helps someone until it can be officially fixed.
/*
* Natively highlighting text is broken on Flutter Web.
* Possible fix at https://github.com/flutter/flutter/pull/55419
*
* This code is a temporary workaround
*/

import "dart:io";

final File jsFile = File("build/web/main.dart.js");
final List<String> lines = jsFile.readAsStringSync().split("\n");

void main() {
  int finished = 0;
  for (int index = 0; index < lines.length; index++) {
    final String line = lines [index];
    if (line.startsWith(r'k.bm(p,"user-select"')) {  // allowing user to select text
      lines [index] = 'k.bm(p,"user-select","auto")';
      finished++;
    } else if (line.startsWith(r'k.bm(p,"-webkit-user-select"')) {  // Safari ^
      lines [index] = 'k.bm(p,"-webkit-user-select","auto")';
      finished++;
    } else if (line.startsWith(r'k.bm(p,"-ms-user-select"')) {  // Edge/IE ^
      lines [index] = 'k.bm(p,"-ms-user-select","auto")';
      finished++;
    } else if (line.startsWith('C.d.V(g,C.d.N(g,"pointer-events")')) {  // catching pointer events
      lines [index] = 'C.d.V(g,C.d.N(g,"pointer-events"),"auto","")';
      finished++;
    } else if (finished == 4) {
      break;
    }
  }
  if (finished != 4) {
    throw StateError("Could not make modifications");
  }
  jsFile.writeAsStringSync(lines.join("\n"));
}

I know this is much less than ideal, but I made my own workaround instead, since this doesn't allow natively copying the selected text. Similar to above, I noticed that the only thing blocking native HTML from doing its thing is user-select: "none" and pointer-events: "none" being set in ` tag.
So I created a simple enable_selection.dart file that removes those lines from the generated main.dart.js file. I'm sure there is probably a reason for it being there, but this works just fine for my use case. Hopefully this helps someone until it can be officially fixed.
/*
* Natively highlighting text is broken on Flutter Web.
* Possible fix at https://github.com/flutter/flutter/pull/55419
*
* This code is a temporary workaround
*/

import "dart:io";

final File jsFile = File("build/web/main.dart.js");
final List<String> lines = jsFile.readAsStringSync().split("\n");

void main() {
  int finished = 0;
  for (int index = 0; index < lines.length; index++) {
    final String line = lines [index];
    if (line.startsWith(r'k.bm(p,"user-select"')) {  // allowing user to select text
      lines [index] = 'k.bm(p,"user-select","auto")';
      finished++;
    } else if (line.startsWith(r'k.bm(p,"-webkit-user-select"')) {  // Safari ^
      lines [index] = 'k.bm(p,"-webkit-user-select","auto")';
      finished++;
    } else if (line.startsWith(r'k.bm(p,"-ms-user-select"')) {  // Edge/IE ^
      lines [index] = 'k.bm(p,"-ms-user-select","auto")';
      finished++;
    } else if (line.startsWith('C.d.V(g,C.d.N(g,"pointer-events")')) {  // catching pointer events
      lines [index] = 'C.d.V(g,C.d.N(g,"pointer-events"),"auto","")';
      finished++;
    } else if (finished == 4) {
      break;
    }
  }
  if (finished != 4) {
    throw StateError("Could not make modifications");
  }
  jsFile.writeAsStringSync(lines.join("\n"));
}

Hi, How do you use this script ? you have some kind of pre-start/post-build hook to override this configurations?  Anyway thanks on finding a workaround :D

Ah, sadly no. I've never really dabbled in hooks before, I just manually run the script after the build (to modify the resulting main.dart.js file. But since it's literally just two commands, I'm sure it's pretty trivial to make a script. Here would be the Batch (Windows) script:
flutter build web
dart enable_selection.dart

/cc @mdebbar is this functionality already covered?

Yes, this PR and its sister PR can be closed.


Update version and switch to PyPI

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Bot is broken so manual update it is.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender


add support for app state argument

Fixes https://github.com/auth0/auth0-flutter/issues/486

[x] All new/changed/fixed functionality is covered by tests (or N/A)
[x] I have added documentation for all new/changed functionality (or N/A)

📋 Changes

This PR adds support for the appState parameter to loginWithRedirect() for the web.
See https://github.com/auth0/auth0-spa-js/blob/f2e566849efa398ca599daf9ebdfbbd62fcb1894/src/global.ts#L298
There is prior art in https://github.com/auth0/auth0-angular/pull/168
🎯 Testing

Added 1 unit test. This can be tested end-to-end by setting up a login with redirect, that uses the app state argument as parameter, and then validating that the app state is returned when the user is redirected back.

It seems I still missed a step, as the app state is not passed to the redirect url when actually coming back to the app. Once that is fixed, this is ready for review.

Failing test was fixed, so this is ready for review!


Can I configure a built-in field without removing it and adding it back?

Following the Guide it seems that to configure the email field, for example, I'll have to remove it and add a new one. By doing that I lose the display order I had initially. If I want to keep this order am I forced to remove all built-in fields and add them back in the order I wish? It's seems the only two options from the Guide.

You might try to trick the existing object within AccountsTemplates._fields:
var email = _.filter(AccountsTemplates._fields, function(f) {return f._id == 'email'});
if (email) {
  email.errStr = 'Check the address!';
}

but I'm not sure whether to suggest something like this or not ;-)


apicaller_get returns only 3998 characters of the web api...

hi again,
it looks like, at least from my end, that i'm only getting 3998 characters back from the query. I've tried using SSMS, SQLCMD, as well as OSQL, but they don't seem to give me back the entire object. have you tried this and if so how do you test if the entire object or string is returned?
tks
Lawrence

@lawrenceong001  Could you give a try?

let me grab the latest and give it a whirl -- tks


cmd/go: define command-line API for tools

As per discussion in #18653.
https://github.com/kisielk/gotool has existed for quite some time as a way for external tools to be able to resolve import paths such as . and ./.... See its godoc: https://godoc.org/github.com/kisielk/gotool
This functionality is implemented in cmd/go but it's not exported and available to Go programs, so gotool does exactly that - it copies the source code and makes it available in an importable package.
@rsc mentions that external tools should instead depend on the go executable:

There is no intent to expose the internals of the go command for direct import by programs that want to behave like it. I would prefer if those programs invoked the go command to find out what they need to find out: the API is the binary itself, not the Go package. Using the binary means that when some detail of the go command changes, all those other tools don't need to be updated. I'd be happy to talk about what those tools need.

I assume ImportPaths would be replaceable by go list in most scenarios.
This issue exists to figure out if the command-line tool needs any changes to accomodate for the gotool use case.
cc @kisielk who wrote the package
cc @mdempsky @dominikh @shurcooL @davecheney @FiloSottile who use the package (aside from myself)

Ignoring @rsc's comment for a second, my initial thought as an external tool developer is that using the go executable would have a couple of drawbacks:

The tool now depends on go being available and being the same Go version as the external tool (or at least compatible)
Boilerplate code is needed to do the command execution and parse its output (going from a one-package-per-line stdout to []string, for example)

Those two can be taken care of in the external tool, but the whole point of gotool is to minimize code duplication and margin for error.

This issue exists to figure out if the command-line tool needs any changes to accomodate for the gotool use case.

As someone who relies on and helps maintain gotool, I don't think you need to worry about this. We'll take care of accommodating for any changes in cmd/go.
I highly prefer to import a pure Go library in order to achieve go list functionality than try to execute go binary. I've read Russ's https://github.com/golang/go/issues/18653#issuecomment-276984685 and I understand his POV. I think he's saying that "by default, use the binary, doing that is supported by us, anything else is not; however, if you're willing to spend more effort maintaining a library that copies the code, we can't stop you". I am happy to maintain gotool so that I can continue to use it, and I'll gladly update it if needed to achieve parity with cmd/go. It's not very hard.
The gotool is a little like unsafe. You shouldn't create/maintain it unless you know what you're doing and willing to deal with the consequences.

@mvdan, @shurcooL, what's the status of this bug?

This issue came to exist because of the following exchange in #18653:

rsc: There is no intent to expose the internals of the go command for direct import by programs that want to behave like it. I would prefer if those programs invoked the go command to find out what they need to find out: the API is the binary itself, not the Go package. Using the binary means that when some detail of the go command changes, all those other tools don't need to be updated. I'd be happy to talk about what those tools need.
mvdan: this sounds like a good idea. Should I open a separate issue and ping people who'll probably be interested?
rsc: Sure, please open a new issue 'cmd/go: define command-line API for tools' and please list there the kinds of things your tools need to find out from the go command. Thanks.

And, as this issue says:

This issue exists to figure out if the command-line tool needs any changes to accomodate for the gotool use case.

From the discussion here so far, as far as I can tell, the answer is that nothing in cmd/go needs to change.
So it can be closed.

Thanks. SGTM.

I agree that it seems like there's nothing to do - the cmd/go bits won't be importable anytime soon, and some people in the community are happy to keep a copy of some of its code separately.
I was keeping this issue open to see if @rsc had anything to add or comment, since he was the one who proposed opening this issue.


[Human Readable IDs] Generate random Human readable IDs and pass them to the forms for usage

Feature Context

OpenSRP maintains a list of unique human-readable IDs assigned to different clients for identification.
We want to build an MVP for this on FHIR core. This initial piece will look at generating a set of numbers that can be read by health practitioners to identify the patients.
We will then use the generated number on the forms by passing it as follows https://github.com/opensrp/fhircore/blob/main/android/quest/src/main/assets/configs/app/profiles/inventory_profile_config.json#L159-L174

Feature Implementation

[ ] In the RulesFactory Class, write a method that generates a set of whole numbers (min 6).
[ ] This method should accept a parameter that describes the length of the number.  If the parameter is not available, generate a number with a length of 6.

Feature acceptance criteria

[ ]  The method should be able to be called from the configuration rules.
[ ] The method should generate numbers with the specified length or a min length of 6 when the length is not user specified.

More context here https://github.com/opensrp/fhircore/discussions/872

what does min 6 mean? If I pass a parameter saying I want an ID of length 4, will it 1) throw an error 2) give me back an ID of length 6, 3) give me an ID of length 4?

I'm not sure from a crypto perspective, or from performance perspective, whether it's better to generate (for example in the length 6 case wlog) 1 random number from 0 to 999999 or to generate 6 random numbers from 0 to 9 then concatenate them.
Maybe it doesn't matter, but if we go with the former approach we need to make sure to pad the returned value with 0s so it's of a length six.
Which leads to another question, is the return type a string? That's what I assumed since IDs are normally strings, but the issues says "a number"

what does min 6 mean? If I pass a parameter saying I want an ID of length 4, will it 1) throw an error 2) give me back an ID of length 6, and 3) give me an ID of length 4?

@pld great questions. To guarantee some sort of uniqueness. Without 0 and repeats the total of 6-digit numbers is 9^6
I think we can allow any size when we build the full feature that accounts for more uniqueness.
I am tempted to say we can remove the ability for the user to define the number size.

I'm not sure from a crypto perspective, or from performance perspective, whether it's better to generate (for example in the length 6 case wlog) 1 random number from 0 to 999999 or to generate 6 random numbers from 0 to 9 then concatenate them.
Maybe it doesn't matter, but if we go with the former approach we need to make sure to pad the returned value with 0s so it's of a length six.
Which leads to another question, is the return type a string? That's what I assumed since IDs are normally strings, but the issue says "a number"

@pld probably a small tweak of the second approach might it faster. We already know the least 6-digit number we can get between 0 - 9999999 is 100000. We can then just write a method to generate a random number between 100000 & 999999
The OpenSRP ID on the forms expects a number.  Based on this I think it's appropriate to return an integer.

what does min 6 mean? If I pass a parameter saying I want an ID of length 4, will it 1) throw an error 2) give me back an ID of length 6, and 3) give me an ID of length 4?

@pld great questions. To guarantee some sort of uniqueness. Without 0 as the first digit and any repeats, the total of 6-digit numbers is 9*10^5
I think we can allow any size when we build the full feature that accounts for more uniqueness.
I am tempted to say we can remove the ability for the user to define the number size.

Alright cool, so for now, no selection of length, this is to generate a six digit integer randomly chosen from the interval [100,000, 999,999]

Perhaps we could try a Hybrid Approach that involves both the Client and the Server and is hopefully simple, efficient and guarantees uniqueness.
See my suggested approach in the discussion here

Perhaps we could try a hybrid approach that involves both the Client and the Server and is hopefully simple, efficient and guarantees uniqueness.
See my suggested approach in the discussion here

Perhaps we could try a hybrid approach that involves both the Client and the Server and is hopefully simple, efficient and guarantees uniqueness.
See my suggested approach in the discussion here

Yeah we'll do that in a follow up, out of scope for this issue


Avoid emojis that don't render well on older systems

Proposed documentation enhancement
Avoid recent emojis that do not render on older but still maintained systems. Here are examples from an Ubuntu 18.04 machine with Google Chrome.

Screen capture of the “New issue” page

U+1F41B Bug from .github/ISSUE_TEMPLATE/bug_report.yml
U+1F4C4 Page Facing Up from .github/ISSUE_TEMPLATE/documentation.yml
U+1F6E0 Hammer and Wrench from .github/ISSUE_TEMPLATE/feature_request.yml
U+1F4AC Speech Balloon from .github/ISSUE_TEMPLATE/config.yml

Screen capture of the “Documentation” issue page

U+1F4C4 Page Facing Up from .github/ISSUE_TEMPLATE/documentation.yml

I'm fine with removing these emojis.

👌

Message ID: @.***>

Yes please remove them. cc @hoechenberger: this is why I'm often pushing against adding emoji everywhere

Do what you must, but I'd argue that system is not being properly maintained if it cannot even display Unicode characters correctly 🤷♀

You are not wrong. In addition, this Ubuntu release is EOL in May, so we should not have to care.

None of NumPy, pandas, or scikit-learn use emojis in this context, either, so +1 for removing from me

I must stress that my Ubuntu Ubuntu 18.04 machine fails to render some emojis (mostly the 4 codepoints I had listed above), not all emojis. While I am supposed to be working on a properly configured Ubuntu 18.04 workstation, I certainly cannot vouch for the IT subcontractor in charge of installing these workstations.
That said, we are in the process of moving to Ubuntu 22.04. Hopefully I won't mind very soon.
But wait! After reading Emoji can not show in Ubuntu 18.04 and installing the fonts-noto-color-emoji package, it works for me.

Nice, so let's leave them in then. Again, 18.04 will be EOL in May, and we normally don't support deprecated platforms.

Just nitpicking, but Ubuntu 18.04 won't be totally EOL before 2028. Extended Security Maintenance (ESM) will remain available until April 2028.

Only if you pay.


Is it possible to insert keywords in a command syntax?

Hi,
I would like to know if it's possible to declare a command syntax like this one (for example):
rename <file> to <new_file>
How to declare the "to" mandatory keyword in that case?
Thanks for your excellent job!

What I would do is something like this:
@Command(name = "rename")
class Rename implements Runnable {
    @Parameters
    File file;

    @Option(names = "to", required = true)
    File new_file;

    public void run() { // business logic here
    }
}

This makes the to option mandatory. The following user input is valid:
rename <file> to <new_file>
rename to <new_file> <file>

This is invalid:
rename <file>
rename <file> <new_file>
rename to <new_file>

Thanks. Almost perfect, except for the second user input that I would prefer to be invalid in my case:
rename to <new_file> <file>
However I would happily accept your solution if the usage was encouraging the preferred user input. Unfortunately this is not the case, as it shows:
Usage: rename to=<new_file> <file>
Is there a way to make it showing the preferred syntax?
Usage: rename <file> to=<new_file>

I found another solution, still not perfect but closer to what I'm looking for:

@Command(name = "rename")
class Rename implements Runnable {

    enum ToKeyword {to}

    @Parameters(description = "File to rename.")
     File file;

    @Parameters(paramLabel = "to")
    ToKeyword to;

    @Parameters(description = "New file name.")
    File newFile;

    public void run() { // business logic here
    }
}

Regarding the user input, it's perfect as it only accepts the one I wish :smile:
The only point is now with the usage:
Missing required parameters: '<file>', 'to', '<newFile>'
Usage: rename <file> to <newFile>
      <file>      File to rename.
      to
      <newFile>   New file name.

I would prefer to hide the 'to' parameter from the usage and only get:
Missing required parameters: '<file>', '<newFile>'
Usage: rename <file> to <newFile>
      <file>      File to rename.
      <newFile>   New file name.

Is it possible?

This is tricky... You want to include it in the synopsis, but exclude it from the error message and from the list of options-with-descriptions.
Both this approach and the earlier approach I suggested may require customizing the usage help message.
There are some examples of how to do this in the picocli-examples module, under customhelp and under synopsis.
One idea is to take your earlier comment:

Is there a way to make it showing the preferred syntax?
Usage: rename <file> to=<new_file>

I have pushed an example. Can you take a look?

Thanks for the trick. Your solution is acceptable regarding my little example as the synopsis now shows the preferred syntax. :+1:
Now I would like to use it for my real-world case, which is a bit more complex and I'm afraid it won't be easy to generalised.
Mainly because I'm using sub-commands and each sub-command may have its own keywords and syntax order.
So this would require writing a custom help factory for each sub-command (but I'm not sure it's actually possible in Picocli), or a big help factory at the command level that manages all these cases (which will quickly become complex).
I think a simpler solution is a mix between your solution with the custom help factory at the command level and my second proposition with the keyword declared as a single possible value enum. Because in that case I already completely control the order of the parameters and keywords using standard Picocli declaration for each of my sub-command. Also the synopsis is already good. Only the list of parameters shown above and below the synopsis should omit the keywords, and maybe this is something that can be written in a generic way with a help factory.
So is it possible to write such a help factory at the command level? Its function would be to remove all keyword parameters from the help usage (except for the synopsis which is already good like it is now). And a keyword parameter can be recognised in a generic way as being declared as an enum which contains only one value (like my ToKeyword example).
If this is achievable, I think it will be just perfect :smile:

Okay, I added one more example that uses your preferred approach.
But you get the idea: you need to use the Help API to customize the usage message format.
Then you need some way to determine which parameters to hide and which to display. In the example I created a custom IExcluded interface, and I hide positional parameters whose type implements this interface from the parameter list in the usage help - and I don't hide them from the synopsis. Hopefully this is generic enough for your actual real-life use case.
If you need this to be different per command, you can obtain the CommandSpec from the PositionalParamSpec or OptionSpec, and apply separate logic per command in your custom Help class.
I hope this gives you enough pointers to get a start with the Help API and to figure out which methods to override to get the desired result. It may take some time to familiarize yourself with this API, but hopefully you can achieve the desired output.
Enjoy picocli! 😉

Amazing! And it works with Sub commands like I expected :+1:
That's getting very closed to what I want, thank you :smile:
There are 2 little things I would like to add but I haven't found the way to do it:

change the parameter color from orange to white in the synopsis when it's a keyword parameter
remove the keyword parameter from the error message when listing the required parameters

But perhaps it's not customizable?

I found how to do the first point (changing the parameter color in the synopsis) by overriding createDetailedSynopsisPositionalsText()
class MyHelpFactory implements IHelpFactory {
    @Override
    public Help create(CommandLine.Model.CommandSpec commandSpec, Help.ColorScheme colorScheme) {
        return new Help(commandSpec, colorScheme) {
            @Override
            public String parameterList(List<PositionalParamSpec> positionalParams) {
                return super.parameterList(filter(positionalParams));
            }

            private List<PositionalParamSpec> filter(List<PositionalParamSpec> optionList) {
                List<PositionalParamSpec> shown = new ArrayList<>();
                for (PositionalParamSpec param : optionList) {

                    // exclude parameters whose type has the IExcluded marker interface
                    if (!IExcluded.class.isAssignableFrom(param.auxiliaryTypes()[0])) {
                        shown.add(param);
                    }
                }
                return shown;
            }

            @Override
            protected Ansi.Text createDetailedSynopsisPositionalsText(Collection<CommandLine.Model.ArgSpec> done) {
                Ansi.Text positionalParamText = ansi().new Text(0);
                List<PositionalParamSpec> positionals = new ArrayList<>(commandSpec.positionalParameters()); // iterate in declaration order
                if (hasAtFileParameter()) {
                    positionals.add(0, AT_FILE_POSITIONAL_PARAM);
                    AT_FILE_POSITIONAL_PARAM.messages(commandSpec.usageMessage().messages());
                }
                positionals.removeAll(done);
                for (PositionalParamSpec positionalParam : positionals) {
                    positionalParamText = concatPositionalText(" ", positionalParamText, colorScheme, positionalParam, parameterLabelRenderer());
                }
                return positionalParamText;
            }
            Ansi.Text concatPositionalText(String prefix, Ansi.Text text, Help.ColorScheme colorScheme, PositionalParamSpec positionalParam, IParamLabelRenderer parameterLabelRenderer) {
                if (!positionalParam.hidden()) {
                    Ansi.Text label = parameterLabelRenderer.renderParameterLabel(positionalParam, colorScheme.ansi(),
                            // Testing if the parameter is a keyword parameter
                            IExcluded.class.isAssignableFrom(positionalParam.auxiliaryTypes()[0]) ?
                                    colorScheme.commandStyles() :   // yes => will appear as white (like for commands)
                                    colorScheme.parameterStyles()); // no => will appear with standard parameter color
                    text = text.concat(prefix).concat(label);
                }
                return text;
            }
        };
    }
}

So basically I made a copy/paste of the original code of this method, but in concatPositionalText() I use the command styles (white color) for keyword parameters (the original code was using the parameter styles for all parameters). It works! Me happy :smiley:
Now for the last detail (removing the keyword parameter from the error message when listing the required parameters), I identified where the code was located: this loop in createMissingParameterMessage():
            for (PositionalParamSpec missing : missingList) {
                if (missing.arity().min > 0) {
                    names += sep + "'" + missing.paramLabel() + "'";
                    indices += sep + missing.index();
                    sep = ", ";
                    count++;
                }
            }

However it happens in a deep calls chain of private methods so I can't override the code like previously...
Is the only solution to patch your code (I was using Picocli as an external dependency so far)?

Hmm, ask yourself: is it okay for users not to specify to?

No, but not typing to can be seen more like a syntax error than a missing parameter.
The idea was to treat these keyword parameters more like commands than standard parameters.
But no worries, and what you said also makes sense, it's just a matter of opinion.
I'm happy with what we already did, that's far good enough.
So I will now stop making your life difficult :wink:
And thank you so much for having made Picocli :pray:

Great, glad to hear that!
Feel free to ask more questions whenever you want!
Enjoy picocli!
:-)


Installation procedure

Could you please specify the complete building or installation steps?
I am trying install this on raspbian.
Finding difficult to install boost on it.
Even after installing libboost could not able to find b2 binary
I am new to boost. Please help me

Hi the steps I did are:

download boost and copy the bjam binary to the root of this project.
Installing boost is hard so you probably want a package manager to install something named like boost-dev.
In ubuntu you download it through apt-get install libboost-dev.
Run bjam toolset=gcc cxxflags="-std=c++11" in the root directory.

Hello!
I can‘t complite it on windows.There is error.
xuan@LAPTOP-O8SDI23U MINGW64 /d/resource/bootstrap-dht (master)
$ b2 -sBOOST_ROOT=D:\resource\boost_1_75_0 toolset=gcc cxxflags="-std=c++11"
Unable to load B2: could not find 'boost-build.jam'
---------------------------------------------------
Attempted search from 'D:\resource\bootstrap-dht' up to the root at 'D:\resource\boost_1_75_0\tools\build\src\engine\b2.exe'
Please consult the documentation at 'https://boostorg.github.io/build/'.

What should I do?
Thanks!


Important Note

Hey everyone!
This is an important note so please consider it before making any pull request.
Whenever you have to make a PR make sure you save or download the file on the same day of making the PR otherwise it creates inconsistency in the project repository. The changes that have been made lately/recently might not reflect after merging the PR made from the previous version of the repo.
And keep contributing!
Thank you.

please assign this issue to me

I would like to work on this, Please assign it to me


[Feature] Add support for audit log writing to data streams

I am using opensearch 2.11.
My auditlog config:
plugins.security.audit.config.index: opensearch-security-auditlog
plugins.security.audit.type: internal_opensearch

"opensearch-security-auditlog" is a datastream.
When opensearch is storing auditlog to the datastream, i get the following error:
java.lang.IllegalArgumentException: only write ops with an op_type of create are allowed in data streams

How can i change the op_type of the auditlogs?

[Triage] Hi @tmanninger, thank you for filing this issue. Right now we do not support the data stream API to allow this use case. Let me mark this as a feature request since it will involve adding this support to the audit logging.
That being said, sounds like we can close this issue when:

[ ] Add data stream audit log support
[ ] Add associated tests
[ ] Add documentation

Are complete.

Any Feedback, any roadmap available for this future request?

Hi @tmanninger, I am not aware of any active efforts to provide this support. If you are eager to see this feature implemented the fastest route would likely be for you to open a pull request adding the feature. Feel free to open a PR with the change and I will try to review it as quickly as I can.

How should i integrate this feature?
Add an option
plugins.security.audit.config.is_datastrean: true
or
plugins.security.audit.config.op_type: create
?

@scrawfor99 i created an PR:
https://github.com/opensearch-project/security/pull/4257
I changed to OP_TYPE to create, because audit logs should never be updated.
It's tested and works in my test environment.
Looks this ok?

Released in 2.18.0: https://github.com/opensearch-project/documentation-website/pull/8356


update to use 0.12 syntax

@ryanking I'll leave this one up to @mckinsel and @calvinnhieu

Pushed a few fixes to resolve make apply errors. Thanks for the upgrade @ryanking ! Will wait for @mckinsel 's approval


Unhandled Exception with GUI despite multiple un- and re-installs

Checklist

[X] I have verified this is the correct repository for opening this issue.
[X] I have verified no other issues exist related to my problem.
[X] I have verified this is not an issue for a specific package.
[X] I have verified this issue is not security related.
[X] I confirm I am using official, and not unofficial, or modified, Chocolatey products.

What You Are Seeing?
I try to open the GUI, and it throws a very large unhandled exception. I tried un- and re-installing the GUI, chocolatey, and various packages, but nothing seems to work.
What is Expected?
The GUI should open and load packages as normal
How Did You Get This To Happen?
I updated Firefox to 116 from within the GUI, and that's when the issue started.
System Details

Operating System: 10.0.22621.0
Windows PowerShell version: 5.1.22621.1778
Chocolatey CLI Version: 2.2.0
Chocolatey Licensed Extension version: Chocolatey v2.2.0
Chocolatey License type: ?
 Line 1, position 1.

   at System.Xml.XmlTextReaderImpl.Throw(Exception e)

   at System.Xml.XmlTextReaderImpl.ThrowInvalidChar(Char[] data, Int32 length, Int32 invCharPos)

   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()

   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()

   at System.Xml.XmlReader.MoveToContent()

   at System.Xml.XmlReader.IsStartElement(String localname, String ns)

   at chocolatey.infrastructure.services.XmlService.<>c__DisplayClass5_0`1.<Deserialize>b__2()

   at chocolatey.infrastructure.tolerance.FaultTolerance.TryCatchWithLoggingException[T](Func`1 function, String errorMessage, Boolean throwError, Boolean logWarningInsteadOfError, Boolean logDebugInsteadOfError, Boolean isSilent)

   at chocolatey.infrastructure.services.XmlService.<>c__DisplayClass5_0`1.<Deserialize>b__1()

   at chocolatey.infrastructure.synchronization.GlobalMutex.Enter[T](Func`1 func, Int32 timeout)

   at chocolatey.infrastructure.tolerance.FaultTolerance.Retry[T](Int32 numberOfTries, Func`1 function, Int32 waitDurationMilliseconds, Int32 increaseRetryByMilliseconds, Boolean isSilent)

   at chocolatey.infrastructure.services.XmlService.Deserialize[XmlType](String xmlFilePath, Int32 retryCount)

   at ChocolateyGui.Common.Windows.Services.ChocolateyService.<GetOutdatedPackages>d__13.MoveNext() in C:\buildAgent\work\774e9aed9020e028\Source\ChocolateyGui.Common.Windows\Services\ChocolateyService.cs:line 121

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at ChocolateyGui.Common.Windows.ViewModels.LocalSourceViewModel.<CheckOutdated>d__81.MoveNext() in C:\buildAgent\work\774e9aed9020e028\Source\ChocolateyGui.Common.Windows\ViewModels\LocalSourceViewModel.cs:line 494

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at ChocolateyGui.Common.Windows.ViewModels.LocalSourceViewModel.<LoadPackages>d__80.MoveNext() in C:\buildAgent\work\774e9aed9020e028\Source\ChocolateyGui.Common.Windows\ViewModels\LocalSourceViewModel.cs:line 470

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at ChocolateyGui.Common.Windows.ViewModels.LocalSourceViewModel.<OnInitialize>d__78.MoveNext() in C:\buildAgent\work\774e9aed9020e028\Source\ChocolateyGui.Common.Windows\ViewModels\LocalSourceViewModel.cs:line 412

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)

   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

   at System.Windows.Threading.DispatcherOperation.InvokeImpl()

   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Windows.Threading.DispatcherOperation.Invoke()

   at System.Windows.Threading.Dispatcher.ProcessQueue()

   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)

   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)

   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)

   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)

   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)

   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)

   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)

   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)

   at System.Windows.Application.RunDispatcher(Object ignore)

   at System.Windows.Application.RunInternal(Window window)

   at ChocolateyGui.App.Main(String[] args) in C:\buildAgent\work\774e9aed9020e028\Source\ChocolateyGui\App.xaml.cs:line 122
---------------------------
OK
---------------------------

@Zerrvr it's been a while since this issue. I'm unable to replicate it. Are you still having this issue with the latest version of ChocolateyGUI?

@Zerrvr could you please reopen the issue?
I was able to unintentionally reproduce it. In my case AppData\Local\Chocolatey GUI\outdatedPackages.xml got broken after a sudden power off and contained only zero bytes in it.

@justdanpo this issue was closed automatically 2 months ago. Please raise a new issue and complete all sections to allow us to capture the required information.


flux-commands: document flux cancel

Problem: We're trying to promote use of flux cancel instead of flux job cancel.
Update cheat sheet to use flux cancel instead of flux job cancel.

noticed an english error, just tacked a commit on top for it


bug in infinite scroll

The inifinite scroll with contributors is going out of frame at a perticular width.

I can make it responsive and fix the bug.
@akshitagupta15june please assign it to me under the label GSSOC 2023.

The homepage is already under some UI changes, issues like responsiveness will be fixed soon.
Pl find another issue to work on.


Add support for banana linux

.NET can't source-build on a distro it doesn't know (that is: not yet in the rid graph).
This is an impediment when building .NET on new versions of an existing distro, or when building it on a new distro.
The build should handle the unknown rid appropriately.
Package maintainers shouldn't have to become rid graph maintainers.
Tasks:

[ ] runtime: support building an unknown rid (https://github.com/dotnet/runtime/pull/74504)
[ ] runtime: make the built runtime know about the rid (https://github.com/dotnet/runtime/issues/53550)
[ ] update source-build/installer
[ ] add test

@dotnet/source-build-contrib

@crummel @dagood we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?
cc @omajid

@crummel Chris Rummel FTE @dagood Davis Goodin FTE we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

I think dotnet-install.sh falls back to linux-x64 rid correctly.
For example, arch-x64 RID doesn't exist in the RID graph, but we get past dotnet-install.sh on Arch Linux just fine: https://github.com/dotnet/source-build/issues/1310.

It seems better to me to look at it from a dev's point of view: what do I expect source-build to do when it can't recognize my OS? My understanding is that the point of adding banana (which I think is a really good idea) is to find situations like that.
For the prebuilt SDK, I imagine I'll normally need linux-x64 (glibc). Then I'd probably want a non-portable build with my new RID in the right place in the RID graph that allows me to restore Microsoft assets if necessary for cross-RID publish, etc. Specifying those each time I want to build seems reasonable, because my OS is unknown (not mentioned in the sources).

I like this idea, it'll keep us on top of new versions and unknown distros.  It does seem like we'll probably have to soft-patch the RID graph though, unless something in the runtime changes to allow arbitrary unknown RIDs.  We can try this out and see what all breaks.

we'll probably have to soft-patch the RID graph

I thought we were already doing that with:
https://github.com/dotnet/source-build/blob/be813dfe3b144fb6e389e07d8de3a7dbc92d0acb/repos/corefx.proj#L97-L107

Oh, I think IsBootstrapping went away when we switched from BuildTools to Arcade.  That should be an easy fix though.

(Edit: Sounds like I may only need to specify the RID I want, or not even that.)

Yes. Source-build should figure out the target rid from /etc/os-release and assume a parent of linux-{arch} (on glibc Linux).
As optional parameters, the user may want to name the target rid (e.g. rhel.8 instead of rhel.8.1)
And specify a list of parent rids.
For this issue, I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

@dagood @crummel can we do this?

The approach sounds good to me, if that's the question. I don't know where it sits in the backlog... seems weird that @leecow is assigned and it's in a closed ZenHub epic. @crummel do you know the background, should it be unassigned to retriage or something like that?

Yeah, we'll assign this to someone in triage.  Thanks for the discussion around how to do this everyone!

@crummel @dagood having this CI build will catch rid issues that are now hitting us post-release.

And considering https://github.com/dotnet/source-build/issues/1428, we should build the source-tarball on banana Linux, and verify it builds on pineapple Linux.

Issues have come up with versions--stuff like banana-x64/banana.2-x64 vs. banana.2.9-x64 failing on this regex:
https://github.com/dotnet/source-build/blob/1eba1a673a160ae4635270fdcf823da4bfe66d3d/tools-local/tasks/Microsoft.DotNet.SourceBuild.Tasks.XPlat/PublishCoreSetupBinaries.cs#L26-L31
Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. Also, the SDK seemed to parse os-release as astra.16-x64 at runtime, not sure if this is related (or a problem). Context starting here: https://gitter.im/dotnet/source-build?at=5f21801125cae90e6d8afb48
I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. Also, the SDK seemed to parse os-release as astra.16-x64 at runtime,

I workaround problem by changing /etc/os-release  from  VERSION=1.6 to VERSION=16. So RID in dotnet --info output is defined at compile time

I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

We actually run Ubuntu 18.04 CI, which doesn't hit this, so I'm wrong here. Will need to look deeper.

Something to consider for this issue is patching up the RID graph in the prebuilt/tooling SDK that we download in the tarball production build. It isn't necessary in all cases (the catch-all linux-x64 may work) but it's probably necessary for cases like alpine-x64 / linux-musl-x64. See https://github.com/dotnet/source-build/issues/1868. If we don't do this as part of resolving this issue, we should probably open a new issue to track support for that.

RHEL 9 is Banana Linux: we can't build .NET Core on it until the rids have rippled through :/

Also starting to see issues on the recently-forked Fedora 35 :(

Now that https://github.com/dotnet/runtime/pull/50818 is in.
Can we add a CI job to source-build that verifies we can build a source-tarball on banana Linux, and use it to build .NET on pineapple Linux?
The CI job can write to /etc/os-release to change the distro id.
@MichaelSimons @crummel @dseefeld can you take a look into this?

cc @ericstj

@MichaelSimons @crummel @dseefeld friendly reminder.
It would be great if we can close the circle by adding a CI job to verify source-build is no longer blocked by unknown rids.

I'm looking into this now, I'll hopefully have some x64 RHEL machines soon so I'm planning on making CI changes for that anyway.

I've updated the top-level comment with a task list.
cc @MichaelSimons @crummel @omajid @ericstj @richlander

This is fixed as part of https://github.com/dotnet/installer/pull/14549.
We should add a CI test for it. Preferably it does two builds where the second build consumes the output of the first. For the two builds we should use two different unknown rids (e.g. pinapple.14 and banana.18).
To control the rids, we can make /etc/os-release writable in the container image we intend to use for the test.
We don't need to apply it to all images, as this is a functional test for the unknown rid scenario.
It would be nice if we have both a glibc-based and a musl-based image to test against.
We can update @omajid's PR (https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/679) to enable the test scenario.

This is working. Closing old issue.