Implementation Plan: Staging Elasticsearch Reindex DAGs

Fixes
Resolves #1987 by @AetherUnbound
Assigned reviewers

[ ] @AetherUnbound (Project lead)
[ ] @sarayourfriend (for Elasticsearch expertise and familiarity with the proposed implementation)

Current round

Note
This discussion is following the Openverse decision-making process. Information about this process can be found
on the Openverse documentation site. Requested reviewers or participants will be following this process. If you are being asked to give input on a specific detail, you do not need to familiarise yourself with the process and follow it.

The discussion is currently in the Revision round.
The deadline for review of this round is 2023-06-27.

I will be looking at this IP this week! Sorry for the delay due to my AFK.

@krysal -- I'm happy to review, but just clarifying whether I'm a required reviewer on this, as I'm not named in the PR?

@krysal I will re-review this during my evening today (before your day tomorrow).

@sarayourfriend

I am requesting changes only to ask for clarification about how filtered indexes would apply here. If index settings are changing, for example, would the associated filtered index need those same settings applied?

That's a good question! It looks more pertinent to have been discussed in the previous plan, but since that one is already merged let's do it here 😄 I'd say it depends on the new settings but probably yes, it'd need the same settings applied. Since these DAGs are planned for experimentation initially, how to apply a new configuration permanently will require a separate conversation, evaluating the results obtained.

Or/additionally: should the filtered index be the default source index, to proactively exclude content we've determined is sensitive?

I don't have a strong opinion here. If people want to default to filtered indexes for safety reasons, I'll change to that. I chose the usual media indices because they are our base and have the most data. The DAG runner can also choose a filtered index as source_index for the -proportional-by-provider DAG in the plan's current form.

Ah, that's a really important point you've raised @sarayourfriend. I don't think any of our current plans cover the filtered indices in staging actually 😮 That might be because both this project and the filtering of results on the API were being proposed alongside each other, so the latter hadn't been finalized while we were planning out the former. As it stands, I don't believe we have an explicit mechanism defined for creating the filtered index on staging. Given that this is not production though, perhaps it would make sense to filter the results while building the <media_type>-full index from the database? That would mean all downstream results based on that index are always filtered, and we don't have to worry about managing both the unfiltered & filtered indices on staging. What do you think @krysal?
We've had a lot of discussion about this IP, I wonder if it might make sense to merge it as-is given it's quite close, with a fast-follow clarification about how we'd manage the filtering (given we won't be actually working on the implementation of this for a little bit since the project is on hold). Would you prefer that Krystle?
And as for the automatic deletion, those are great points. I don't think we need to incorporate that aspect of it in the proposal here - we can leave it as an MSR task or something similar!

Given that this is not production though, perhaps it would make sense to filter the results while building the <media_type>-full index from the database? That would mean all downstream results based on that index are always filtered, and we don't have to worry about managing both the unfiltered & filtered indices on staging. What do you think @krysal?

@AetherUnbound The current way of building the filtered index is using the ES's Reindex API, it's not exactly a direct process form from the database. That would require creating a new process but I think we can just trigger the DAG for creating the filtered index as an additional step for building the <media_type>-full index. I'd prefer to have both indexes, the <media_type>-full and <media_type>-filtered, to provide more flexibility and work with the same indexes we have for production. What I am not sure about is if the DAG to create the filtered index is enabled for both environments or if it only applies to production. In the latter case, it would require some modifications. But it would be a really nice addition and actually closer to the part of the data refresh process that we're extracting here 😄

We've had a lot of discussion about this IP, I wonder if it might make sense to merge it as-is given it's quite close, with a fast-follow clarification about how we'd manage the filtering (given we won't be actually working on the implementation of this for a little bit since the project is on hold). Would you prefer that Krystle?

Agree on merging this, as the filtered index is a requirement that was not initially included. I'll change the default source to the filtered index for the subset-by-provider DAG, as it's an easy change. Thank you both for the excellent suggestions.

Sounds good! Yes unfortunately the filtered index DAG is only enabled for production, so we'd need to either have a similar DAG factory for it or set the environment as a DAG parameter (though I think our team preference is for the former to prevent accidental production operations).

Thank you all!


would it be possible to search docs from https://ramdajs.com/ ?

hi, right now you have to click on "docs" to begin a search for a function
rom a  "functional" perspective (no pun intended), to be frictionless,
would it be possible to search docs directly from https://ramdajs.com/ ?
also, just imagine we include the cookbook in the search, maybe below a divider, so people can learn / refer faster to this library, i think it might help us move fast (and not die slow)

Sorry, I missed this when it was posted, and only saw in now during a cleanup.
This sounds like a pretty good idea.  I'm all ears for a suggested implementation, or, even better, a pull request.

Thanks, I realized it’s a super minor issue and we should probably focus on higher yield things. Might be pretty easy but I’m not up to speed on the codebase

I think at least linking to the cookbook in the docs would be a great idea


 I've been working off the multi-step form example here and sandboxing some new features I need as a proof of concept.  The playground example shows a case where an onClick handler doesn't seem to handle being passed an object. Open that example, open the dev console, and click the LogObj button.  The definition for that button is:
    {
      $formkit: 'button',
      onClick: '$log({"nextStep":"bla", "test":2})',
      children: 'LogObj'
    },

The log function is defined in Playground.vue as:
  log: target => () => {
    console.log('target:', target)
  },

In the console I just see: target: {"nextStep":"bla".
At one point onClick was also handling URLs weird for me but now I can't reproduce that in the playground. Is it possible formkit's parsing of variables in schema values is having trouble with certain values/types being used there? Am I just wrong in assuming it can evaluate a JS object? Maybe I need to be passing a JSON string and decoding it if I want to get away with something like the above?
Environment
• OS: MacOS
• Browser: Chrome
• Version: 101.0.4951.64 (Official Build) (x86_64)

Schema does not support object notation — it looks a bit like javascript, but it is very much not javascript — lots of reasons for this, but one of the biggest is we do not use eval for CSP reasons.
You can pass an object to a function, but it needs to be stored in a variable first $log($someObject).

Thanks @justin-schroeder. Do you have any other details, docs, or code you can point me to that lays out the limitations? I'm just thinking ahead to cases where users maybe can not add values to a variable, such as if the form was being completely described and designed from the schema. Does anything in the parsing prevent passing a JSON string representing an object, for example? That seemed to work with the one example I tried.

Yes! The docs on this are pretty comprehensive, and haha, to be clear by comprehensive I mean if its not in the docs, it doesn't do it — not that we have perfect or complete docs 😂. As far as schema "logic" goes, there are basically 4 things you can do:
References: https://formkit.com/advanced/schema#references
Expressions: https://formkit.com/advanced/schema#expressions
Conditionals: https://formkit.com/advanced/schema#conditionals
Loops: https://formkit.com/advanced/schema#loops
References allow you to references data and call functions. Expressions let you perform boolean logic operators, and arithmetic. Conditionals let you make schemas dynamically render some blocks and not others. Loops let you write loops. This combination of items allows nearly Turing complete logic in the schema, but doesn't get into the JavaScript weeds.
The list of things you cannot do in schema that you can do in native JavaScript is pretty much unending, so not worth enumerating.
Nothing prevents passing string literals, those strings could be stringified JSON if you want, but, like with any language you'll need to be careful about all the quotes and escapes and what not. Either way, I would highly recommend having a "data" and a "schema" object in logical implementation of the schema. The best way to think of the schema is as a vue <template>. Vue templates are not dynamic at all, they are compiled down to render functions that cannot be modified after the fact, but you can interact with that template by providing it data and methods, and then using things like v-if and v-for in the template to interact with that data. Exact same story with a schema + data. Think of the schema as a static template that you provide data and functions to and you'll have a much better time with it.

Thanks @justin-schroeder. I'm mostly concerned with the use case of doing everything from the schema, because I want non-engineers to be able to edit form definitions, whether that's tweaking the JSON directly at the start or eventually some UI on top of that which modifies the JSON definition to save in a database/etc.
That said, the use case of needing to pass a custom object in a schema definition is probably something I can avoid (or hide once a UI layer is building the JSON). Maybe this is a bad idea and you could tell me why, but perhaps it would be useful to be able to define a separate data section in the schema which gets merged into the data that is accessible to the rest of the schema. So I could define something like:
const schema = [
  {
    $cmp: 'FormKit',
    props: {
      type: 'form',
      id: 'form',
      onSubmit: '$submit',
      plugins: '$plugins',
      actions: false,
      data: {
         redirectUrl: `https://www.google.com`
      }
    },

And then later in the schema $redirectUrl could be used. This would be better than having to hardcode any repeated string literals that may get used such as if I had a redirect utility called in multiple places/conditions.

Hey @totalhack — separate from any schema modification if you’re just trying to access literal strings such as a hardcoded URL perhaps you could use a group and name it something like “meta” and fill it with type hidden fields that contain those values you could access such as meta.redirectUrl?

Thanks @andrew-boyd, I'll consider that as a workaround -- wouldn't that mean those hidden fields also get posted with the form though? Can just ignore that in most cases, depends how strict the endpoint is about the format being posted too of course.

That’s correct, but you get a non-reactive copy the form data passed to your submit handler so you could omit the meta group from that object before posting if your endpoint is strict about it.

Hey @totalhack — separate from any schema modification if you’re just trying to access literal strings such as a hardcoded URL perhaps you could use a group and name it something like “meta” and fill it with type hidden fields that contain those values you could access such as meta.redirectUrl?

Hey @andrew-boyd, just wanted to note that I took a different approach to this. Instead I just added a "fake" node to the schema that simply holds metadata, and in a wrapper to FormKitSchema I look for that and update a meta field in the data value.  For example:
const schema = [
  {
    type: 'meta',
    data: {
      someTestField: "Hey you can access this elsewhere as $meta.someTestField"
    }
  },

It doesn't seem that formkit minds that I put an invalid node in the schema, it just ignores it. This is a lot cleaner than adding groups and hidden inputs IMO.


Missing assembly references

Hi! I'm getting a lot of ...
"[ERROR:OmniSharp.MSBuild.MSBuildProjectSystem] The type or namespace name `SomeClass' could not be found. Are you missing an assembly reference?"

Edit: These are third party assemblies btw.

I'm also getting this, loads of them. Not too sure what I did but I got one error and a bunch of warnings. The error was also regarding OmniSharp and DNVM/DNX stuff missing, so I followed the link and I have fixed that (I hope) - but all these warnings are present and it seems everything relating to Unity C# is missing (Vector3, StartCoroutine, etc).

IntelliSense is working fine for me. I only have problems with 3rd party referensed libraries (DLLs).

I'll setup a test with third party libraries tonight and get back to you on this :)

Sorry if I'm supposed to make a new issue for this (New to a lot of this stuff) and I'm not sure if mine is similar to what Southgrove experianced.
Here is a screenshot of the OmniSharp Log

and from editor

Hmm, ok so I've got a work around so it works for me, but it looks like I have to do it each time.
I open Unity, and turn off "Enable Integration", I tried turning it back on then opening a script, still nothing - It's only when I try to open a script with integration turned off (It'll open, but no side bar or anything working) and I close it again, and THEN turn back on enable integration, that the scripts will will work properly and I get no warnings of projects not loading.
The moment I close it all, then open Unity again, and double click a script - I am presented with the a screen like above, and have to do the steps all over again to get going.
Some other things I've noticed through this - If I close Unity, after getting Code to work with full integration, and then relaunch Code, everything will still work. But the very moment I open Unity again and go into my Project, it's back to square one.

This is looking like an issue with Mono not being installed outside of the Unity install.
I think I have a fix for that specifically where I'll hint back to the Unity libraries instead.
I'll try to get to this this weekend, etc.

I was able to load a CLR 2 DLL into Unity, and am able to see its classes/etc inside of my test project in Visual Studio Code. :(

Hmm. I've got a bunch of conditional compiling going on. Could that be it perhaps?

I'm not sure that should break everything as far as you've described. The problem for me is that I can't really fully debug the Visual Studio Code side as far/much as I would like.
They don't seem to support conditionals in the code itself, or atleast in showing blocks that are not going to be compiled). However, I would assume that OmniSharp supports the conditionals out of the box.
Can you create a small reproduction project and zip it up for me? send it to reapazor 'at' gmail 'dot' com and I'll double check everything on my end against it exactly.
I also think I ruled out the mono issue, but I updated the README to always suggest installing the latest mono just incase :)

Alright. I'll set up an example project in a couple of hours. :)

Forgot about the project stuff and ended up playing Metal Gear Solid V all night. :)
Will fix test project tomorrow!

 I'm just having some problems recreating the issue, hehe.

Well if you want, im not gonna object.
I understand if you cant though :)

I'll give it another go today with another plugin.

Hey! Sent you an email with a project that fails. =)

I might suggest one other thing.  I had a similar issue during the 0.8 beta.
Tye uninstalling VSCode and installing it fresh, the upgrade may have missed something in the OmniSharp update.

Hi! Same issue here:
[ERROR:OmniSharp.MSBuild.MSBuildProjectSystem] The type or namespace name `SomeClass' could not be found. Are you missing an assembly reference?
I just have mad a fresh install of VSCode.
I'm running OSX 10.9.5, VSCode 0.8.0, Unity 5.2.1f1, mono is up to date, DNX too.
I tried with an empty project and get this log:
[WARNING:OmniSharp.MSBuild.MSBuildProjectSystem] Skipped unsupported project type 'Assembly-CSharp.csproj'
And then it says "Select Project" and there's no project.
thanks for your plugin :)

Hey folks ... if you would be so kind as to try the latest (2.2) and see if that fixes your problem. I was able to reproduce this problem reliably this weekend and come up with a solution (i think).

Ok, initial tests were positive and it seems to be working fine with the new "preferences" VSCode option.
Definitely hiding .meta files but not .sln / .csproj, is this expected? (basically root folder files)

Right now i have it leaving the solution / project files still visible ... I'm asking for verification from the @code team on that. I had some issues when I hid them myself.
What do you mean by sync loss? @code guys said it was fine to link to the CMD :)
Ya I gotta update that PDF for next release.


SelectedNode not working?

Hi, I'm moving my WPF app that was using Nodify to Avalonia and was so far using it without much problems but I now need SelectedNode property and it doesn't seem to work?
EditorView.axaml
<controls:NodifyEditor x:Name="Editor" ... SelectedItem="{Binding SelectedNode}">

EditorViewModel.cs
[ObservableProperty] private NodeViewModelBase _selectedNode;

It stays null the whole time as far as I can see. Is it problem on my side? Or does it need to be fixed?
My project if you need to look into it https://github.com/BartShoot/NoodleCV/tree/main/NoodleCV.App
If you point me in the right direction I could try to contribute PR with the fix

Fixed, if there are any issues. Feel free to reopen the issue


ui signer: add a button to show the psbt to sign as QRcode

For what reason, and for which role do you want to add a Qr code?

CryptoAdvance hardware wallet use Qr code for air gapped communication for example.
This issue is low priority.

We already had this conversation a number of times, but why not restricting the HW interface to wired for the moment?


Round bounds size.

Bounds size should be in pixels, so round it to avoid
any floating point inaccuracies from projection.
Closes #4255.

Looks good!


FIX. The method 'name' does not exist on 'Form'

Commits squashed and merged in https://github.com/silverstripe/silverstripe-framework/commit/5637431a3fd81d9404a873a94ba7d21cb223093d. Thanks @uniun :)


Adding changes from https://github.com/Readify/DevEvents/pull/39

Added CFP start/close dates to table
Added honourable mentions, all-year CFP and TBD lists
Changed date format to DD-MMM-YYYY
Crossed out past events
Added Perth Web Accessibility Camp, Decompress, CampJS

Closes #39 cc @amykapernick @fendersperth
Need to figure out whether to keep the DD/MM/YYYY format (i.e. change back) or change to DD-MMM-YYYY and change CONTRIBUTING.md and the 2017 table...

Thoughts on how to approach the date format @ghuntley @alexmackey?

Looks good to me @robdmoore. I wondered about CFP dates but not sure this will be kept up to date. Meh lets try it and see what happens :) 👍

Main issue is there is an inconsistent date format now as per my comment - thoughts on which option we use to address that?
--
On 4 Feb 2018, at 10:06 am, Alex Mackey <notifications<EMAIL_ADDRESS>wrote:
Looks good to me @robdmoorehttps://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frobdmoore&data=02|01||ea1b90a1d50346ade4b508d56b73e4fc|86ad559a0fdb4325b0a693cb899ec1e5|0|0|636533067872420023&sdata=XfqO8SWhqyAfzpiUZVI3C7NAl3%2F8ewBw9jFnMTQ9JZk%3D&reserved=0. I wondered about CFP dates but not sure this will be kept up to date. Meh lets try it and see what happens :) 👍
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FReadify%2FDevEvents%2Fpull%2F40%23issuecomment-362873972&data=02|01||ea1b90a1d50346ade4b508d56b73e4fc|86ad559a0fdb4325b0a693cb899ec1e5|0|0|636533067872576279&sdata=4SdCMjBovKzcmzKq7caCygnl8wlFzi3trmvphaLz3Aw%3D&reserved=0, or mute the threadhttps://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAjBXwleVsc02o8br0-dWTqxEXNiWny-ks5tRRCdgaJpZM4R4Z5u&data=02|01||ea1b90a1d50346ade4b508d56b73e4fc|86ad559a0fdb4325b0a693cb899ec1e5|0|0|636533067872576279&sdata=penJ2rIFdpGwLebDbN1Zx08yC4A8y6Vb9veHl8lh%2F10%3D&reserved=0.

I know there are other dates to complete but I think I like DD-MMM-YYYY better. Its also clearer if this is used internationally?
I wanted to keep this simple to encourage contributions but also wonder about having this as a json file and some kind of build process to produce the md.

ISO 8601. It's clear. It's the international standard for data interchange formats.
yyyy-MM-dd

DD-MMM-YYYY is harder to read and seems to format across three lines in my browser. Agree with Tatham that ISO 8601 (yyyy-MM-dd) is a better option.

The main point of the site however is something for people to read rather than machines (at this point). I know yyyy-MM-dd is not too tricky to read but its not how you would expect to read a date either?

I consider it to be a readable format, but that's purely subjective. It supports future parsing and sorting, with a standards alignment, and remains unambiguous when it comes to humans (or machines) reading them.

Giving that this is a page to users and specific to AU, I'd prefer to keep dates as people read in Australia. DD/MM/YYYY
As a user, if I come to any other date format other than that I need to pause and think, which is not a good UX in my POV.
I understand that other formats make our life easier, but we are here to make our life easy and the user experience harder? I'm all favour to the user.

What about internationals that want to visit an au conference?
--
On 5 Feb 2018, at 10:44 am, Luiz Adilson da Silva Bon <notifications<EMAIL_ADDRESS>wrote:
Giving that this is a page to users and specific to AU, I'd prefer to keep dates as people read in Australia. DD/MM/YYYY
As a user, if I come to any other date format other than that I need to pause and think, which is not a good UX in my POV.
I understand that other formats make our life easier, but we are here to make our life easy and the user experience harder? I'm all favour to the user.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FReadify%2FDevEvents%2Fpull%2F40%23issuecomment-362967136&data=02|01||9f2b3924d964415acebd08d56c4255f7|86ad559a0fdb4325b0a693cb899ec1e5|0|0|636533954523968138&sdata=oifx%2Bw4B9QdIGDyB4BSUN3jgKcZPH58cZ%2BCwXyUNzuE%3D&reserved=0, or mute the threadhttps://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAjBX48NQjHguBtYLzeFtmUqei_usxPEks5tRmr2gaJpZM4R4Z5u&data=02|01||9f2b3924d964415acebd08d56c4255f7|86ad559a0fdb4325b0a693cb899ec1e5|0|0|636533954523968138&sdata=Fxq%2BKLP%2BgXnOC8cjumX8XbZKsg607hFt1NsiC1VcmvA%3D&reserved=0.

In that case, the audience is international, not only AU?
Checking international conference websites quickly, all of them use dd MMMM. So looks like a pattern to keep the month as name and not numbers.
My opinion is based on having a better user experience instead of easy development, so defining the target audience will help a lot to define what is the best format.


Remove in collision goals docs

Docs for https://github.com/ros-navigation/navigation2/pull/4595

@tonynajjar this comment --> https://github.com/ros-navigation/docs.nav2.org/pull/579/files#r1709734286

Sorry, Idk how I missed it, on it


[ENH] update bids apps listing

add bibsnet and petdeface

https://bids-website--478.org.readthedocs.build/en/478/tools/bids-apps.html

Does this use the same jinja template as the bids-apps website?

nope: given the old bids app website will be fused into the new bids website, I nuked whatever I was using before

Does this use the same jinja template as the bids-apps website?

nope: given the old bids app website will be fused into the new bids website, I nuked whatever I was using before

🙈


;remwaifus all causes a crash.

Fixed. I think this was actually part of a larger problem, but I went ahead and made this a functional command.


Move modules to storybook-core repo

Move core mantra modules into an it's own repository. Then we can reuse the same UI in different storybooks projects.

@mnmtanish always, add a description to a PR. There's always something to say.

Added the description after creating the PR. I guess it didn't come with the email.
Btw, the repo name should change to storybook-ui. I'll send another PR for that.


Fix Issue 9362 - Add a method to remove one item to std.container.SList

https://issues.dlang.org/show_bug.cgi?id=9362
If everyone agrees with this I will go ahead and do the same for Dlist.

LGTM

@ZombineDev @wilzbach this PR should be ready for merge. Jenkins seems to be unhappy with something unrelated with this PR.

Being a public symbol this still needs an approval from @andralex

@andralex @ZombineDev should be ready for merge

@wilzbach this fails the style checker, which segfaults. I'll pull this, could you please investigate the matter?


Are there any comparison chart on Azure DevOps Server Vs Azure DevOps Service Vs GHE Cloud/Server?

Current recommendation for customer who are working on DevOps Strategy is to use Azure DevOps Service + GitHub Enteprise Cloud (GHE) or Server.  However we do not seem to have a document that compare all these offering for organization to plan better for future. Can we add this document somewhere? Or if it exist, then I would like to know the link.
Another question routinely asked is what is the road-map for Azure DevOps Server or Service as everything moving to GHE in future. How to find the answer to this question?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 808dac3a-225a-f564-ea5e-de6bad168da4
Version Independent ID: decabfe6-ae8d-183c-3698-acb829da81f5
Content: Understand differences between Azure DevOps Services and Azure DevOps Server - Azure DevOps
Content Source: docs/user-guide/about-azure-devops-services-tfs.md
Product: devops
Technology: devops-new-user
GitHub Login: @chcomley
Microsoft Alias: chcomley

Hi @Anand-Moghe. For information about the Azure DevOps Feature Timeline, you can go here. As for further information about comparisons with GH, I unfortunately don't have any. If you have any further questions, I'd recommend contacting your Microsoft representative and accessing the Developer Community here. Thanks for  reaching out.


Update AAR Creation for Tiki SDK v2 and Implement Pre-Built UIs in Example App

Currently, the example app is using compiled code whose dependencies need to be updated. To resolve these and include the latest updates, it is necessary to update the AAR creation for Tiki SDK v2 with Fluer 3.7 and est the pre-built UIs in the example app
This update is essential to test the latest version of Tiki SDK with the new pre-built UIs functionality. Once the AAR creation is updated, the example app will be able to fully test the pre-built UIs settings/offer flow.

In this process check #95
In the fat-aar build, we are including Moshi dependency since it does not work in integration tests when building aar locally.
In the maven central repository, the pom file includes Moshi as a dependency.
It leads to a dependency collision when importing the SDK from Maven Central: both the tiki-sdk-android binary and the project that is importing it will have Moshi classes.
Acceptance criteria
1 - TIKI SDK Android can be used from a Maven repository without collision
2 - TIKI SDK Android can be built locally and include Moshi dependency

The AAR creation is done and all dependencies resolved. Integration tests runs correctly and just fails because of missing terms.md file, that will be included in the next PR.
The usage of Jetpack Compose by the TIKI SDK UI makes it necessary to the project that uses TIKI SDK to import it. In #133 this is going to be fixed, by including the compose foundation library as a dependency in the pom file.
#95 will be handled by #133 too.


Return 429s instead of 503s in adaptive concurrency filter

Hi team,
When the adaptive concurrency filter of a server is triggered, clients may feel confused that the server is unavailable instead of that the server is proactively rejecting requests. I'm wondering if it makes more sense to change the status to 429s instead of 503s? It seems netflix's concurrency limits library uses 429s.

It would make sense to me if this was configurable like the global rate limit. We could default to 503 but allow users to override it.


diff static and dynamic parts of a template

fix #42

I don#t have a strategy to go forward here. Will start afresh when I do know.


⚠ Bofrost.lu has degraded performance

In 73587ff, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2219 ms

Resolved: Bofrost.lu performance has improved in c6a1ab8 after 13 minutes.


Ketrem preprod docs

Docs Build status updates of commit 327cf72:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

windows-driver-docs-pr/dashboard/index.yml
:x:Error

Details

windows-driver-docs-pr/dashboard/index.yml

Line 84, Column 13: [Error: yaml-syntax-error] While parsing a block mapping, did not find expected key.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 57355d3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows-driver-docs-pr/dashboard/index.yml
:white_check_mark:Succeeded

windows-driver-docs-pr/dashboard/manage-preprod-submissions.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 57355d3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows-driver-docs-pr/dashboard/index.yml
:white_check_mark:Succeeded

windows-driver-docs-pr/dashboard/manage-preprod-submissions.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Add a nixos vm test for webghc

This adds a nixos vm test for web-ghc in nix/tests/vm-tests/web-ghc

group tests into lintingTests and vmTests
pass plutus to tests/default.nix to get access to web-ghc
Tests that the configured port is open, tests version, health and
compile endpoint of web-ghc

Note 1: Maybe web-ghc isn't the most essential piece of software but i wanted to pick something with a small scope to start with and showcase how tests are written. For now you can have a look at https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests for more info
Note 2: I should probably add some notes on running and debugging this test, maybe a README in the vm-tests folder or so
Note 3: @shmish111 i added you since you wrote web-ghc, let me know if you can think of something that i should add as a test

Pre-submit checklist:

Branch

[X] Commit sequence broadly makes sense
[X] Key commits have useful messages
[X] Relevant tickets are mentioned in commit messages

PR

[X] Self-reviewed the diff
[X] Useful pull request description
[X] Reviewer requested

Pre-merge checklist:

[ ] Someone approved it
[ ] Commits have useful messages
[ ] Review clarifications made it into the code
[ ] History is moderately tidy; or going to squash-merge

How does this run on Hydra? In the past I've found nix vms to be incredibly slow to start and stop on AWS and I think we run Hydra instances on AWS.

How does this run on Hydra? In the past I've found nix vms to be incredibly slow to start and stop on AWS and I think we run Hydra instances on AWS.

vm tests are executed using qemu. Checked the log and it took 25s to execute and is thus - fair to say i guess - irrelevant considering overall CI build times.

How does this run on Hydra? In the past I've found nix vms to be incredibly slow to start and stop on AWS and I think we run Hydra instances on AWS.

vm tests are executed . Checked the log and it took 25s to execute and is thus - fair to say i guess - irrelevant considering overall CI build times.

oh, 25s is nice


[Bug]: Empty metacache file created by manifest crashes Prism Launcher at launch

Prerequisites

[X] I have written a descriptive issue title.
[X] I have searched all issues/PRs to ensure it has not already been reported or fixed.
[X] I have verified that I am using the latest version of Scoop and corresponding bucket.

Package Name
prismlauncher / prismlauncher-qt5
Expected/Current Behaviour
I'm experiencing a crash every time I try to open the launcher. My friend had the same problem a few days ago.
Here are the last few lines of the log:
     0.037 D | Instance path              :  "instances"
     0.038 D | Loading Instances...
     0.038 D | Discovering instances in "C:/Users/Hyeon/scoop/apps/prismlauncher/current/instances"
     0.038 D | <> Instances loaded.
     0.038 D | Loading accounts...
     0.038 C | "Failed to parse account list file: illegal value at offset 0"
     0.038 D | <> Accounts loaded.
     0.038 C | Exception: "HttpMetaCache root is not an object"

According to https://github.com/PrismLauncher/PrismLauncher/issues/1094, Prism Launcher crashes raising "HttpMetaCache root is not an object" exception if there is a corrupt metacache file in its directory at launch.
Currently, the manifest creates an empty metacache during pre_install and persists it. This seemed to be a problem.
First I deleted the file once and the launcher worked fine.
And instead of deleting it, I modified the content of the metacache to {}, and that worked too.
Steps to Reproduce
> scoop install games/prismlauncher
Installing 'prismlauncher' (7.1) [64bit] from games bucket
Loading PrismLauncher-Windows-MSVC-Portable-7.1.zip from cache.
Checking hash of PrismLauncher-Windows-MSVC-Portable-7.1.zip ... ok.
Extracting PrismLauncher-Windows-MSVC-Portable-7.1.zip ... done.
Running pre_install script...
Linking ~\scoop\apps\prismlauncher\current => ~\scoop\apps\prismlauncher\7.1
Creating shim for 'prismlauncher'.
Creating shortcut for Prism Launcher (prismlauncher.exe)
...
Persisting metacache
...
Running post_install script...
'prismlauncher' (7.1) was installed successfully!

Possible Solution
Make pre_install write some valid content to metacache file.
The minimal valid content I found is:
{}

Scoop and Buckets Version
> scoop --version
Current Scoop version:
v0.3.1 - Released at 2022-11-15

'games' bucket:
24e648a0 (HEAD -> master, origin/master, origin/HEAD) pcsx2-dev: Update to version 1.7.4734

Scoop Config
> scoop config

last_update           : 2023-07-16 오전 12:11:55
alias                 : @{gui=scoop-gui; force-update=scoop-force-update}
aria2-warning-enabled : False
scoop_repo            : https://github.com/ScoopInstaller/Scoop
scoop_branch          : master

PowerShell Version
$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.3.6
PSEdition                      Core
GitCommitId                    7.3.6
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Additional Softwares
No response

still an issue, awaiting pull request review/merge
pasting my logs (note that earlier on, there's also some notions that there's a missing translation file; I'm not sure if this is being addressed):

here's a full log that I am getting
PS C:\Users\micha> prismlauncher.exe
prismlauncher.exe : The term 'prismlauncher.exe' is not recognized as the name of a cmdlet, function, script file, or o
perable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try aga
in.
At line:1 char:1
+ prismlauncher.exe
+ ~~~~~~~~~~~~~~~~~
  + CategoryInfo          : ObjectNotFound: (prismlauncher.exe:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

PS C:\Users\micha> scoop uninstall prismlauncher
ERROR 'prismlauncher' isn't installed.
PS C:\Users\micha> scoop install prismlauncher
Installing 'prismlauncher' (7.2) [64bit] from games bucket
Loading PrismLauncher-Windows-MSVC-Portable-7.2.zip from cache
Checking hash of PrismLauncher-Windows-MSVC-Portable-7.2.zip ... ok.
Extracting PrismLauncher-Windows-MSVC-Portable-7.2.zip ... done.
Running pre_install script...
Linking ~\scoop\apps\prismlauncher\current => ~\scoop\apps\prismlauncher\7.2
Creating shim for 'prismlauncher'.
Creating shortcut for Prism Launcher (prismlauncher.exe)
Persisting assets
Persisting cache
Persisting icons
Persisting instances
Persisting libraries
Persisting logs
Persisting meta
Persisting mods
Persisting themes
Persisting translations
Persisting accounts.json
Persisting metacache
Persisting prismlauncher.cfg
Running post_install script...
'prismlauncher' (7.2) was installed successfully!
Notes
-----
To add Prism Launcher file association options for .ZIPs and .MRPACKS, run this:
"C:\Users\micha\scoop\apps\prismlauncher\current\install-associations.reg"

This package is now using the portable version of Prism Launcher, and data should have been migrated automatically.
If you are using a global install on a system with more than one user, you will need to copy a user's data from
%appdata% to the new Scoop PrismLauncher persist directory
PS C:\Users\micha> prismlauncher.exe

   0.000 D | Testing "C:/Users/micha/scoop/apps/prismlauncher/current/qtlogging.ini" ...
   0.000 D | Loading logging rules from: "C:/Users/micha/scoop/apps/prismlauncher/current/qtlogging.ini"
   0.000 D | Setting log rules:
   0.000 D |      "*.debug=true"
   0.000 D |      "launcher.auth.credentials.debug=false"
   0.000 D |      "launcher.task=false"
   0.000 D |      "launcher.task.net=true"
   0.000 D |      "launcher.task.net.download.debug=false"
   0.000 D |      "launcher.task.net.metacache=false"
   0.000 D |      "launcher.task.net.metacache.http=true"
   0.003 D |      "launcher.task.net.upload=true"
   0.003 D |      "qt.*.debug=false"
   0.003 D | <> Log initialized.
   0.003 D | <> No migration needed from "PolyMC"
   0.003 D | <> No migration needed from "MultiMC"
   0.003 D | "Prism Launcher" , (c) 2013-2021  "Prism Launcher Contributors\n∩┐╜ 2021-2022 PolyMC Contributors \n∩┐╜ 2012-2021 MultiMC Contributors"
   0.003 D | Version                    :  "7.2"
   0.003 D | Platform                   :  "official"
   0.004 D | Git commit                 :  ""
   0.004 D | Git refspec                :  "refs/heads/stable"
   0.004 D | Work dir before adjustment :  "C:/Users/micha"
   0.004 D | Work dir after adjustment  :  "C:/Users/micha/scoop/apps/prismlauncher/current"
   0.004 D | Adjusted by                :  "Portable data path"
   0.004 D | Binary path                :  "C:/Users/micha/scoop/apps/prismlauncher/current"
   0.004 D | Application root path      :  "C:/Users/micha/scoop/apps/prismlauncher/current"
   0.004 D | <> Paths set.
   0.007 D | Detected default console font: "Courier New" , substitutions: ""
   0.014 D | <> Settings loaded.
   0.014 D | Detecting proxy settings...
   0.014 D | Using no proxy is an option!
   0.014 D | <> Network done.
   0.015 C | Exception: "Unable to open C:/Users/micha/scoop/apps/prismlauncher/current/translations/index_v2.json for reading: The system cannot find the file specified."
   0.015 C | Translations Download Failed: index file not readable
   0.016 W | Selected invalid language "" , defaulting to "en_US"
   0.016 D | Your language is ""
   0.016 D | <> Translations loaded.
   0.017 D | Started watching  "C:/Users/micha/scoop/apps/prismlauncher/current/icons"
   0.017 D | Sorting icon list...
   0.017 D | <> Instance icons intialized.
   0.018 D | [Theme] <> Icon themes initialized.
   0.018 D | [Theme] <> Initializing Widget Themes
   0.018 D | [Theme] Determining System Theme...
   0.018 D | [Theme] System theme seems to be: "windowsvista"
   0.018 D | [Theme] Considering theme from theme factory: "windowsvista"
   0.018 D | [Theme] System theme has been determined to be: "windowsvista"
   0.018 D | [Theme] Loading Built-in Theme: "system"
   0.018 D | [Theme] Loading Built-in Theme: "dark"
   0.018 D | [Theme] Loading Built-in Theme: "bright"
   0.019 D | [Theme] Theme Folder Path:  "C:/Users/micha/scoop/apps/prismlauncher/current/themes"
   0.019 D | [Theme] <> Widget themes initialized.
   0.019 D | Instance path              :  "instances"
   0.019 D | Loading Instances...
   0.020 D | Discovering instances in "C:/Users/micha/scoop/apps/prismlauncher/current/instances"
   0.020 D | <> Instances loaded.
   0.020 D | Loading accounts...
   0.020 C | "Failed to parse account list file: illegal value at offset 0"
   0.020 D | <> Accounts loaded.
   0.020 C | Exception: "HttpMetaCache root is not an object"


Story Viewers Pagination

I've pushed a PR addressing the issue. @TolgaKaramese

@sbertix I opened this issue due to Mahdi's request :)

Hahahaha okay 😊 still, the PR is there 🙈

Okay, thanks ☺


RP2350 Support

Raspberry Pi has released the RP2350 (Pico 2), which should have the same SDK compatibility making code written for RP2040 portable to RP2350. However, there are several initial challenges blocking direct implementation:

There are new PICO_BOARD and PICO_PLATFORM variables for pico2/rp2350 that need to be passed
Currently the hardware/rp2040 directory name is tied directly to PICO_BOARD definition
FreeRTOS does not natively support the RP2350 yet (see here)

See this commit on Raspberry Pi's FreeRTOS fork for the "unofficial" port for RP2350

To use Raspberry Pi's unofficial port - merge the RP2350-specific changes into an existing copy of FreeRTOS kernel source, open a terminal in the kernel source directory and run:
git remote add raspberrypi https://github.com/raspberrypi/FreeRTOS-Kernel.git
git checkout -b rp2350
git pull raspberrypi 4f7299d6ea746b27a9dd19e87af568e34bd65b15

This will merge Raspberry Pi's changes. To use the RP2350 port, portable/ThirdParty/GCC/RP2350_ARM_NTZ/FreeRTOS_Kernel_import.cmake is sourced, and the PICO_PLATFORM variable must be set to "rp2350" to get it to work.

Additional config options are necessary in FreeRTOSConfig.h to support ARMv8m (RP2350 is now Cortex-m33 vs ARMv7m/Cortex-m0+ on RP2040) - see this post

FreeRTOSConfig.h with RP2350-specific configurations from Raspberry Pi in pico-examples - note configMAX_SYSCALL_INTERRUPT_PRIORITY option is also necessary

Merged into v0.3-pre (#26) to be part of 0.3 release


Editable dependency in dev environment only, and source the git version in packaging environment

Hello! Thanks for the awesome work on uv. The amount of things you're getting right and the speed of your development team is inspiring!
Say I have a package foo and another package bar that I'm both developing.
bar depends on foo.
In my dev environment, I'll have a local version of both foo and bar that I'm working on.
./devel/foo
./devel/bar

Each package has its own pyproject.toml. foo itself has some optional dependencies and I want bar to depend on foo without the optional dependencies, because foo is exposing some specific set of features to bar and has its own other functionalities that depend on heavy Python libraries that I definitely don't want to have as dependencies of bar.
While I'm working on bar, I want to have the foo dependency in editable mode.
But when I'm packaging bar, I want it to fetch the foo dependency from a git repositry.
In other words, I would like the dependency in the dev dependency group to be
[tool.uv.sources]
foo = { path = "../foo" }

And in the packaging environment (in the normal dependencies part of the pyproject.toml) to be
[tool.uv.sources]
foo = { git = "ssh://git@mygit.com:myorg/mypackage" }

Right now, if I add foo as a dependency in either, it's gonna overwrite the other version.
What's the right way to manage this situation with uv?

Is this about the same as https://github.com/astral-sh/uv/issues/7945 ?

@zanieb very much looks like so, thanks for pointing it out: it did not came up in my searches on the issue tracker. closing this in favor of the existing issue


Add openmetrics exemplar support

This is a work in progress implementation for #175
Opening this as a draft to have a discussion to see if this is a valid approach to go forward with, or if there is a different path that is cleaner/better.
I've tried to keep all changes to the prometheus exporter, as this is a prometheus-specific feature, but have had to add downcasting to the recorder, so that a handle to the specific recorder can be fetched. But maybe there is some more generic api that can be used. I'm not very familiar with other metrics systems, but maybe it is common to add additional data to observations?
There is an example of how to increase a counter and record a sample in a histogram with exemplars. This is extra problematic for histograms, as we need to know what bucket to assign the exemplar with.
I'm also thinking about what would be a good macro api for this. Since an exemplar is just an other set of labels, we can't just add it as an other parameter.
e.g. with the following its hard to know what labels should be metric labels and which should be exemplar labels
counter_with_exemplar!("my_counter", "some_label" => "value", "trace_id" => "123");

We could probably say that exemplar labels are always expressions, so something like would probably be possible (I haven't written any macros, but I'm guessing this is possilbe)
counter_with_exemplar!("my_counter", vec![Label::from_parts("trace_id", "123")], "some_label" => "value");
counter_with_exemplar!("my_counter_without_labels", vec![Label::from_parts("trace_id", "123")]);

It might be confusing still, as it is easy to mix the different labels up.
Also, where would these macros live? I don't think we can add them to metrics-macros (behind a feature), since that would be a cyclic dependency if it needed to depend on the PrometheusRecorder.

Is there anyway you can give me a high-level overview of exemplars and how a typical application, with one of the official Prometheus client SDKs, is using them?
I'm finding it hard to conceptualize, and based on the changes so far, I think we need to step back and talk about the feature first before trying to design the interface to it.

Is there anyway you can give me a high-level overview of exemplars and how a typical application, with one of the official Prometheus client SDKs, is using them?

Absolutely, I'll write some stuff that I know about it here, let me know if there is any other details that you would like to know.
Use case
Exemplars is a prometheus invention that can be use to attach additional data to some samples for a metric. An exemplar consist of the recorded value, a timestamp when it happen, and additional labels, the timestamp is optional (havent tried what happens if it is not set, guessing that the position when rendering it might get the collection time and not exactly when it happen). This is then used in e.g grafana to show little dots in graphs when creating graphs over these metrics. When hovering these dots the additional labels are shown. The most common use for this is to add a trace id to the exemplar labels, so that grafana can add a link into the tracing system.
This is one of the main trace discoverability features when using tempo as a tracing backend.
An example screenshot of how it is displayed in grafana:

Exporting
For prometheus to be able to ingest exemplars, they need to be exported with the metric. Each label set for a metric can expose one exemplar. So in prometheus there will at most be one exemplar per metric + label set per collection. If multiple exemplars are written to the same metric + label set within the same collection, it will just overwrite the exemplar. (I would say that exemplars make most sense when using prometheus histograms where there can be one exemplar per bucket, since the bucket is also a label)
The format it is exposed in is defined in open metrics.
Other implementations
I've mostly looked at how this is implemented in the prometheus golang client. E.g how it is used on histograms. It is backed by a atomic value store that just gets overwritten. The value that is stored is found here
Other thoughts
I noticed now that exemplars should not be implemented for summaries and gauges, only for histograms and counters, and for counters the openmetrics documentation is a big vague. IMO it would be enough to implement only for histograms, but that is still the hardest one so we might as well implement it for both.
I'm not sure what happens if a metric doesn't get their exemplar updated and gets collected several times. I think prometheus handles that, but I need to do some more research.

Just a note: this is still on my backlog to review, things have just been a bit hectic for me lately. 😅

No worries and no stress, but thanks for letting me know!
I'm also on discord most of the time if you want to discuss something when you look at this

So, my main question after reading your explanation (thank you for that!): do the exemplars have to be logically related?
Which is to say... if I have two concurrent tasks/threads/whatever emitting metrics and both of them hit the random number generator lottery of "you should track these metrics as exemplars", do all the metrics they touch need to have the same examplar label (trace ID, etc) or could some of the metrics have trace ID 1, and some have trace ID 2, etc?

They can be different, there is no connection between exemplars over different metrics.

They can be different, there is no connection between exemplars over different metrics.

Alright, that's good news. 👍🏻
Depending on the behavior necessary, it seems like it could be possible to get away with sampling a value at the point of actually rendering the metrics. That is, every time the metrics are rendered -- which is just when we get a scrape request, or our interval to hit the push gateway ticks -- we collect the outstanding histogram samples and pick one of them to be our new exemplar.
Avoiding new exporter-specific methods seems like the highest priority item in my mind. We should ideally be able to just collect exemplars with people using the metrics macros and the Prometheus exporter the same way they always have.

Sorry for being so slow to reply.
Yes, getting the exemplar on render would definitely be enough, and it sound like you are on to something smart, but I don't understand how exactly :sweat_smile:. How would one add an exemplar, if there where no exporter specific method for it?

Right, so my thought is that the render logic would essentially be responsible for figuring out if it was time to sample a new exemplar for each unique histogram.
So we'd have the exemplar value itself, and probably a timestamp for "when was this exemplar observed?". In the render logic, where it checks to see if it needs to consume any more raw samples from the underlying histogram storage, we'd see how long ago we last captured an exemplar for the given histogram. If we've exceeded our timeout, then we take one of the samples we just consumed and make it our exemplar, and update our timestamp.
Explained more contextually:
We start with an exporter that has a default initial state: no metrics observed yet, etc. We'll refer to the time that these actions/operations occur with the t=.. notation, denoting the time in seconds.

At t=0, render is called, and we consume all histograms from the registry, and discover a new one, histogram_a.
histogram_a has 10 samples (10 is just a random number, doesn't matter if it's 1 or 10000) and since we have not seen this histogram before, our "it's time to capture an exemplar" logic kicks in, and we randomly select one of those 10 samples as the exemplar value.
At t=2, render is called again, and we get 5 more samples for histogram_a when we consume it from the registry. For the sake of explanation, let's pretend our exemplar selection logic doesn't select a new exemplar unless it's been over 5 seconds since the current one was selected. Since only two seconds have elapsed, we don't choose a new exemplar.
At t=6, render is called again, and we get 7 more samples for histogram_a when we consume it from the registry. Since we last selected an exemplar for this histogram at t=0`, our exemplar selection logic now kicks in, and we randomly select one of the 7 samples we just consumed and use that as the new exemplar for the histogram.

So, overall, I'm forcing some design decisions here for the sake of explaining my idea:

exemplars are randomly selected from the samples we consume for a histogram
exemplars are only refreshed if enough time has elapsed

But how do we register the exemplar labels with the value? they need to be added when recording the value, as they contain data that is not know by the metric itself, most commonly a trace id that refers to some external tracing system.
That trace id will be different for every value recorded (as long as there are not multiple metrics recorded during the same request, which I thought your initial question about)

Are the labels for exemplars only meant for exemplars period? Like would you not typically include, say, a trade ID label unless you wanted it to be an exemplar?

Yes exactly, its a separate label set that the regular label set for the metric itself. The common use case is to attach a trace id to a specific observation, and what I mean is a distributed trace id that comes from outside, like from jaeger, tempo, zipkin etc.
Usually when you build some kind of web server with incoming http requests, there will be some ingress gateway that initiates a trace, creates a trace id and attaches it to the http request as headers, and then your web server reads these and attaches a bunch of "spans" to it, and upload those to whatever tracing backend you have, a trace id can span over multiple web servers, if they do requests to each other within the same initial request.
The exemplar is then used to find a trace in the tracing backend from a graph built by metrics. Usually on latency histograms, so that when you see your latency graph and want to figure out why some requests take a long time, you click on the little exemplar dot representing a specific observed request that took a long time to process.
Using the golang client, the definition would look something like this, specifying the metrics labels and buckets
var (
	histogram = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "foo_latency",
			Buckets: []float64{0.01, 0.1, 1, 10},
		},
		[]string{"path"}
	)
)

Then when observing a value, it would be something like this, very simplified, but getting the trace id from the request, and adding it to the exemplar label-set.
histogram.With(prometheus.Labels{"path": "/foo"}).(prometheus.ExemplarObserver).ObserveWithExemplar(
	latency, prometheus.Labels{"traceID": req.Header.Get("x-trace-id")},
)

This is what the prometheus exporter will render, just to clarify what it looks like:
# TYPE foo_latency histogram
foo_latency_bucket{path="/bar", le="0.01"} 0
foo_latency_bucket{path="/bar", le="0.1"} 8
foo_latency_bucket{path="/bar", le="1"} 11 # {trace_id="xyz123"} 0.67<PHONE_NUMBER>.512
foo_latency_bucket{path="/bar", le="10"} 17 # {trace_id="abc123"} 9.8<PHONE_NUMBER>.789
foo_latency_bucket{path="/bar", le="+Inf"} 17

In this example there are two different requests that have been recorded with an exemplar, one in bucket le=1 and one in bucket le=10, with their exact value, when they where observed, and this extra set of labels that include their trace id. (there can maximum be one observed exemplar per metric + metric labelset + bucket, if two are observed for the same, the later one just replaces the first one)
Sorry if I'm poorly trying to explain things that you already know, but just trying to explain the flow of how these things are used in the kind of area that I work with.

Hey @tobz, I lost the momentum a bit here, but would appreciate some feedback. I'm happy to continue on this if I know what way to go.

Hey @fredr! I'm no stranger to losing momentum. :)
I've started getting more serious about planning out the remaining work to bring metrics to a 1.0 release, and the general theme of that work is simplifying the public API surface as much possible: avoiding niche methods/functions, allowing for more flexible inputs, to let metrics adapt in the future.
With that said, I think my biggest concern, sort of right from the beginning: it doesn't feel right for there to be macros/methods/etc that are specific to Prometheus. I'm certainly not against third-party exporters having their own specific macros or something of the sort... but it's not the design pattern I want to promote in metrics or any of the official metrics ecosystem crates.
What I would want to see is figuring out a way to make the determination of when exemplars should be tracked something that functions more like a scoped behavior i.e. a function that takes a closure, and changes a thread-local to influence the behavior of any code running the closure. Alternatively, and maybe I'm still not entirely understanding how exemplars are typically tracked/triggered, but some sort of approach that was deterministic i.e. for any metric that has a specific label key, which is configured as part of the exporter itself, sample updates to metrics with that label at a configurable rate to derive exemplars.
With an approach as described above, it maintains one of the original design goals of metrics: if you're using a library, in your application, that is instrumented with metrics, you get that metric data "for free". It could be useful to include the metrics emitted by dependencies as part of an exemplar, but you can't do that if they need to use a specific macro.
I'm happy to continue providing feedback on the design as long as it's along the lines as described above, because that represents both the most ergonomic path, and easiest to maintain path, in my eyes. If executing on what I've posted above feels like it would consume too much time, I totally understand.

I've started getting more serious about planning out the remaining work to bring metrics to a 1.0 release, and the general theme of that work is simplifying the public API surface as much possible: avoiding niche methods/functions, allowing for more flexible inputs, to let metrics adapt in the future.

That sounds great, do you know already if that will have any implications on how buckets are registered for prometheus? or any other upcoming prometheus related changes?

With an approach as described above, it maintains one of the original design goals of metrics: if you're using a library, in your application, that is instrumented with metrics, you get that metric data "for free". It could be useful to include the metrics emitted by dependencies as part of an exemplar, but you can't do that if they need to use a specific macro.

Alright, my thinking behide the current poc implementation and suggestions was to keep all things that are specific to prometheus in the prometheus-exporter, but you have a good point that you then wouldn't get those specific features out of the box when using libraries that exposes metrics via this lib.

What I would want to see is figuring out a way to make the determination of when exemplars should be tracked something that functions more like a scoped behavior i.e. a function that takes a closure, and changes a thread-local to influence the behavior of any code running the closure

I'm not sure how to implement that, so I'll have to do some digging, if there is any such implementations in this or other crates that you know about, I would appreciate any pointers.

I'm happy to continue providing feedback on the design as long as it's along the lines as described above, because that represents both the most ergonomic path, and easiest to maintain path, in my eyes. If executing on what I've posted above feels like it would consume too much time, I totally understand.

Thanks, I'll keep at it whenever I have time to spare, this would be a really useful feature for trace discoverability.

@tobz

it doesn't feel right for there to be macros/methods/etc that are specific to Prometheus

I think there's a misunderstanding here and I'd really like the chance to add context to the discussion. Please take all the following as constructive input from a would-be fan. To some users like me, this would be the difference between always using and eagerly recommending metrics, or going out of my way to warn people to avoid it as a known dead end. I know that might sound like an over-reaction, but I hope it'll be clear where I'm coming from by the end of this post.
First to fill in some gaps in the discussion:

OpenMetrics is bigger than Prometheus, it's preserving what worked with Prometheus but expanding it to more features and more ingestion implementations. It seems to have stalled as an IETF standard, but it's already implemented in a number of exposition and ingestion systems, so it's at the very least a de facto standard. It is safe to consider this the new lowest common denominator for metric systems, and that lowest common denominator includes exemplars.

The fact it's not a ratified standard is a small mark against it, but it's still much closer to a standard than the pre-OpenMetrics Prometheus exposition format that you already support and that metrics is functionally just a reasonable subset of. The Prometheus format's lack of specification or standardization didn't stop you or anyone else from supporting it, and OpenMetrics goes a lot further in specifying everything you need to know for exposition, ingestion, and applied usage, so if anything it's a firmer foundation on which to build a united front like this facade.

Exemplars are the compromise between timeseries databases with many samples of small label sets, and event databases with fewer samples but larger label sets. Exemplars let individual samples have more labels without blowing out the dimensionality of the timeseries as a whole. It is not specific to OpenMetrics, much less to Promethus. It's a reality that every timeseries database has to face, otherwise it'll be abandoned in favor of a more powerful event database like InfluxDB.

OpenTelemetry is even more ambitious than OpenMetrics, and also supports exemplars. It's also just a specification, and maybe one day it will be formalized as a standard, but the industry is rallying around it already.

So both relevant specifications in this space support exemplars. Other specifications and implementations can reasonably expect exemplars in the data. Whatever standards come out of all of this, going forward, I can't imagine they'll leave exemplars out ever again. It would be like leaving out histograms.

Of course, not every implementation has caught up, but that's absolutely on them. There are browsers out there that don't support JS, and we just don't use those browsers, their existence doesn't somehow mean JS itself never needs to be written or executed again. At one point in the past there was only one browser that supported JS, now there are specifications and multiple implementations, it wouldn't be accurate or useful to dismiss JS as "specific to" any of them.

The prometheus_client crate already supports exemplars today. (It was named for consistency with other libraries, but it is clearly billed as an OpenMetrics exposition library).

To a Rust developer, a metrics facade may be the point of interoperability everything hinges around. But zooming out, across languages and libraries and so on, OpenMetrics and OpenTelemetry are those points of interoperability, little else is relevant on the same scale. If data needs to be exposed, code will be written to expose it, and if metrics makes that impossible then that code just won't use metrics. Implementation details are negotiable, functional requirements often aren't.

Already, today, a project can use prometheus_client to ensure all useful information is available for monitoring. At worst they can translate from OpenMetrics to another backend, perhaps using a standalone service that isn't even specific to Rust. Writing that translation once wouldn't be much more work than just writing a recorder just for metrics specifically. Some deployments I know already use Telegraf for such a translation today.

Thus OpenMetrics has the most leverage in shaping ecosystem choices, libraries at best get to save people some trouble in exposing OpenMetrics, and prometheus_client offers more here than prometheus or metrics at this point in time.

Look, if someone's hobby project supported parsing base64 reaction meme gifs from metrics to render inline in alerts, I would understand metrics not supporting that because it's a niche, ad-hoc, unspecified, likely unheardof extension. But exemplars are not that, they are a part of multiple published specifications each with multiple mainstream implementations enjoying wide industry adoption.
You've mentioned in several places that you wanted metrics to follow log and tracing, which sounds good on paper, but I think the fact there are two should serve as a warning. If metrics does not grow to support the features people need in this space, then at best there'll be another facade that does, just like tracing does strictly more than log. However, that would be a strictly worse situation, because it will clutter the ecosystem indefinitely as all the existing macros and implementations would still have to be supported in addition to the new ones. The whole ecosystem is better off having just one metrics that does everything and we can all rally around.
It's hard enough explaining to folks why both log and tracing exist, and I know lots of people are intimidated by the scope of tracing even though it's no harder to use than log for most purposes. Rust already gets this reputation for all sorts of things, monitoring shouldn't be another one. I want it to be easy for everyone to export metrics, but I don't want anyone investing in a dead-end API that they'll have to migrate off as soon as their needs outgrow what metrics is willing to support.
Looking forward
Now let me offer constructive suggestions on how to go about this. I think this PR would have gone very differently if metrics-exporter-prometheus used the new prometheus_client instead of the known-dead-end prometheus crate. Then the underlying support for exemplars would have already been there, and it would only be a matter of providing an API around that.
I don't see a way around migrating from prometheus to prometheus_client. Proposals to support OpenMetrics in general and exemplars in particular for prometheus were shot down specifically because prometheus_client already solved these problems in code that ships today. Given prometheus is a dead end, metrics will have to have a prometheus_client recorder sooner or later. When that inevitably happens, it will be much easier to support exemplars, and much harder to rationalize refusing to support them. In other words, metrics has the choice of being left behind with prometheus or to evolve alongside prometheus_client.
Maybe metrics v1.0.0 can launch without exemplars then e.g. v1.1.0 can add them in a backwards compatible way. The Go library showed just how wrong this can go, but this has more chance of succeeding in Rust than Go because at least Rust traits can evolve using default method implementations (i.e. recorders can ignore exemplars in the default implementation and add support if and when possible).
Personally I still think v1.0.0 is the perfect opportunity for the ecosystem to rally around a facade that does support exemplars, and what better time to introduce the world to a recorder based on prometheus_client which will remain the right way forward to capitalize on OpenMetrics without leaving vital functionality out in the cold. I don't think anybody will care if it took a few more months to get to that point, I think way more people will care whether it successfully captured the future of the ecosystem.
Thank you for your patience with this wall of text. I spent hours today writing this because, if there's any chance this sends metrics in a better direction, it will have been well worth it.

@tobz

it doesn't feel right for there to be macros/methods/etc that are specific to Prometheus

I think there's a misunderstanding here and I'd really like the chance to add context to the discussion. Please take all the following as constructive input from a would-be fan. To some users like me, this would be the difference between always using and eagerly recommending metrics, or going out of my way to warn people to avoid it as a known dead end. I know that might sound like an over-reaction, but I hope it'll be clear where I'm coming from by the end of this post.
First to fill in some gaps in the discussion:

OpenMetrics is bigger than Prometheus, it's preserving what worked with Prometheus but expanding it to more features and more ingestion implementations. It seems to have stalled as an IETF standard, but it's already implemented in a number of exposition and ingestion systems, so it's at the very least a de facto standard. It is safe to consider this the new lowest common denominator for metric systems, and that lowest common denominator includes exemplars.

The fact it's not a ratified standard is a small mark against it, but it's still much closer to a standard than the pre-OpenMetrics Prometheus exposition format that you already support and that metrics is functionally just a reasonable subset of. The Prometheus format's lack of specification or standardization didn't stop you or anyone else from supporting it, and OpenMetrics goes a lot further in specifying everything you need to know for exposition, ingestion, and applied usage, so if anything it's a firmer foundation on which to build a united front like this facade.

Exemplars are the compromise between timeseries databases with many samples of small label sets, and event databases with fewer samples but larger label sets. Exemplars let individual samples have more labels without blowing out the dimensionality of the timeseries as a whole. It is not specific to OpenMetrics, much less to Promethus. It's a reality that every timeseries database has to face, otherwise it'll be abandoned in favor of a more powerful event database like InfluxDB.

OpenTelemetry is even more ambitious than OpenMetrics, and also supports exemplars. It's also just a specification, and maybe one day it will be formalized as a standard, but the industry is rallying around it already.

So both relevant specifications in this space support exemplars. Other specifications and implementations can reasonably expect exemplars in the data. Whatever standards come out of all of this, going forward, I can't imagine they'll leave exemplars out ever again. It would be like leaving out histograms.

Of course, not every implementation has caught up, but that's absolutely on them. There are browsers out there that don't support JS, and we just don't use those browsers, their existence doesn't somehow mean JS itself never needs to be written or executed again. At one point in the past there was only one browser that supported JS, now there are specifications and multiple implementations, it wouldn't be accurate or useful to dismiss JS as "specific to" any of them.

The prometheus_client crate already supports exemplars today. (It was named for consistency with other libraries, but it is clearly billed as an OpenMetrics exposition library).

To a Rust developer, a metrics facade may be the point of interoperability everything hinges around. But zooming out, across languages and libraries and so on, OpenMetrics and OpenTelemetry are those points of interoperability, little else is relevant on the same scale. If data needs to be exposed, code will be written to expose it, and if metrics makes that impossible then that code just won't use metrics. Implementation details are negotiable, functional requirements often aren't.

Already, today, a project can use prometheus_client to ensure all useful information is available for monitoring. At worst they can translate from OpenMetrics to another backend, perhaps using a standalone service that isn't even specific to Rust. Writing that translation once wouldn't be much more work than just writing a recorder just for metrics specifically. Some deployments I know already use Telegraf for such a translation today.

Thus OpenMetrics has the most leverage in shaping ecosystem choices, libraries at best get to save people some trouble in exposing OpenMetrics, and prometheus_client offers more here than prometheus or metrics at this point in time.

Look, if someone's hobby project supported parsing base64 reaction meme gifs from metrics to render inline in alerts, I would understand metrics not supporting that because it's a niche, ad-hoc, unspecified, likely unheardof extension. But exemplars are not that, they are a part of multiple published specifications each with multiple mainstream implementations enjoying wide industry adoption.
You've mentioned in several places that you wanted metrics to follow log and tracing, which sounds good on paper, but I think the fact there are two should serve as a warning. If metrics does not grow to support the features people need in this space, then at best there'll be another facade that does, just like tracing does strictly more than log. However, that would be a strictly worse situation, because it will clutter the ecosystem indefinitely as all the existing macros and implementations would still have to be supported in addition to the new ones. The whole ecosystem is better off having just one metrics that does everything and we can all rally around.
It's hard enough explaining to folks why both log and tracing exist, and I know lots of people are intimidated by the scope of tracing even though it's no harder to use than log for most purposes. Rust already gets this reputation for all sorts of things, monitoring shouldn't be another one. I want it to be easy for everyone to export metrics, but I don't want anyone investing in a dead-end API that they'll have to migrate off as soon as their needs outgrow what metrics is willing to support.
Looking forward
Now let me offer constructive suggestions on how to go about this. I think this PR would have gone very differently if metrics-exporter-prometheus used the new prometheus_client instead of the known-dead-end prometheus crate. Then the underlying support for exemplars would have already been there, and it would only be a matter of providing an API around that.
I don't see a way around migrating from prometheus to prometheus_client. Proposals to support OpenMetrics in general and exemplars in particular for prometheus were shot down specifically because prometheus_client already solved these problems in code that ships today. Given prometheus is a dead end, metrics will have to have a prometheus_client recorder sooner or later. When that inevitably happens, it will be much easier to support exemplars, and much harder to rationalize refusing to support them. In other words, metrics has the choice of being left behind with prometheus or to evolve alongside prometheus_client.
Maybe metrics v1.0.0 can launch without exemplars then e.g. v1.1.0 can add them in a backwards compatible way. The Go library showed just how wrong this can go, but this has more chance of succeeding in Rust than Go because at least Rust traits can evolve using default method implementations (i.e. recorders can ignore exemplars in the default implementation and add support if and when possible).
Personally I still think v1.0.0 is the perfect opportunity for the ecosystem to rally around a facade that does support exemplars, and what better time to introduce the world to a recorder based on prometheus_client which will remain the right way forward to capitalize on OpenMetrics without leaving vital functionality out in the cold. I don't think anybody will care if it took a few more months to get to that point, I think way more people will care whether it successfully captured the future of the ecosystem.
Thank you for your patience with this wall of text. I spent hours today writing this because, if there's any chance this sends metrics in a better direction, it will have been well worth it.


Update changed model spec for domain_dashboard and project_dashboard

Category

[ ] New feature
[ ] Bug fix
[x] Improvement
[ ] Refactor
[x] etc

Description

update changed model spec for domain_dashboard and project_dashboard
add setup.py

Known issue

#2

@whdalsrnt
Thanks, I followed your review
Please check again!


Widget-level a11y tests

Our test framework should support testing the proper generation of the semantics tree for a widget or set of widgets.
Internally, we have SemanticsTester for that. We should consider making that part of the API offered by Flutter's testing package to make it accessible for every developer.
For that, we should clean-up and improve its API.

/cc @yjbanov Would you be interested in taking a look at this?

I would, but i18n work will block my progress on this for some time.

cc @jonahwilliams
There's some tests that reference this issue (in particular, examples/catalog/test/custom_semantics_test.dart).

I've already sort of started of this issue with getSemanticsData - that also needs some improvements though

Mostly done, https://github.com/flutter/flutter/issues/19691 as a follow up


update the naming as the directory of the package does not align with…

… the package name
@YijunXieMS

Can one of the admins verify this patch?


Some dropdown menu on postDropdownBtn doesn't follow selected app language

Describe the bug
Send via DM, Embed post, and Mute Words and Tags on postDropdownBtn doesn't follow selected app language
To Reproduce
Steps to reproduce the behavior:

tap on postDropdownBtn

Expected behavior
should follow selected app lang
Screenshots

Details

Platform: all platforms
Platform version: -
App version: Build version: 1.87.0; Bundle info: fe01f32 (prod); Bundle date: 24062805; Platform: web

Hi @xtianpu, this is because the Spanish localization is very out-of-date. @brodieavoult is updating it in #4319, you might want to help him out there :)

hello @surfdude29 oh well. I just checked for other langs, looks like Send via DM still in english, but for chinese, korean and japanese all looks good. I just updated my screenshots.


First uninstall function attempt

I did try...

Aside from the comments, I like that you covered most of the points I was looking at.
A few things though we probably should also remove is the /etc/pufferd directory where configs and assorted data is as well.
You also followed the golang style (betting you just followed how install worked, but still), which has the OS specific code, good job :)

How do you suggest to remove the "/etc/pufferd"?
I wouldn't use a remove by the golang itself (since it could leave a trace)
I would instead use a "rm -rf"

Please also make sure this builds, not entirely sure if this can build (do not rely on the CI server for now)

Okay, I think all errors get fixed, if anything is okay I would proceed to test in a VM

TODO:
Read data from config file to delete data
Windows version
Test

At the moment I'm not at the computer, so I did not see how the program recognize the OS...
Can you tell mW where's the snippet which recognize the OS?

You do not directly code for it.
Golang will use the file name to determine if it should include that file for a build for a specific OS (the _linux and _windows), and only include those that matter for that OS.
This means that you need a uninstall_windows.go file which implements the publicly used function StartProcess and does whatever it needs.
https://github.com/PufferPanel/pufferd/blob/master/pufferd.go#L141 is an example, which we implement differently for Windows and Linux (https://github.com/PufferPanel/pufferd/blob/master/install/service_linux.go)
As long as the function exists for both OSes, Golang will build it and you don't have to code for it directly.

Ooh, Awsome!
It wasn't on the ebook I read :P

I started thinking about how to modify the function...
At the moment I would love to have a sample of the config file (since I'm not able to host a PufferPanel right now)
And I was thinking about passing the config as an argument (if possible)
And meanwhile I would try to code the windows version
(And is the Windows config file different?)

You do not need to worry about the config format or location, we handle that using the --config option already
You have access to the config.Get functions, which ask for a "key" and will give you the value from the config if it exists
https://github.com/PufferPanel/pufferd/tree/master/data contains the 2 configs that we by default use, including their keys and values.

Awsome, thanks!

I have a question...
Should I remove the folders in the config only (so leaving the /var/lib/pufferd folder in case of default path) or I should actually try to see if the path contains a "pufferd" path and delete it?
"serverfolder": "/var/lib/pufferd/servers",
"templatefolder": "/var/lib/pufferd/templates",
"datafolder": "/etc/pufferd"

Delete the folder in the config is the way I would do it.
I have plans on redoing that config, so that the /var/lib/pufferd folder isn't broken up that way.

I guess everything's alright now, can I test if it work?

You should always be trying to test as you go.
If you need help building it though, I can alter the build server to help prepare builds for you, but note it doesn't log if there's an error (which I believe your code might)

I should have fixed anything, today I'll try to compile the program and test it

Okay, obviously the "uninstall" var caused a conflict with the "uninstall" package, so i renamed it to "uninstaller"

I'm stuck since if a config file is not found, I would like to ask if delete the default config path (in the default config), so I would like to improve the "GetOrDefault" in the config package to make it return the default var in the default config, can I do it?

Okay, I tested it and it works both in Debian and Fedora Server


Not working with Samsung Galaxy S5

Hi,
i've tested the flashlight with my Samsung Galaxy S5 and Phonegap build.
What i did:
Adding the following line to my config.xml:
<gap:plugin name="cordova-plugin-flashlight" source="npm" />
In Javascript:
`window.plugins.flashlight.available(function(isAvailable) {
if (isAvailable) {
	// switch on
	window.plugins.flashlight.switchOn(
	  function() {}, // optional success callback
	  function() {}, // optional error callback
	  {intensity: 0.3} // optional as well
	);

	// switch off after 3 seconds
	setTimeout(function() {
	  window.plugins.flashlight.switchOff(); // success/error callbacks may be passed
	}, 3000);

  } else {
	alert("Flashlight not available on this device");
  }
});`

Result: The light is switched on, then a short black screen and then the camera view and the german fault message "Warnung, Kamerafehler" (warning, camera fault).
Do you have an idea, what causes this crash?
Best regards
Daniel

OK...should have said that i use the flashlight together with a camera plugin, means,
first i switch on the flash light and second the camera.
As soon as the camera viewer is shown, the error message is displayed.
But the flash light stays on.


0.8.0 requires at least glibc 2.29

I upgraded from 0.7.x (don't remember which one, sorry) to the new 0.8.0 and apparently the binary now refers to some GLIBC_2.28 and GLIBC_2.29 symbols. I'm currently running Linux Mint 19.3 which is based on Ubuntu 18.04 using glibc 2.27 so 0.8.0 doesn't run there.
I realize this is a rather old distro but it's still supported until April 2023 because it was an LTS release.
 No hard feelings if you don't and an extra incentive for me to upgrade 😄

Not sure if my PPA helps you (I am not sure if Mint can install PPAs or not):

https://launchpad.net/~gslin/+archive/ubuntu/jless

I would really appreciate binary builds supporting older GLIBC version... Current AlmaLinux 8.4 glibc is v2.28
Static build would be even better, IMO

@gslin I plucked the binary from one of your debs, and that worked for me, so I'm helped at this moment, thanks!

I looked into this page that suggest using a container that has old version of glibc. Probably this can be implemented in the GH Workflows too, according to that same page, would make the built binary support CentOS 7 as a bonus @PaulJuliusMartinez.


Openning 2 or more webcams is not possible ?

hi jyjblrd,
I ran into a problem that I cannot open 2 or more webcams at the same time on my computer. Opening first webcam is ok, but openning the second one fails. This happens when I plug in 2 or more webcams on my USB hub which connects to one USB3.0 port of my computer, however, if I plug in 2 webcams on 2 seperate USB ports of my computer, everything works fine, the error information is as follows.
when using openCV,
ret, frame = cap.read() always gets a False for ret which means openning failure
I also tried VLC, the error was
capture device might be in use
This is some information for reference.
laptop: Thinkpad L14
os: windows10 home edition 22H2
USB hub: external power supply and USB3.0 support
Can you tell me how you connect 4 cameras to your computer in your instance ?

Hi,
Has this issue been solved? (and if so, please let others know how you fixed it in case others have the same issue!)
I vaguely remember having issues if I plugged too many cameras into the same cheap USB hub, but had no issues with my powered usb hub.

I used night vision usb cameras, I never tried pseye. Pseye python library must have dealt with multi-camera problem as long as you use a powered hub.
I use Windows 10,  I got 2 USB controller ports on my computer, one connected a cheap usb hub, plugged in two cameras, the other connected a powered usb hub, plugged in other two cameras.
I adjusted camera parameter in code that make resolution to 320x240, this make sure you don't have bandwidth issue.
I started 4 individual processes to read camera frames, started a websocket server, the camera readers sent frame data to websocket server, I added some code to read from websocket server to get frames in helpers.py.

Hmm, not sure about how windows handles multiple webcams but I assume it should be fine? Unfortunately I can't help much as I don't have a windows computer or multiple webcams.


Moving parameter of a chosen element is restricted

Hello,
I tested your extension in the following link below.
https://search.daum.net/search?w=tot&DA=YZR&t__nil_searchbox=btn&sug=&sugo=&q=서울+날씨

And as you can see the chosen element has a restricted moving parameter and cannot be seen outside that parameter.
I'm not sure if it's a bug or it's intended to work that way but, if possible, enabling the chosen element to move anywhere in the browser will definitely be a good feature for the project.
Thank you

It happens because of a CSS property overflow: hidden applied to the selected element's parent elements.
We could make it work by just simply enabling the visibility of the wrapping elements. (see a screenshot below)

However, it may produce the unexpected and harmful behaviour throughout the overall website's default UI and UX. Although WEBuffet is meant to be manipulating websites as your taste, it should only affect the selected elements by the users. At worst, it may destruct the whole view structure of the page (luckily not for the given url).
In the same context, a translated element may not be shown under the other elements that have higher z-indexes.
For now, we would keep obeying the HTML box-model hierarchy and the original overflow visibility.
I'm going to specify more details about this inside the README.
Thanks for your issue report.

I close this issue because it's not a bug.


[FEATURE] Support Multiple Data Source in ML Commons Dashboards Plugin

Is your feature request related to a problem?
With multiple data source enabled, users of OpenSearch Dashboards are able to create data source connections and query local and remote OpenSearch clusters. Enabling multiple data source at dashboards plugin would ensure users that have been using the multiple data source feature to have a consistent experience in the dashboard, and allows users to use the same dashboard for different clusters.
What solution would you like?
Support multiple data source in the ml commons dashboards plugin and allow user to use one dashboard for different backend clusters
What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.
Do you have any additional context?
Add any other context or screenshots about the feature request here.

Once https://github.com/opensearch-project/ml-commons-dashboards/pull/317 is merged, we can close this issue @wanglam

The MDS has been merged to 2.x branch.


Create designs for exporting information 'report style'

The users would like to have a functionality on the map which helps them in creating reports.
Comments from user:
The Opportunity Analysis section is the part of the report that is generated from the map. We would like to be extract map pages into the report in a consistent style/format (as opposed to copying snapshots into a word document as I have document as I have done)  with fields/spaces to include text. This would include consistent way to show legends without having to show all the stuff that is currently in the workbench. Where time series data is used we would also like to be a be able to drop in charts that can be generated."

@philipgrimmett hi, it would be very helpful if we have some draft designs next week, so we are closer to making a decision on complexity, resourcing and time with our engineers and client.

@AnaBelgun I think one thing we need to get clarification on is this statement:

We would like to be extract map pages into the report

What does it mean to extract into a report? Alternate statements like "generate a report" or "create content to be pasted into a report" make sense to me. But "extract map pages into a report" sounds like they're expecting us to automate their Microsoft Word in order to push images/content into it, which I'm pretty sure is not a thing that a web site is allowed to do by web browser security.

@kring yes, that was clarified yesterday: no automated extraction and inclusion into their word or other templates.
Some suggestions we discussed:

for example, when creating a map view which has: data selected, legends, feature information panel and graphs, find a way to export that into an image (or other way) which separates that information (legend, actual map, chart, feature info); if you just copy and paste screen image the info is unclear
create a snapshot with the map view and a blanc box for free text although in the end the blanc text box is not so important, since people will include that in a word document (separately from the map)

We should link this to: https://github.com/TerriaJS/terriajs/issues/2820

Prototype: https://invis.io/35FXNA1CFGW#/280265196_workbench_With_Data

Here are some things we can allow the user to do, if we want:

Download the map image (PNG or JPG)
Download legend images. For PNG or JPG legend images we'll just give them to the user as-is. For SVG images, we may want to allow the user to download the SVG or download it as a PNG.
Set the size/resolution of the map. i.e. maybe you want to print at a higher resolution than your screen. Note that in this case there will be a "please while while we load some stuff" type thing going on. There may be a brief loading process even without letting the user specify the size/resolution.
Download a ZIP file containing all the resources, including the map image, legend images in multiple formats, text in markdown/HTML/PDF/Word format (not sure how the last two will go, but it might be doable or even easy), etc.
Print the entire exported view.


Add Migration Guide

This adds a Migration Guide outlining the steps necessary to migrate to the latest testing APIs. Based on https://github.com/emberjs/ember-mocha/pull/200 for ember-mocha.
Also fixes a mocha leftover in the Readme (hi @Turbo87 😉)

Thank you!


Correction for link to data lake product page

Issue description
The URL should be:
https://help.sap.com/docs/hana-cloud-data-lake
Feedback Type (Optional)
linking
Page Title on SAP Help Portal (prefilled)
Integrating Data to and From SAP HANA Cloud Data Lake
Page URL on SAP Help Portal (prefilled)
https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/e84545bd205b4f9f9c1731144c7d3075.html?q=data lake

Hello John,
thanks a lot for your feedback. I've updated the link with the SAP Datasphere 2024.12 version. You can find the updated documentation soon in the internal preview for Integrating Data to and From SAP HANA Cloud Data Lake.
Best Regards,
Conny (from the SAP Datasphere UA team)


add DeCals catalog for truth

Closes #623
Tasks

[x] create new class (a la PhotoFullCatalog) to read catalog into FullCatalog
[x] filter to range of stripe 82 frame we have been using (r > 336.5825) & (r < 336.7461) & (d > -1.044) & (d < -0.8189)
[x] determine correct cut in magnitude (what is the maximum magnitude detected by PHOTO in frame?)
[ ] use interactive plotting to check reasonable locations and sources are predicted
[ ] check magnitudes agree with BLISS more or less or PHOTO
[ ] recreate detection metrics but with this new catalog

Resources

Catalog containing stripe 82 data we have been using: https://portal.nersc.gov/cfs/cosmo/data/legacysurvey/dr9/south/sweep/9.0/
Description of sweep catalogs: https://www.legacysurvey.org/dr9/files/#sweep-catalogs-region-sweep (subset of tractor catalogs)
Convert flux to mags: https://www.legacysurvey.org/dr5/description/#photometry

Hmm I might need to look more into the WCS, since I can't use the SDSS one. Maybe this comment will help?

Image stacks are on tangent-plane (WCS TAN) projections, 3600 × 3600 pixels, at 0.262 arcseconds per pixel.

@dereklhansen @jeff-regier
I investigated some bright objects for which BLISS get's it easily but it seems the DeCals catalog is way off.
The conclusion: I have no idea what is going on 🤷 . Initially I thought, the skyviewer and DeCals tractor catalog I downloaded were different, now I confirmed that they are indeed the same and have the same (ra, dec) and objid for the objects I was looking into.
I carefully look into a pair of bright objects. Both of which BLISS gets it right (if you look at the SDSS image) but for one of them the tractor catalog appears way off. However, if you look at the skyviewer image it looks like DeCals is spot on for both. Initially I thought this might be a WCS error, but as you can see in the second pair of images, there are sources for which tractor and BLISS are both spot on the SDSS image and the Skyviewer image. Again, it's important the objid match to make sure we are referring to the same object, and they do.
So I'm out of leads for now...
Note: The legend should read 'galaxy decals' instead of 'galaxy coadd'

For the second object you mention, SDSS Sky Navigator seems at odds with DECaLS too:

The Gaia catalogs agree with DECaLS. The DESI spectrograph catalog also agree with DECaLS, reporting the direction as 336.6499, -0.8339.
This may be an SDSS alignment issue.

The second object is a moving star! I asked Dustin Lang about it. He writes

SDSS was taken around year 2000, and the DECam data was taken 2013-2018, and the star has moved in between!  From the Gaia-EDR3 data it has a measured proper motion of about (100, -100) milli-arcseconds per year (and a significant parallax), so in 15 years it moves a visible amount!

I decided to merge and keep everythign the same, later we can remove SDSS coadds catalog once the issues are fixed and we all agree on this (didn't want to change Derek's code)


Remove _ functions from the SyscallHandler trait

In the SyscallHandler trait implementation, we have functions with similar names like deploy and _deploy or _get_caller_address and get_caller_address, which cause some confusion.
In all these cases, the _ functions have a different implementation for each OsSyscallHandler and BusinessLogicSyscallHandler, while the functions without _ have the same implementation for both.
The idea of this issue is to remove these _ functions, to avoid confusion.
One possible solution could be to implement a SyscallWrapper struct.
pub struct SyscallWrapper<T>(T)
where
    T: SyscallHandler;

The SyscallHandler trait takes care of these _ functions that differ for each syscall. And the normal functions (the ones that have the same implementation for each syscall) can be implemented as SyscallWrapper methods.

This issue doesn't seem that complex. I this is true, I should be able to do it 😅
Can anyone confirm?