[XLA:GPU] Removing redundant scratch_allocator from miopen_conv_algorithm_picker

Incremental approach in syncing miopen_conv_algorithm_picker from upstream. This PR Removes the redundant scratch_allocator (a copy from tensorflow/compiler/xla/service/gpu/scratch_allocator.h)

Test performed
//tensorflow/compiler/xla/tests:half_test compiles and runs

@jerryyin Also please pay attention to first_call_from_algorithm_picker. I believe this is one of the very reason this pass exists.

@whchung Sounds good, thanks for the hint. It is my impression too that we should be able to abstract a decent number of common functionalities and keep minimal difference in each pass. Just so that it is clear, I wasn't look at doing the whole thing all at once (was trying to do it in G's small PR flavor :-)). Another variant I was trying to taking into consideration was Deven's on-going immediate mode implementation that makes good amount of code on par with that from cudnn picker. I will sync up with you on the follow up PRs later.

We synced offline. Closing the PR and use the commit and acutal merge of miopen/cudnn conv algorithm picker.


Allow multiplying Quantity by float

This is only applying a factor so it should be safe for any Quantity.
This PR also improve multiplication tests to use hypothesis and test with all constructors.

Based on:

#874

Reabsed on top of the merged #874, this is ready for review @daniel-zullo-frequenz /

Thanks @daniel-zullo-frequenz, I applied all your suggestions. Since it affects different commits, I'd rather leave the applied suggestions as a separate commit to avoid too much rabasing and amending.


type of nursing: non-reimbursed-services - other

http://build.fhir.org/ig/hl7-be/referral/branches/issue-91/ValueSet-be-vs-type-of-nursing.html
I suppose the list will be updated.
But what is the difference between non-reimbursed-services & other?

I thought that every possible service for a nomenclature would be added and so it would always be non-reimbursed-services?

WG: this is a remnant of a previous situation that will be removed. We need the codes from #70


Add possibility to freeze first column

For some projects it's good to know the first column for files where the first column is an identifier for the row and it has a lot of columns. Some of the files I am editing has 50+ columns so you will either have to select the row or remember the position to edit columns far to the right in the editor. This can become a bit of an issue if you are scrolling while looking at columns for different rows.
Amazing extension though, thanks.
The feature with showing the header is also very nice, could be the same for this to be honest. We are using strings to identify the id of the column instead of the index. Could be the same here as you did with the header, replace the column index with the first column. Both features would be amazing though.

I'm not sure if I understood this correctly...
The first column has an id that identifies the row (in addition to the row index)?
e.g.

id
col2
col3
...
col50

0
abb
...
...
...
...

1
bbb
...
...
...
...

2
bcb
...
...
...
...

And when you scroll to the right to see col50 you not longer see the id column and this is the issue?
If yes, are you editing such files often (should be a config option) or only occasionally (should be a setting in the extension ui)?

Exactly. I am doing this very often, it would be quite nice with a config option for it.

I've added a new config option initiallyFixedColumnsLeft and (renamed fixFirstXRows to initiallyFixedRowsTop).
The ui now also offers changing this for the current table.

Amazing thanks!

Thank you @janisdd for the freeze column setting!


Sync main and dev

This PR syncs main and dev to add the sample enhancements for the Update Graphics sample to the dev branch.

Looks fine from your latest commits.
Just a suggestion, is it worth a direct 'push' from @Rachael-E to clean it up, since we've got all those gradle / jfrog changes showing every time we sync at the moment?


Similar Model - During Training, all predictions are close to mean of training data

Hello Authors
I am trying to implement a similar model but with a graph attention network in front of the transformer to capture spatial correlations. However, during training, I am not able to capture the trend in the data for each of the network elements. (PEMS-BAY) dataset.
Could you please help with a one-liner comment on the two questions below?
1 - When training, how do the predictions from the final layer of the model look? Do they capture the trend of the input data or produce values close to the mean value of the training data?
2 - During training, what does the training loss look like in your experiments - is it lower than the final MAE reported for 60 min interval?
during training, the final layer outputs (n_timesteps, n_sensors, 1) value in one go while in testing, it does 1, n_sensors,1 for the horizon number of timesteps. In general, test loss and training loss need to converge with epochs of training, but test loss is generally slightly higher than the training loss - does that apply to this auto-regressive model? (not applicable to Informer as it does a one-shot prediction, but in general for a transformer)

We recommend our similar work at https://github.com/zhouhaoyi/POLLA


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/45116

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/52283


added auto-HPO feature with WandB

As the title says, related to #47

The core modifications are in tools/hpo, and data_juicer/config.py
The others are from pre-commit auto-modification (isort, yapf, ......)

Fixed the comments by @HYLcool , you can track the modifications in these commits: this and this.


Generalization

9abdcf5 is based on this documentation

70d89a5 -> https://github.com/facebook/jest/issues/4997

Currently, authenticated users can only view their own posts.

add mongo db integration test --> Actually, we tested the schemas in isolation. Thus, these are unit tests.


Send out Hack Night feedback and recap emails

I have a template for the feedback email I can send out, but I need @MaxWofford to send me a list of the attendees and projects from the Hack Night on Friday.

This is a dupe of #609. Closing.


Improve command rule

Potential API:
command(
  name = "hello",
  ops = [
    "ls -la somedir",
    ["ls", "-la", "somedir"],
    [ops.r(":ls"), "-la", "somedir"]
    ops.shell("ls", "-la", "somedir"),
    ops.shell(ops.r(":ls"), "-la", "somedir"),
    ops.python('print(' + ops.r(":ls") + ')'),
  ]
  cwd = "{{ os.environ['BUILD_WORKSPACE_DIRECTORY'] }}"
)

[ ] Ability to run multiple statements in a sequence
[ ] Add documentation
[ ] Add tests

[ ] Works with nested py_binary which tries to resolve runfile
[ ] Works with singular $(location ...) expansion
[ ] Works with escaping quoting characters (in env)

[ ] Use Bazel aspects to generate binstubs in /opt/bin
[ ] Use Bazel aspects to generate help message like Taskfile when running help

Possible api:
task_test(
    name = "provisioner_test",
    cmds = [
        cmd.python("""
        print("we are done!")
        """).defer(),
        "CONTAINER_ID=$($start_dev_image)",
        {"defer": "docker rm -f $CONTAINER_ID"},
        "$provision @docker/$CONTAINER_ID deploy.py",
        "$test --hosts='docker://$CONTAINER_ID'",
    ],
    desc = "Test the provisioner by creating a container, running pyinfra, and running tests.",
    env = {
        "SETUP_ENV": "test",
        "start_dev_image": cmd.shell(cmd.r("//provisioner:dev_image"), '--', '--detach'),
        "provision": cmd.r("//provisioner:provision"),
        "test": cmd.r("//provisioner:test"),
    },
)

With communication between Python and bash:
task(
    name = "connect",
    cmds = [
        "source $($aws_credentials)",
        "$connect $@",
    ],
    env = {
        "aws_credentials": cmd.python("""
        print("export AWS_ACCESS_KEY=something")
        """),
        "connect": cmd.l("//tools/dev_env:connect")
    }
)


Keybase folders not working

Keybase folders don't appear in explorer, using the Windows client. It shows an error saying: 'Your Keybase folders are not currently showing up in your Explorer. Display in Explorer'. Upon pressing display in explorer, then attempting to repair the installation, the issue is not fixed. The excerpt from the log file follows:
...
[0B5C:29B8][2017-12-15T16:38:41]i360: Creating a system restore point.
[0B5C:29B8][2017-12-15T16:38:41]w363: Could not create system restore point, error: 0x80070422. Continuing...
[11E8:3D90][2017-12-15T16:38:41]i371: Updating session, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{68547b87-03be-4f7b-ac55-338371da4432}, resume: Active, restart initiated: No, disable resume: No
[11E8:0D14][2017-12-15T16:38:41]w343: Prompt for source of package: NetFx40Web, payload: NetFx40Web, path: D:\Users\bsala\Downloads\redist\dotNetFx40_Full_setup.exe
[11E8:0D14][2017-12-15T16:38:41]i338: Acquiring package: NetFx40Web, payload: NetFx40Web, download from: http://go.microsoft.com/fwlink/?linkid=182805
[0B5C:2420][2017-12-15T16:38:44]e000: Error 0x80092026: Failed authenticode verification of payload: C:\ProgramData\Package Cache\.unverified\NetFx40Web
[0B5C:2420][2017-12-15T16:38:44]e000: Error 0x80092026: Failed to verify signature of payload: NetFx40Web
[0B5C:2420][2017-12-15T16:38:44]e310: Failed to verify payload: NetFx40Web at path: C:\ProgramData\Package Cache\.unverified\NetFx40Web, error: 0x80092026. Deleting file.
[0B5C:2420][2017-12-15T16:38:44]e000: Error 0x80092026: Failed to cache payload: NetFx40Web
[11E8:0D14][2017-12-15T16:38:44]e314: Failed to cache payload: NetFx40Web from working path: C:\Users\bsala\AppData\Local\Temp\{3EA1C895-E1F9-47AC-8C3C-8E6BD1265776}\NetFx40Web, error: 0x80092026.
[11E8:0D14][2017-12-15T16:38:44]e349: Application requested retry of payload: NetFx40Web, encountered error: 0x80092026. Retrying...
[11E8:0D14][2017-12-15T16:38:44]w343: Prompt for source of package: NetFx40Web, payload: NetFx40Web, path: D:\Users\bsala\Downloads\redist\dotNetFx40_Full_setup.exe
[11E8:0D14][2017-12-15T16:38:47]i338: Acquiring package: NetFx40Web, payload: NetFx40Web, download from: http://go.microsoft.com/fwlink/?linkid=182805
[0B5C:2420][2017-12-15T16:38:52]e000: Error 0x80092026: Failed authenticode verification of payload: C:\ProgramData\Package Cache\.unverified\NetFx40Web
[0B5C:2420][2017-12-15T16:38:52]e000: Error 0x80092026: Failed to verify signature of payload: NetFx40Web
[0B5C:2420][2017-12-15T16:38:52]e310: Failed to verify payload: NetFx40Web at path: C:\ProgramData\Package Cache\.unverified\NetFx40Web, error: 0x80092026. Deleting file.
[0B5C:2420][2017-12-15T16:38:52]e000: Error 0x80092026: Failed to cache payload: NetFx40Web
[11E8:0D14][2017-12-15T16:38:52]e314: Failed to cache payload: NetFx40Web from working path: C:\Users\bsala\AppData\Local\Temp\{3EA1C895-E1F9-47AC-8C3C-8E6BD1265776}\NetFx40Web, error: 0x80092026.
[11E8:0D14][2017-12-15T16:38:52]e349: Application requested retry of payload: NetFx40Web, encountered error: 0x80092026. Retrying...
[11E8:0D14][2017-12-15T16:38:52]w343: Prompt for source of package: NetFx40Web, payload: NetFx40Web, path: D:\Users\bsala\Downloads\redist\dotNetFx40_Full_setup.exe
[11E8:0D14][2017-12-15T16:38:55]i338: Acquiring package: NetFx40Web, payload: NetFx40Web, download from: http://go.microsoft.com/fwlink/?linkid=182805
[0B5C:2420][2017-12-15T16:38:58]e000: Error 0x80092026: Failed authenticode verification of payload: C:\ProgramData\Package Cache\.unverified\NetFx40Web
[0B5C:2420][2017-12-15T16:38:58]e000: Error 0x80092026: Failed to verify signature of payload: NetFx40Web
[0B5C:2420][2017-12-15T16:38:58]e310: Failed to verify payload: NetFx40Web at path: C:\ProgramData\Package Cache\.unverified\NetFx40Web, error: 0x80092026. Deleting file.
[0B5C:2420][2017-12-15T16:38:58]e000: Error 0x80092026: Failed to cache payload: NetFx40Web
[11E8:0D14][2017-12-15T16:38:58]e314: Failed to cache payload: NetFx40Web from working path: C:\Users\bsala\AppData\Local\Temp\{3EA1C895-E1F9-47AC-8C3C-8E6BD1265776}\NetFx40Web, error: 0x80092026.
[11E8:3D90][2017-12-15T16:38:58]e000: Error 0x80092026: Failed while caching, aborting execution.
[11E8:3D90][2017-12-15T16:38:58]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{68547b87-03be-4f7b-ac55-338371da4432}, resume: ARP, restart: None, disable resume: No
[11E8:3D90][2017-12-15T16:38:58]i371: Updating session, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{68547b87-03be-4f7b-ac55-338371da4432}, resume: ARP, restart initiated: No, disable resume: No
[11E8:3D90][2017-12-15T16:38:58]i399: Apply complete, result: 0x80092026, restart: None, ba requested restart:  No

No other symptoms are present.
I am running on Windows 10 Home (10.0.16299)

cc: @zanderz
Meanwhile you can still access folders using the keybase fs suite of commands from a terminal.

This appears to be a problem with Windows, as noted in a Microsoft support article. (https://support.microsoft.com/en-us/help/2715304/error-message-when-you-try-to-validate-a-copy-of-windows-the-cryptogra) (Since this issue was opened, I have had the same problem with a couple of other installers).
Setting the registry value as explained in the article appears to have fixed the problem.
For reference, HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\State had been set to 63c00. Setting to 23c00 fixed the issue.
Hopefully this might help others!

Thanks for finding that! :thumbsup:


Troubleshoot - Double clicking/tapping outside of troubleshoot modal reopens the modal

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.71-0
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: Yes, reproducible on both
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause Internal Team
Action Performed:

Navigate to staging.new.expensify.com
Press CTRL + D to open troubleshoot modal
Double click outside of the modal

Expected Result:
The modal closes
Actual Result:
The modal closes and reopens again
Workaround:
Unknown
Platforms:

[x] Android: Standalone
[x] Android: HybridApp
[x] Android: mWeb Chrome
[x] iOS: Standalone
[x] iOS: HybridApp
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/54a4e442-9c04-473b-a4d7-10fb22ba3cf7
View all open jobs on GitHub

I haven't' had a chance to review. Will look at this this weekend or first thing Monday morning

Can confirm this is reproducible - just need to sort which project it should fall under 😅

Chatted in Slack that this is likely not a priority - so Closing


Make optional the prevent assertion of view with 0 width and or height.

There is a possibility of false positives results hidden by #280, where we started facing them in our code base.
When we update some code that unintentionally makes the view under test have a size of .zero, the tests are still passing while it should actually fail, resulting in a false positive and hiding a possible inconsistent state.
This PR introduces an extra parameter that optionally provides the possibility to force fail or skip the failure when it happens. The default value for this parameter is  to skip failure and maintain the behaviour as it is, as mentioned on  #280
Here is a very simple exemple of how to reproduce it:
Imagine that you have a view under test already with a reference recorded.

Them for some reason, mistake or anything it became a 0 height.
Because of the diffable image that could not be generated, but the test succeed,
It is a false positive.

Now with the shouldSkipFailureOnZeroSize, you choose how to handle it.

While the false positives are definitely annoying, we're not sure adding another config option is the way to address the problem. Previously, 0x0 images were perhaps more annoying, though, as they'd crash a test run.
The core issue is that Snapshot Testing does not make it easy to encode failure into strategies. If strategies could throw, we could make failing with a 0x0 image more graceful in the future. This (as well as async concurrency support) are things we'd like to address in a future redesign of some of the core components of the library.
We're open to alternative solutions, but we're trying not to add surface area churn to existing APIs. Let us know what you think!

While the false positives are definitely annoying, we're not sure adding another config option is the way to address the problem. Previously, 0x0 images were perhaps more annoying, though, as they'd crash a test run.
The core issue is that Snapshot Testing does not make it easy to encode failure into strategies. If strategies could throw, we could make failing with a 0x0 image more graceful in the future. This (as well as async concurrency support) are things we'd like to address in a future redesign of some of the core components of the library.
We're open to alternative solutions, but we're trying not to add surface area churn to existing APIs. Let us know what you think!

Agree with you, also with the new version that i think closes this issue. o/
Thanks!


Remove node-json-db requirement, fixes #50

Seems that electron doesn't play nice with node-json-db.
Might be a filesystem access thing.
I have no idea what i'm doing, but now I have a nice shiny Zulip.app in ~/Applications that works, so hooray!

I just checked out your pull request, and ran npm run-script build-osx. The resulting app sadly still displays an error (not the same one as before, though!)
A JavaScript error occurred in the main process

Uncaught Exception:
TypeError: Error processing argument at index 0, conversion failure from undefined
    at TypeError (native)
    at set (/Users/asf/Hacks/zulip-electron/dist/Zulip-darwin-x64/Zulip.app/Contents/Resources/electron.asar/browser/api/web-contents.js:161:22)
    at EventEmitter.<anonymous> (/Users/asf/Hacks/zulip-electron/dist/Zulip-darwin-x64/Zulip.app/Contents/Resources/app.asar/app/main/index.js:211:17)
    at emitOne (events.js:96:13)
    at EventEmitter.emit (events.js:188:7)
    at EventEmitter.<anonymous> (/Users/asf/Hacks/zulip-electron/dist/Zulip-darwin-x64/Zulip.app/Contents/Resources/electron.asar/browser/api/web-contents.js:165:13)
    at emitTwo (events.js:106:13)
    at EventEmitter.emit (events.js:191:7)

I'll dig at this some, but maybe you can tell what this is about (:

It seems that conf is unset in the handler:
ipc.on('get-domain', function(e, domain) {
  e.returnValue = conf.get("domain");
});

To get my app build working, I simply replaced the call to conf.get with the constant for RC's zulip instance. Once that was set (I added it in Preferences to be safe), I could check out the unmodified code from this pull request and use the app as normal. Diagnosis: This doesn't yet handle the case where a new user has no preferences set yet.
Otherwise, this looks super good to me! I'm very excited to have an .app bundle that works, thanks so much!! ((:

configstore drops the configuration in ~/configstore/Zulip-Desktop.json - that's not quite ideal on the mac, but I think it's still pretty tolerable (:
I just tried deleting this file & starting the app. While it only showed a blank window, there was no error and I could set a server with Cmd-, - this looks much better to me!

@jedahan @antifuchs thanks for working on this. I'm using node-json-db since it's easier to use and can handle db pretty easily. I'm not sure why you got https://github.com/zulip/zulip-electron/issues/50. Electron works fine with file systems.

@antifuchs @jedahan we have released v0.0.1-alpha with dmg and .app for OS X. You can install it here https://github.com/zulip/zulip-electron/releases . Let me know if it works for you.

I just downloaded the .zip version, and it works quite well! Starts up & launches with the right icon. The only (minor) thing is that OS X complains it's not code-signed. I guess that's expected, though!

@antifuchs yes it's coming in next release 🚀 .

@jedahan Thanks for working on this ✋ .

@antifuchs

I'm very excited to have an .app bundle that works.

Pushed this feature to master. Now you run create dmg, .app etc installers using npm run dist 😄


feat(tutorial): complete step 5 - Poughkeepsie

Closes #

URL:   http://carbon-tutorial-tsirc.mybluemix.net/#/

Changelog
New

{{new thing}}

Changed

{{change thing}}

Removed

{{removed thing}}

Congratulations! 🥇 You have successfully completed all 5 sections of the carbon tutorial.  One day you will get a tshirt and badge.


Can we have something like a very minimal package?

Hello. I just found this project few hours ago and I've got quite excited about it at first. It was nice to find out that I can have power of GSAP-like timelines with Velocity animation engine... However it's quite big in size and in some... actually in many cases most of the features it comes will not be needed.
Real example: Currently I have some UI components in works (just like those fancy javascript-powered things from Bootstrap or Semantic-UI frameworks) which I'd like to release to the public in the future. But right now it's being developed in context of the project where Velocity.js is used all over the board, so those components are relying on it as well.
And it may look like switching from using Velocity directly to Tweene is a good idea, because this way it will make use of whatever existing animation engine instead, so this is probably going to be the most lightweight dependency that will enable us to have animations, right? Well, not really. Current package of Tweene is about 140kb in size uncompressed, which is more than half a size of Velocity.js size and twice bigger than GSAP's TweenLite.
I'm not familiar with the code, so I don't if it's functional without Timelines or not. I would assume that is and this alone can save us like 20% of the size. But without knowing details of the code internals it hard to even assume what components can be omitted from the bundle to get minimal working configuration.
And because of all that I would love to see some sort of minimal package or maybe even a simplified version of Tweene (like TweenLite/TweenMax in GSAP) to be a part of the project.
Because currently it's just cheaper to go with a dependency of gsap's TweenLite, rather than Tweene. And even then 70kb of extra code is still a lot

Hi,
to have a fair comparison it would be better to check the size of minified versions of the files (comments are removed).
TweenLite + TimelineLite are together something like 38KB, Velocity is 35KB, Tweene-velocity is 39KB.
Using only TweenLite you cannot do any DOM animation, because you will need also the CSSPlugin, that are other 37KB minified, and you will miss also loop feature.
Said that, I'm rewriting the whole library to have a better organization of the code and to add other cool features, like staggering and the ability to seek / jump to any time position, the new version would probably require less space, we will see.

to have a fair comparison it would be better to check the size of minified versions of the files
TweenLite + TimelineLite are together something like 38KB, Velocity is 35KB, Tweene-velocity is 39KB.
Using only TweenLite you cannot do any DOM animation, because you will need also the CSSPlugin, that are other 37KB minified, and you will miss also loop feature.

Fair notice.

I'm rewriting the whole library to have a better organization of the code and to add other cool features, like staggering and the ability to seek / jump to any time position, the new version would probably require less space, we will see.

I'm excited to see new features, but that's not exactly the thing I was asking for in OP.
What I was hoping to see, is a version with a bare minimum of features, that will be just enough to bring simple unchained animations to custom UI components, without inflating it's size too much. (Size with all the dependencies, required to make it work I mean).
Something, with help of which it will be possible to start and stop animations, to have progress and complete callbacks and maybe a way to specify easing functions. And, well, maybe with a proxy methods to get/set css properties. Without timelines, pause, repeat/yoyo features and even without bundling easings to the package, relying instead to the ones that's already available within underlying animation engine (and if the one we wanted to use isn't present - fallback to whatever default/available option is).
Judging by the API that's definitely not the use-case you had in mind while building Tweene, but since you already did all the work of wrapping various animation engines under the universal API - I thought it wont hurt if I ask about lite version. After all having such version in context of this project looks more rational to me than building it as a separate project.
But I haven't looked further though, maybe something like this already exists.


Fixed targeted tag (0.1.1 doesn't exist yet)

@obuckley-kenzan This could be an important fix, since the package.jsonis targeting a tag that doesn't exist. The repo obtained us borked.


[label_projection] seurat_transfer method out of date

Describe the bug
The seurat_transfer method  gives an error when trying to create a seurat object from anndata:
Error in validityMethod(as(object, superClass)) :
  object 'Csparse_validate' not found

To Reproduce
Steps to reproduce the behavior:

run viash test src/tasks/label_projection/methods/seurat_transferdata/config.vsh.yaml
See error

Expected behavior
the test should pass and say All checks succeeded!
Additional context
According to search results this should be fixed by updating the Matrix Package to >=1.5.3 and load the Matrix package first.
after investigation the package is at version 1.6.0. This cause has been ruled out.
After looking at the script it seems the syntax used is from Seurat version 3.1.0 whereas the version of the Seurat package installed is 4.3.0.
I have tried updating the function anndataToSeurat with following code:
obj <- Seurat::CreateSeuratObject(
    counts = adata$layers[["counts"]],
    data = adata$layers[["normalized"]],
    meta.data = adata$obs,
    assay = "RNA",
    features = adata$var_names,
    project = "anndataToSeurat"
)

This gave errors such as:
Error: None of the features provided are in this Assay object In addition: Warning message: In CreateSeuratObject.default(counts = adata$layers[["counts"]], : Some cells in meta.data not present in provided counts matrix

I've also tried to update to beta version 5 of Seurat but failed.

Disabled the component for now


feat: reset options using moreServerArgs

The change sets the default value of several linters to be false, so that they are not imposed on dependencies of mathlib, but sets them on for mathlib.
#15521 removes the ad-hoc crippling of some of the linters, to prevent them from acting on downstream projects.  Even though #15521 is not technically a dependency of this one, it probably makes sense to not merge #15521 before this one (and it should definitely be merged simultaneously/just after this one!).
The Zulip discussion about this settled on the following choices for global vs mathlib-only linter options:
Global:

globalAttributeIn
haveLetLinter
dupNamespace
unusedTacticLinter

Mathlib local:

hashCommandLinter
missingEndLinter
cdotLinter
longLineLinter
oldObtainLinter
refineLinter
setOptionLinter

Thanks for doing this! A few quick questions/requests; I'm happy to review in more detail next week.

Can you document (say, in the commit message, or somewhere in an overview comment) which linters are enabled by default and which ones are not? Eric's comment on zulip seems spot on, copy-pasting that is fine :-) I'd just like to retain that knowledge somewhere besides "some zulip thread".
Does this still enable the linters for Archive and Counterexamples? What about tests? (I quite liked that the linters previously did apply to them...)

I updated the message above, adding Mario's triaging of the linter options.
As for Archive and Counterexamples, the linters are not active there by default.  I agree that this is undesirable, but I am not sure how to make them act there, except by adding each set_option on each file :disappointed:

As for Archive and Counterexamples, the linters are not active there by default. I agree that this is undesirable, but I am not sure how to make them act there, except by adding each set_option on each file 😞

I think you can just add this to the lakefile, can't you? Like this:
lean_lib Archive where
  leanOptions := mathlibOnlyLinters.map fun s => { s with name := `weak ++ s.name }
  moreServerOptions := mathlibOnlyLinters

As for Archive and Counterexamples, the linters are not active there by default. I agree that this is undesirable, but I am not sure how to make them act there, except by adding each set_option on each file 😞

I think you can just add this to the lakefile, can't you? Like this:
lean_lib Archive where
  leanOptions := mathlibOnlyLinters.map fun s => { s with name := `weak ++ s.name }
  moreServerOptions := mathlibOnlyLinters

(and same for test)
I suggest, you abstract the part
#[ ⟨`pp.unicode.fun, true⟩, -- pretty-prints `fun a ↦ b`
   ⟨`autoImplicit, false⟩ ] ++ mathlibOnlyLinters.map fun s => { s with name := `weak ++ s.name }

as well and add the options to all 3 as @grunweg suggested.

I was not aware that worked - this is great! (Otherwise, I would have suggested adding these options in the .yml files, which is clearly not desirable.

(I'm not sure if tests require autoImplicit false, but adding it as necessary should not be too hard.

I took the liberty to implement @joneugster's suggestion. I am still amazed that lakefiles allow this!

I did some local testing: this works well for Archive and Counterexamples.
It does not work for tests; the linters are disabled there --- but this is the status quo: this appears to be because tests is a lean_exe; making it a lean_lib makes the configuration take effect. (I remember a known bug along these lines; I'm not sure if there is an issue for this.)

Note to self: make sure to adjust for whichever of #15896 and this one lands first.


Define InitHooksMetaclass to replace record_init

Now instead of requiring every class that overrides __init__ to specify @record_init, they all perform this behavior by default. This also opens the door to other kinds of validation and tracking.

This is very cool and has some nice implications for test writers, etc. I like the fact that __init__ doesn't need to be decorated in every subclass of BaseTest, which is clunky and can be problematic if missed by a test writer. I've got a couple of thoughts, though.

If a test writer adds a _before_init or _after_init method to their test, the corresponding methods further down the chain of inheritance will not be executed. Generally, I like to trust that folks won't try to do things like this, but I think it's a plausible foot gun.
The side effects of instantiating one of these objects is very mysterious and magical, which I'm never particularly fond of. It's pretty benign here, but it still is a pattern I'm not thrilled with. What would you think about something like a class decorator instead? That's explicit and obvious, can do pretty much everything a metaclass can do, and doesn't require decorating the internals of a class. We might even be able to wrap up registration into the class decorators.

If a test writer adds a _before_init or _after_init method to their test, the corresponding methods further down the chain of inheritance will not be executed.

We are okay as long as they call super()._before_init in their custom method (unit tests demonstrate this behavior). Also, it seems highly unlikely that someone would override _before_init without having understood that the base is doing something with that method, which is different than someone overriding __init__.

What would you think about something like a class decorator instead?

This suffers from a similar issue as @record_init in that Test authors have to remember to include it. It is slightly better in that all Tests would have it instead of some, so copy-and-paste is more likely to get you to the right place.
That said, I'm happy to write a PR for a @newhelm_test if you'd like to see a head to head comparison.

We might even be able to wrap up registration into the class decorators.

A single class can have multiple registrations, and I want to support registration outside of class creation (e.g. Coffee could register BBQ Tests), so I'm hesitant to go down that road.

This suffers from a similar issue as @record_init in that Test authors have to remember to include it. It is slightly better in that all Tests would have it instead of some, so copy-and-paste is more likely to get you to the right place.

I think it's fine to require test authors to include it and that being explicit there is a benefit. It's also something we can check for during registration, for example, if you weren't inclined to have registration be part of the class decoration.

A single class can have multiple registrations, and I want to support registration outside of class creation (e.g. Coffee could register BBQ Tests), so I'm hesitant to go down that road.

Yep, I understand that. We can have multiple class decorations on a class. I'm also not married to this idea, but it would:

Reduce the things a test writer needs to do in order to get a working test to 1 and;
Give us a mechanism to ensure that the stuff that needs to be there, like the recording mechanisms, are present

There could be an alternative registration method for advanced users, definitely.

Closing this PR in favor of #287 which is less magic but achieves the same goal. The only disadvantage is its less automatic, but that is worth the price.


Review section

I implemented part of the Review Section. Instead of the backend API Call, it just uses stored Object

I have fixed the delete functionality.


How to set ElasticSearch account password？

When the account and password are set on my ElasticSearch server, how can my program be set to connect to the server？

It's key=pair; syntax with known keys: https://github.com/exceptionless/Exceptionless/blob/master/src/Exceptionless.Core/Configuration/ElasticsearchOptions.cs#L66-L67
Like this: server=http://localhost:9200;username=blah;password=blah;


Got Error

I got error when run the command:
python3 src/data/create_tfrecords.py
Error:
Traceback (most recent call last):   File "src/data/create_tfrecords.py", line 8, in <module>     import cv2   File "/home/aaaa/.local/lib/python3.7/site-packages/cv2/__init__.py", line 5, in <module>     from .cv2 import * ImportError: libGL.so.1: cannot open shared object file: No such file or directory

Any one know how to fix?
Thanks

Any one know how to fix?
Thanks


Is there a way to plot multiple datasets to a single graph?

Firstly, loving the simplicity of Graphos.
I do have a question however; Is there a way to plot multiple datasets to a single graph, and if not are there any plans to introduce this feature?
i.e.
queryset1 = model.objects.filter(salesman='adam')
queryset2 = model.objects.filter(salesman='bob')
data_source = [ModelDataSource(queryset1, fields=['monthly_sales']),
                        ModelDataSource(queryset2, fields=['monthly_sales'])]

I cant find any reference in the documentation

@Xeteskian
You can do this with a SimpleDataSource. I assume you want to plot two data serieses, one for adam and other for bob. I assume you want to plot montly sales for them.
SimpleDataSource data can be visualized like this.
Month    adam    bob
Jan        500       600
Feb        410       546
March    651       630

Code for this would roughly be:
months = ['months', 'Jan', 'Feb', 'March']
adam_monthly_sales =       model.objects.filter(salesman='adam').order_by('month').values_list('monthly_sales', flat=True)

adam_monthly_sales = ['adam'] + adam_monthly_sales
bob_monthly_sales = model.objects.filter(salesman='bob').order_by('month').values_list('monthly_sales', flat=True)
bob_monthly_sales = ['bob' + bob_monthly_sales]
data = zip(months, adam_monthly_sales, bob_monthly_sales)

@Xeteskian I assume your problem is resolved with last comment, so closing this.


(For later) Change some env calls to remove defaults

To be used later!
Goal is to enforce more separation of concerns so that the pipeline has things set from env vars
Remove some defaults so that errors are more explicit when they're not provided and we don't get confused when the defaults are used, and the references are changing anyway.
ACCESSION2TAXID and LINEAGE_SHELF are moved out of common.py to the sub-files in which they are used because otherwise they'd need to be set for every command call or err.
Let me know if I should go even further or change fewer of them?

Cool. The gsnap and rapsearch index locations are actually set in idseq-infra: they get downloaded when the gsnap/rapsearch servers are set up. So you may want to raise an accompanying PR for changing those

Note for future:

Basically the explicitly set paths need to be loaded into the env somehow. E.g. first failure will be on LINEAGE_SHELF being undefined which previously would be set to 's3://czbiohub-infectious-disease/references/taxid-lineages.db'


Old Firebase profiles not merging

Hello
It has come to my attention that users created before the option to only allow one email address per user is set won't be merged.
For example:

I have a user who has logged ind using Facebook. He gets userId facebook:12345678900
I set the option to only allow one email address per user.
Now new users get a different kind of userId, this is fine. These users can merge different providers. All is good for the new users.
Now my old user wishes to log in using his Google account (maybe he switched to Android from an iPhone). His Google account is using the same email address as his Facebook account.
My old user now has two accounts in my system.

Is this by design?
Why didn't the accounts merge like the new users?
He could've just kept the old userId (I know the old userId have the word facebook in it, but still everything would have worked)
Thank you in advance.

@svangsgaard yes this is by design but I see how it can be a confusing experience.  Because there's nothing FirebaseUI can do about this I am going to close the issue, but I will file a bug with the team and see if the opinion on this has changed.


Invariant Violation Issue

connect-react-native-sdk throwing an issue while suing this module. We have followed exactly the same procedure as mentioned in the document
More details
Usage
<FinicityConnect
connectUrl={'my url'}
eventHandlers={eventHandlers}
linkingUri={'#UNIVERSAL_LINK#'}

error:-
Invariant Violation: View config getter callback for component RCTModalHostView must be a function (received undefined)
Looking for a solution

Duplicate of https://github.com/Finicity/connect-react-native-sdk/issues/2


Incorrect word wrapping around punctuation marks and furigana

It seems ttu is doing some weird word wrapping when the new line starts with some punctuation mark like  "。、・・・？「」" and so on. When it has a punctuation mark going over a new line it's removing the last moji in the line and adding it to the new one leaving an awkward blank space at the end of the last line
for example:
https://i.imgur.com/ukl8pCy.jpg or https://i.imgur.com/2IgvMyq.jpg
This also seems to happen when there are no punctuation marks but the word going over the next line has furigana over it: https://i.imgur.com/GSv7YG8.jpg
Would be nice if there was a way (or toggle to choose) to have it behave like Japanese books do where if a new line would be to start with a punctuation mark it still gets added to the end of the last line instead of having blank spaces. Example from a physical book: https://i.imgur.com/l4XbZL5.jpg
Other software like Kindle also displays this correctly: https://i.imgur.com/jtlgqh1.jpg I'm assuming what they do is that the last one or two moji space for each line is hidden and not usable by anything but punctuation mark

Hi,
you could test with with an extension if adding "word-break: keep-all;" as style to the main div container suits more your preference (though it will probably leave more space open across the page / do other breaks you don't like)
Rather than that i think its pretty limited in how the browser rendering can be influenced on this detail level and other options on js level are to heavy / not in my personal interest
Feel free to comment on the outcome of your test but this is probably the max i personally would add (though i leave it open in case someone else wants to take over / comment)

Hey thanks for answering, I just tried the keep-all option but like you said it isn't also very good and it starts new lines with punctuation marks which it's also wrong :(


[BUG] Attaching to running instance doesn't work (fzf-lua)

I am using LazyVim which uses this setup https://github.com/LazyVim/LazyVim/blob/ad22adab7d446018875de2dc96168295cad9298e/lua/lazyvim/plugins/extras/dap/nlua.lua#L31
Here is my config on top:
https://github.com/kapral18/dotfiles/blob/ad539b7b57222293fdcae58fa2dde498789a34dd/home/dot_config/exact_nvim/exact_lua/exact_plugins/osv.lua
Initially I spotted the issue when trying to debug plugins inside lazy.nvim plugin folder, but to showcase the gist of it a simplified example will do as well.

I create main.lua:

local function test()
	print(1)
	print(2)
	print(3)
	print(4)
	print(5)
	print(6)
end

vim.keymap.set("n", "<Space>0", test, { buffer = true })

I run F5 from my config and see Server started on port 8086.

I open a second nvim instance with the same file, and set breakpoint on line 3 (with print(2)).

Call nvim-dap continue and attach to 8086

I see the message "Running: Attach to running Neovim instance" but couple of seconds later got a message about  debug adapter not responding

I switch to debugee instance and run the file to make sure keymap is registered with :luafile %

I try the keymap and only see printed number in debuggee. Debugger doesn't respond.

I exit debuggee instance and the debugger suddenly pops up the dap widget UI

But at no point the breakpoint is caught.

Thank you for opening an issue.
At first glance, I don't see what is wrong. The steps (which are nicely explained) seem to be correct.
Could you try launching the server with logging enabled. This would translate to changing the following line in your config:
keys = {
    { "<F5>", [[:lua require"osv".launch({port = 8086, log=true})<CR>]], desc = "Launch OSV", noremap = true },
  },

This should output a log file within the nvim-data folder which can be found by running :echo stdpath('data'), it should be named osv.log. Please see there and post it if it can help.

@jbyuki
After I invoked with logs and terminated debuggee I get this osv.log:
{
  event = "terminated",
  seq = 1,
  type = "event"
}
{
  body = {
    exitCode = 0
  },
  event = "exited",
  seq = 2,
  type = "event"
}
{
  arguments = {
    adapterID = "nvim-dap",
    clientID = "neovim",
    clientName = "neovim",
    columnsStartAt1 = true,
    linesStartAt1 = true,
    locale = "en_US.UTF-8",
    pathFormat = "path",
    supportsProgressReporting = true,
    supportsRunInTerminalRequest = true,
    supportsStartDebuggingRequest = true,
    supportsVariableType = true
  },
  command = "initialize",
  seq = 1,
  type = "request"
}

I still cannot pinpoint what is going on. I should have asked ealier but could you log nvim-dap as well?
https://github.com/mfussenegger/nvim-dap/blob/5a2f7121869394502521c52b2bc581ab22c69447/doc/dap.txt#L984-L1005
Maybe something to try would be to change the port. It may be a port blocking issue although I doubt it.
Also make sure to update the plugins. And what is your OS/nvim version?

It would be awesome if you could log osv again. I just added a new commit to make a more exhaustive log.

MacOS Sonoma 14.5
NVIM v0.10.0
Build type: Release
LuaJIT 2.1.1713773202
Run "nvim -V1 -v" for more info
For some reason I didn't get any dap.log after setting the log to trace.
osv.log
{
event = "terminated",
seq = 1,
type = "event"
}
{
body = {
exitCode = 0
},
event = "exited",
seq = 2,
type = "event"
}
{
arguments = {
adapterID = "nvim-dap",
clientID = "neovim",
clientName = "neovim",
columnsStartAt1 = true,
linesStartAt1 = true,
locale = "en_US.UTF-8",
pathFormat = "path",
supportsProgressReporting = true,
supportsRunInTerminalRequest = true,
supportsStartDebuggingRequest = true,
supportsVariableType = true
},
command = "initialize",
seq = 1,
type = "request"
}
{
arguments = {
adapterID = "nvim-dap",
clientID = "neovim",
clientName = "neovim",
columnsStartAt1 = true,
linesStartAt1 = true,
locale = "en_US.UTF-8",
pathFormat = "path",
supportsProgressReporting = true,
supportsRunInTerminalRequest = true,
supportsStartDebuggingRequest = true,
supportsVariableType = true
},
command = "initialize",
seq = 1,
type = "request"
}
{
body = {
supportTerminateDebuggee = true,
supportsConditionalBreakpoints = true,
supportsHitConditionalBreakpoints = true,
supportsSetVariable = true
},
command = "initialize",
request_seq = 1,
seq = 1,
success = true,
type = "response"
}
{
event = "initialized",
seq = 2,
type = "event"
}

I will get back later but make sure to check
vim.fn.stdpath('cache'). It is not the same folder as the one used by osv, it should be there.

you are right my bad
here is the file

[ INFO ] 2024-05-19T01:52:10Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:53:00Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:54:09Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:55:30Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:56:02Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:56:03Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T01:56:05Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"

[ INFO ] 2024-05-19T02:20:18Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ ERROR ] 2024-05-19T02:20:44Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:965 ]	"No callback found. Did the debug adapter send duplicate responses?"	{
  body = {
    supportTerminateDebuggee = true,
    supportsConditionalBreakpoints = true,
    supportsHitConditionalBreakpoints = true,
    supportsSetVariable = true
  },
  command = "initialize",
  request_seq = 1,
  seq = 3,
  success = true,
  type = "response"
}

[ INFO ] 2024-05-19T02:26:47Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"

[ INFO ] 2024-05-19T02:57:52Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T02:58:20Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T02:58:23Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T02:59:44Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"

[ INFO ] 2024-05-19T16:20:02Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ INFO ] 2024-05-19T16:23:03Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"
[ ERROR ] 2024-05-19T16:23:21Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:965 ]	"No callback found. Did the debug adapter send duplicate responses?"	{
  body = {
    supportTerminateDebuggee = true,
    supportsConditionalBreakpoints = true,
    supportsHitConditionalBreakpoints = true,
    supportsSetVariable = true
  },
  command = "initialize",
  request_seq = 1,
  seq = 3,
  success = true,
  type = "response"
}
[ ERROR ] 2024-05-19T16:23:21Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:965 ]	"No callback found. Did the debug adapter send duplicate responses?"	{
  body = {
    supportTerminateDebuggee = true,
    supportsConditionalBreakpoints = true,
    supportsHitConditionalBreakpoints = true,
    supportsSetVariable = true
  },
  command = "initialize",
  request_seq = 1,
  seq = 5,
  success = true,
  type = "response"
}

[ INFO ] 2024-05-22T12:01:03Z+0200 ] ...al18/.local/share/nvim/lazy/nvim-dap/lua/dap/session.lua:1812 ]	"Session closed due to disconnect"

Would running this help before running launch().
:lua require"osv".stop = function() end

Then execute launch and the rest. This should avoid the early exit which is happening for some reason.

I just had a look in the config you linked. From the look of it, there is a substantial amount of plugin installed. I wouldn't be suprised if something is interacting together with osv in a bad way. The best would be to disable all plugins except a few ones and try. Thus by successive disabling/enabling, you could pinpoint what is going wrong. The plugin osv

Would running this help before running launch().
:lua require"osv".stop = function() end

Then execute launch and the rest. This should avoid the early exit which is happening for some reason.

this didn't work

I just had a look in the config you linked. From the look of it, there is a substantial amount of plugin installed. I wouldn't be suprised if something is interacting together with osv in a bad way. The best would be to disable all plugins except a few ones and try. Thus by successive disabling/enabling, you could pinpoint what is going wrong. This is a standard practice for debugging plugins. The plugin osv is supposed to work with nvim-dap.
Another possibility is the OS which I cannot test right now.

I will check with vanilla lazyvim starter then

Ok I checked. It wasn't neovim. But I found the issue the issue is this call https://github.com/kapral18/dotfiles/blob/ad539b7b57222293fdcae58fa2dde498789a34dd/home/dot_config/exact_nvim/exact_lua/exact_plugins/fzf.lua#L339
and I mean just the require, not the references, because I commented all them out and it hangs precisely on this require call. Something apparently happens inside fzf-lua actions that causes the osv to hang.
Not sure but maybe @ibhagwan has an idea?

Interesting. It was not known that fzf-lua had issues with osv
The problem is that probably someone needs to know the inner workings of both osv and fzf-lua which is most likely an empty set right now.
I don't follow how you pinpointed the error but it would be nice to pinpoint it further. So the require means it's inside this file actions.lua of fzf-lua when it's first executed. Which could be inside the utils.lua which is in turn required, possibly.

The only thing I can think of is the overload of the require method in headless instances as part of the fzf-lua libuv command proxy.
https://github.com/ibhagwan/fzf-lua/blob/e65b86532380c4e60490bc19b647d0c4faa58096/lua/fzf-lua/libuv.lua#L16
It is possible that since you’re loading fzf-lua.actions file directly the global fzf directory var isn’t being set causing libuv to think it’s running the command wrapper.
You can try changing the actions require to require'fzf.lua'.actions (so the code calls init.lua first) or just add require'fzf.lua' in the line above the line you linked?

hey @ibhagwan you absolutely hit the nail on the head, I can confirm that with require'fzf-lua'.actions it works again

hey @ibhagwan you absolutely hit the nail on the head, I can confirm that with require'fzf-lua'.actions it works again

Great!
I think I might add a global so this doesn’t happen even in this extreme edge cases.

@ibhagwan Thanks for the input! Glad this was sorted out. I will add a small note in the readme in case someone has the same issue.
@kapral18 I will close this issue now as it looks solved.

Tysm @jbyuki! And ty @kapral18 for great debugging.


Refactoring config to use local variables

Description

Type of change

[x] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[x] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue # https://issues.redhat.com/browse/OLS-379

Checklist before requesting a review

[x] I have performed a self-review of my code.
[x] PR has passed all pre-merge test jobs.
[x] If it is a core feature, I have added thorough tests.

Testing
Tested integrating with operator. Works as expected.
[vchalla@vchalla-thinkpadp1gen2 ~]$ oc logs pod/lightspeed-app-server-5c8646c79f-d994z -f
Defaulted container "lightspeed-service-api" out of: lightspeed-service-api, lightspeed-service-user-data-collector
There was a problem when trying to write in your cache folder (/.cache/huggingface/hub). You should set the environment variable TRANSFORMERS_CACHE to a writable directory.
2024-05-03 06:35:16,885 [__main__:runner.py:49] INFO: Config loaded from /etc/ols/..2024_05_03_06_33_24.3598472738/olsconfig.yaml
2024-05-03 06:35:16,886 [sentence_transformers.SentenceTransformer:SentenceTransformer.py:113] INFO: Load pretrained SentenceTransformer: /app-root/embeddings_model
2024-05-03 06:35:19,879 [sentence_transformers.SentenceTransformer:SentenceTransformer.py:231] INFO: 2 prompts are loaded, with the keys: ['query', 'text']
2024-05-03 06:35:19,880 [ols.src.rag_index.index_loader:index_loader.py:65] INFO: Setting up settings for index load...
LLM is explicitly disabled. Using MockLLM.
2024-05-03 06:35:22,789 [ols.src.rag_index.index_loader:index_loader.py:68] INFO: Setting up storage context for index load...
2024-05-03 06:35:22,792 [faiss.loader:loader.py:74] INFO: Loading faiss with AVX512 support.
2024-05-03 06:35:23,387 [faiss.loader:loader.py:76] INFO: Successfully loaded faiss with AVX512 support.
2024-05-03 06:35:23,402 [root:base.py:99] INFO: Loading llama_index.vector_stores.faiss.base from /app-root/vector_db/ocp_product_docs/4.15/default__vector_store.json.
2024-05-03 06:35:23,582 [ols.src.rag_index.index_loader:index_loader.py:81] INFO: Loading vector index...
2024-05-03 06:35:23,582 [llama_index.core.indices.loading:loading.py:66] INFO: Loading indices with ids: ['ocp-product-docs-4_15']
2024-05-03 06:35:24,486 [ols.src.rag_index.index_loader:index_loader.py:86] INFO: Vector index is loaded.
2024-05-03 06:35:34,575 [ols.app.main:main.py:30] INFO: Embedded Gradio UI is disabled. To enable set `enable_dev_ui: true` in the `dev_config` section of the configuration file.
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8080 (Press CTRL+C to quit)
2024-05-03 06:35:55,608 [ols.utils.auth_dependency:auth_dependency.py:192] WARNING: Auth checks disabled, skipping
2024-05-03 06:35:55,609 [ols.app.endpoints.ols:ols.py:82] INFO: User ID 00000000-0000-0000-0000-000000000000
2024-05-03 06:35:55,609 [ols.app.endpoints.ols:ols.py:147] INFO: b201e7c8-9ea8-42ba-a864-ddf8999c0d0e New conversation
2024-05-03 06:35:55,609 [ols.app.endpoints.ols:ols.py:88] INFO: b201e7c8-9ea8-42ba-a864-ddf8999c0d0e Incoming request: write a deployment yaml for the mongodb image
2024-05-03 06:35:55,609 [ols.src.query_helpers.question_validator:question_validator.py:48] INFO: b201e7c8-9ea8-42ba-a864-ddf8999c0d0e call settings: conversation_id: b201e7c8-9ea8-42ba-a864-ddf8999c0d0e, query: write a deployment yaml for the mongodb image, provider: bam, model: ibm/granite-13b-chat-v2, verbose: False
2024-05-03 06:35:56,220 [httpx:_client.py:1758] INFO: HTTP Request: GET https://bam-api.res.ibm.com/v2/text/generation/limits?version=2023-11-22 "HTTP/1.1 200 OK"
2024-05-03 06:35:58,029 [httpx:_client.py:1758] INFO: HTTP Request: POST https://bam-api.res.ibm.com/v2/text/generation?version=2024-03-19 "HTTP/1.1 200 OK"
2024-05-03 06:35:58,046 [genai.extensions.langchain.llm:llm.py:200] INFO: Output of GENAI call: ALLOWED

Batches: 100%|██████████| 1/1 [00:06<00:00,  6.21s/it]
2024-05-03 06:36:04,666 [httpx:_client.py:1758] INFO: HTTP Request: GET https://bam-api.res.ibm.com/v2/text/generation/limits?version=2023-11-22 "HTTP/1.1 200 OK"
INFO:     Shutting down
INFO:     Waiting for connections to close. (CTRL+C to force quit)
2024-05-03 06:36:24,886 [httpx:_client.py:1758] INFO: HTTP Request: POST https://bam-api.res.ibm.com/v2/text/generation?version=2024-03-19 "HTTP/1.1 200 OK"
2024-05-03 06:36:24,888 [genai.extensions.langchain.llm:llm.py:200] INFO: Output of GENAI call: ```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
spec:
  selector:
    matchLabels:
      app: mongodb
  replicas: 3
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:4.4
        ports:
        - containerPort: 27017

This YAML file creates a deployment called `mongodb-deployment` with three replicas of the MongoDB image (version 4.4). The container exposes port 27017, which is the default port for MongoDB. Make sure to replace the image name and port number according to your requirements.

Keep in mind that this is just an example, and you might need to adjust the configuration based on your specific use case, such as adding environment variables, configuring storage, or setting up network policies.

For more information on OpenShift and Kubernetes Deployments, you can refer to the official documentation:

* [OpenShift Deployments](https://docs.openshift.com/container-platform/4.5/dev_guide/deploying_applications/index.html)
* [Kubernetes Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
2024-05-03 06:36:24,888 [ols.app.endpoints.ols:ols.py:229] INFO: b201e7c8-9ea8-42ba-a864-ddf8999c0d0e Storing conversation history.

Codecov Report
Attention: Patch coverage is 88.37209% with 15 lines in your changes are missing coverage. Please review.

Project coverage is 91.35%. Comparing base (c52740f) to head (d512843).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #865      +/-   ##
==========================================
- Coverage   94.57%   91.35%   -3.23%
==========================================
  Files          55       55
  Lines        2011     2070      +59
==========================================
- Hits         1902     1891      -11
- Misses        109      179      +70

Files
Coverage Δ

ols/app/endpoints/feedback.py
70.21% <100.00%> (-29.79%)
:arrow_down:

ols/app/endpoints/ols.py
97.94% <100.00%> (+0.02%)
:arrow_up:

ols/src/llms/llm_loader.py
100.00% <100.00%> (ø)

ols/src/llms/providers/provider.py
100.00% <100.00%> (ø)

ols/src/query_helpers/docs_summarizer.py
100.00% <100.00%> (ø)

ols/src/query_helpers/query_helper.py
100.00% <100.00%> (ø)

ols/utils/query_filter.py
100.00% <100.00%> (ø)

ols/utils/token_handler.py
100.00% <ø> (ø)

ols/utils/auth_dependency.py
73.14% <83.33%> (+0.50%)
:arrow_up:

ols/src/query_helpers/question_validator.py
54.54% <33.33%> (-14.21%)
:arrow_down:

... and 2 more

... and 2 files with indirect coverage changes

For discussion: I would find something like this
from ols.configuration import config

config.ols_config.user_data_collection.feedback_disabled
config.rag_index
config.ols_config.default_model
config.dev_config.run_on_localhost
...

more understandable and straightforward. No additional shortcuts for config subdivisions (llm_config,dev_config,...) that only complicates the whole thing. The config can be as simple as a module constant (initialized Config instance).
Try to imagine the Config (base pydantic model) as a wrapper for the individual configurations - like olsconfig.yaml for its content.

I agree with @onmete

@onmete @tisnik Please bear with me as this PR is still in progress. I will retag once I update it. Thanks

For discussion: I would find something like this
from ols.configuration import config

config.ols_config.user_data_collection.feedback_disabled
config.rag_index
config.ols_config.default_model
config.dev_config.run_on_localhost
...

more understandable and straightforward. No additional shortcuts for config subdivisions (llm_config,dev_config,...) that only complicates the whole thing. The config can be as simple as a module constant (initialized Config instance).
Try to imagine the Config (base pydantic model) as a wrapper for the individual configurations - like olsconfig.yaml for its content.

Please help me understand on what you exactly meant by pyndantic approach here, how will we even be able to invoke config.ols_config without ols_config being a globally available attribute of the config class?

It means using the wrapper class ols.app.models.config.Config we already have, as the config entrypoint. You can access every part of the configuration from there (if something is missing, it should be added).
Regarding initialization, something like this could work.
# in the config.py
with open(os.environ.get("OLS_CONFIG_FILE", "olsconfig.yaml"), "r", encoding="utf-8") as f:
    data = yaml.safe_load(f)
config = config_model.Config(data)

The config is module constant/singleton.
It is just a though, there might be a better way how to set it up.

Further changes to be done using pydantic approach. poc pr: https://github.com/openshift/lightspeed-service/pull/877


H3Cell and H3Edge debug in hexadecimal

I added a custom Debug implementation for H3Cell and H3Edge to avoid seeing the H3Index in decimal which is useless

That is quite a bit nicer than before - thank you.


Action in Reminder runs but message is not displayed on command line

Hi
I am facing an issue where the scheduler (reminder scheduled) job starts and action runs, but the message is only visible in logs and not on command line.
I have upgraded to latest version of Rasa Core:
rasa_core==0.13.2
rasa_core_sdk==0.12.2
rasa_nlu==0.14.4
Python==3.6.5
OS: Mac OSX
Here is the screen shot of logs for your reference:

Please let me know, if you any other details to resolve this.

Thanks for raising this issue, @Ghostvv  will get back to you about it soon.

It is duplicate issue, please see: https://github.com/RasaHQ/rasa_core/issues/886

Hi @Ghostvv, I want to reopen the issue because of the following problem.
I tried to do put the print statements as suggested in #886, but the problem is that it prints on the command line and goes to 'action_listen'. However, as part of my story, I expect to trigger another action after this action is complete, but that never happens. I am attaching the screenshot of my logs:

I am also attaching the story:
## story02
  - action_bedtime_reminder
  - slot{"sleepQualityRating" : null}
  - slot{"refreshQualityRating" : null}
  - slot{"nightType": null}
  - action_trigger_query_sleep_api

So, after the bedtime reminder action is ran and message is printed on command line (using print statements), it goes to 'action_listen'. I expect it to trigger action 'action_trigger_query_sleep_api'.

this is another issue, regarding policy prediction.
In your case story02 doesn't correspond to the tracker state as can be seen from the logs up there. I recommend using augmented memoization policy

@Ghostvv thanks for the solution. It works fine :)


Docs refer to config file named after the cluster but they have a random name

Description
In describing how to get the clusterconfig the documents suggest that the files should be named according to the management cluster name.
If you deployed the management cluster from the installer interface, the configuration file is here: ~/.config/tanzu/tkg/clusterconfigs/<MGMT-CLUSTER-NAME>.yaml

However this is not the case (see https://github.com/vmware-tanzu/community-edition/issues/2172).
Location

Additional Context
This can be fixed by addressing the underlying issue in tanzu framework, or updating the docs.

Thanks for raising the awareness on this. Let's continue on with https://github.com/vmware-tanzu/community-edition/issues/2172


Add #![no_std] to allow no_std codebase to use your library

Your library seems to not depend on std, it would be a waste if no_std codebase cannot use this.

[dependencies.wyz]
default-features = false


Any option to use emphasis inside figcaption?

Hello, first thanks for creating the plugin, it has been useful for me.
I was wondering if there is any option to use emphasis inside the figcaption, like bold, italic or both?

This plugin transforms the alt text you specify for images using standard markdown syntax.
![Image](http://url/a.png)

becomes

Image

The alt text is treated as plain text by default in most markdown parsers and is rendered as such. Straying away from this behavior is out of scope of this plugin.

If you really want to render something in bold or italics in the figcaption, I'd recommend you to use raw HTML in markdown.

Emphasized Image

If you need to render the captions in bold or italics, you can customize this with CSS.
figcaption {
  font-weight: 700; /* for bold */
  font-style: italic;
}


feat(Posture Management): Add posture management service sdk - DRAFT

PR summary
This PR contains the following for Posture Management

generated service SDK
generated unit tests
generated integration tests
generated examples

Further todo before merge:

[x] align integration tests
[x] introduce configuration values via .env file with green integration tests
[x] double-check URL in the README.md file
[x] double check examples

PR Checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Current vs new behavior

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

Other information

I close this pull request and open a new one because the web ui can't resolve the single commit, and I'd like to avoid using force push


[C++] Stop defining ARROW_TEST_MEMCHECK in config.h.cmake

Describe the bug, including details regarding any error messages, version, and platform.
We already have the ARROW_VALGRIND macro for Valgrind. We don't need to define a new macro for Valgrind.
See also: https://github.com/apache/arrow/pull/41163#discussion_r1562204543
Component(s)
C++

Issue resolved by pull request 41177
https://github.com/apache/arrow/pull/41177


Fresh k3s v1.23.3+k3s1 airgap install, can't fetch mirrored-pause:3.6

Environmental Info:
K3s Version:
k3s version v1.23.3+k3s1 (5fb370e5)
go version go1.17.5

Node(s) CPU architecture, OS, and Version:
Linux node-a0 5.15.33-v8+ #1 SMP PREEMPT Thu Apr 14 09:47:24 CDT 2022 aarch64 GNU/Linux
Custom debootstrap Debian build with custom Linux 5.15.33-v8+ on a Raspberry Pi 4

Cluster Configuration:
Airgapped cluster composed of 9 Raspberry Pi 4 nodes named from node-a0 to -a9. node-a0, -a5, and -a9 are HA servers. The rest are agents. All are network booted using DHCP and ATA over Ethernet for their root storage, served from another Intel based system called diskful-aa, which also runs a Docker registry with all of the airgap images loaded. The non-k3s network is <IP_ADDRESS>/24, with diskful-aa on <IP_ADDRESS>.
Installation of k3s on server node-a0 has been done using the k3s install script and k3s binary stored on disk as /usr/local/bin/install.sh and /usr/local/bin/k3s, respectively, and installed like this:
export INSTALL_K3S_EXEC=server
export INSTALL_K3S_SKIP_DOWNLOAD=true
export K3S_TOKEN=xxxxxx
/usr/local/bin/install.sh --cluster-init

Nodes are not configured with public routing or public DNS resolution. After reading through a number of these kinds of bugs, I've come up with the following registries.yaml configuration on the server. Note that <IP_ADDRESS> is diskful-aa.local and contains the registry:
---
mirrors:
  "docker.io":
    endpoint:
      - "https://<IP_ADDRESS>:5000"
  "*":
    endpoint:
      - "https://<IP_ADDRESS>:5000"
  "":
    endpoint:
      - "https://<IP_ADDRESS>:5000"
  "<IP_ADDRESS>":
    endpoint:
      - "https://<IP_ADDRESS>:5000"

configs:
  "<IP_ADDRESS>:5000":
    tls:
      insecure_skip_verify: true

Describe the bug:
After running the installation script for k3s, the binaries are setup appropriately, and the k3s kubelet is started, and I can query it using sudo kubectl get pods, but none of the pods manage to start. Instead, I see the following output from kubectl describe -n kube-system pod/coredns-5789895cd-9dbnn:
...
  Type     Reason                  Age                      From     Message
  ----     ------                  ----                     ----     -------
  Warning  FailedCreatePodSandBox  3m25s (x12071 over 43h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:44097->[::1]:53: read: connection refused

It looks like, despite the fact that I've configured registries.yaml to rewrite docker.io, * and "" to https://<IP_ADDRESS>:5000, containerd is still trying to fetch from registry-1.docker.io.

Steps To Reproduce:

Installed K3s:
Verify the system is running using kubectl --all-namespaces get pods

Expected behavior:

I should see all pods in the Running state.

Actual behavior:

Pods are stuck in the ContainerCreating state.

Additional context / logs:
Logs continue to spew the same event over and over, which is that registry-1.docker.io is unresolveable (which is fine -- it's not supposed to be). Happy to attach more from the logs, though I'm not sure they'll be entirely all that useful.

Backporting

Does not need backporting -- happy to upgrade to latest, if necessary.

I've come up with the following registries.yaml configuration on the server

registries.yaml is not cluster-level configuration. You must distribute a copy of this file to each node you want to use your private registry configuration.

I realize that. registries.yaml is present in the form I report above on all nodes in the cluster.
I guess I wasn't quite clear enough: only node-a0 is actually installed at this point. I haven't installed k3s on the other nodes yet because I was waiting for a0's control plane to be ready when I noticed the pods hadn't started yet. So at this point in time, there is exactly one member of the cluster: node-a0.

I realize these kinds of bugreports are a bit of a bugbear for ya, and I almost didn't report this one because I didn't want to harass you. Are there any other documents I can go read through for how containerd that would help me resolve this?

registries.yaml is just templated into the much less nicely formatted containerd config at /var/lib/rancher/k3s/agent/etc/containerd/config.toml. All the actual functionality of trying the various endpoints is build into containerd and documented here: https://github.com/containerd/containerd/blob/release/1.5/docs/cri/registry.md#configure-registry-endpoint

Yeah, I kept trying restarts of k3s via systemctl restart k3s and also by uninstalling/reinstalling, to no avail. It's hard to consider reporting this against containerd, though, since it's bundled inside of k3s. If it was a system service I had to install, I'd be more inclined to report it there instead.
I'll see about checking the TOML directly and see if that makes any sense and report back. Thank you, Brandon!

So my /var/lib/rancher/k3s/agent/etc/containerd/config.toml is this:
[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "<IP_ADDRESS>"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/41bc7e830c39dee90b39295b0d9c7b456c4e00284499a23d7d40cbca7a547d89/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.cri.registry.mirrors]

[plugins.cri.registry.mirrors.""]
  endpoint = ["https://<IP_ADDRESS>:5000"]

[plugins.cri.registry.mirrors."*"]
  endpoint = ["https://<IP_ADDRESS>:5000"]

[plugins.cri.registry.mirrors."<IP_ADDRESS>"]
  endpoint = ["https://<IP_ADDRESS>:5000"]

[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://<IP_ADDRESS>:5000"]

[plugins.cri.registry.configs."<IP_ADDRESS>:5000".tls]
  insecure_skip_verify = true

According to the page you linked, it seems to be correct, though the form is different (not version 2). Doing some additional checks with crictl pull rancher/mirrored-pause:3.6 results in the same output as what I was seeing in kubectl:
root@node-a0:~# crictl pull rancher/mirrored-pause:3.6
FATA[0001] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:33132->[::1]:53: read: connection refused

And to prove the registry is actually working:
root@node-a0:~# crictl pull <IP_ADDRESS>:5000/mirrored-pause:3.6
Image is up to date for sha256:7d46a07936af93fcce097459055f93ab07331509aa55f4a2a90d95a3ace1850e
root@node-a0:~#

The containerd registry mirror stuff is generally quite stable; all of the issues I've seen in recent history have ended up being a misconfiguration of some sort.
You didn't mention it, but you're not using --docker or your own --container-runtime-endpoint are you?
Have you tried removing the "*" and "" entries?

I don't doubt it's a misconfig somewhere -- question is where? I've been able to narrow it down to containerd with your pointers, thankfully. :)
No, I'm not using --docker or --container-runtime-endpoint. I'm relying on k3s for everything, actually. I was originally trying this with the simplistic form listed in the k3s docs, with just entries for docker.io and my registry config, with much the same results.

On a lark, I added an entry for rancher, thinking the problem would be something about the prefix in rancher/mirror-pause, and got the same results with crictl pull. Likewise, removing "*" and "" from the lists produces the same results after restarting k3s.

Are there logs from containerd that I can use to help diagnose this issue? With the way k3s starts up, it's not obvious how I can get containerd into a debug mode or log anything else that would be useful.

Pay dirt! containerd logs to /var/lib/rancher/k3s/agent/containerd/containerd.log. And here's what I've found in there, which is quite telling:
time="2022-04-21T02:24:16.394813375Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:coredns-5789895cd-9dbnn,Uid:52a197d7-df88-4995-94b6-8c572913d267,Namespace:kube-system,Attempt:0,}"
time="2022-04-21T02:24:16.415829256Z" level=info msg="trying next host - response was http.StatusNotFound" host="<IP_ADDRESS>:5000"
time="2022-04-21T02:24:16.419561955Z" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\": dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:41642->[::1]:53: read: connection refused" host=registry-1.docker.io

So apparently it's attempting to find rancher/mirrored-pause on my server, but my server is returning a 404. It's not clear what actual URL it's using to do the fetch, though.

Yeah, as the containerd docs mention, it will always fall back to the default endpoint. Unfortunately the CRI API is bad about bubbling up multiple errors, so all you'll see in crictl or the kubelet logs is the terminal failure when it can't pull from the default docker.io endpoint. To see whether or not it's failing to pull from your registry you need to look at the actual containerd logs.
At this point I would probably go look at the logs for your registry mirror and see what it has to say.

Okay, checking the registry, I see stuff like this in its logs:
time="2022-04-19T21:19:34.875490266Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=3.6" err.message="manifest unknown" go.version=go1.11.2 http.request.host="<IP_ADDRESS>:5000" http.request.id=916dc685-7d47-4352-8d1c-43ea44887187 http.request.method=HEAD http.request.remoteaddr="<IP_ADDRESS>:42048" http.request.uri="/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io" http.request.useragent="containerd/v1.5.9-k3s1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.143157ms http.response.status=404 http.response.written=93 vars.name="rancher/mirrored-pause" vars.reference=3.6

It appears containerd is requesting properly. The log messages in kubectl, sadly, don't reflect this mess since it captures this. ;P
So I think I probably pushed the images into the registry incorrectly which is the "misconfiguration" you mentioned before. In this case, I used an automated script I wrote to do the job using the k3s-images.txt and the tarball, which is here:
#!/bin/bash

registry="diskful-aa:5000"

function die
{
    echo "*** $@ exited with error code $?"
     Bleh. I'll stop bothering you about this now -- thank you Brandon, your help was invaluable! :)

There may be issues with how you're tagging and pushing things, but since you mentioned that you have a mix of Intel and arm64 nodes, you should be aware that docker can do dumb things when loading and pushing cross-arch images. The tarballs contain only platform-specific images, but if you're running a registry you really want the multiarch manifest lists pushed to each tag, not just a single-platform manifest. You might look at using a tool like skopeo or podman instead of just docker. There's some ongoing discussion of mirroring tools over at
https://github.com/rancher/rke2/issues/2732#issuecomment-1097342396

Oh, diskful-aa doesn't participate in the cluster, so it's not really a mixed arch cluster, fortunately. I've done that with raw K8S back in the day and it was a nightmare, so no more of that kind of crazy for me. It's just a machine that exposes slices of a really big disk array out to the network via NFS and AOE, and also happens to be a convenient place for the repository.
I'll definitely take a look at skopeo or podman, though. I'd much prefer an OSS solution to Docker's closed source madness. :P
Thanks again! :D

Okay, so to put the final fix to make this work for future folks looking at issues trying to figure this out, you have to push your containers with the proper names, which isn't clear in the K3S documentation, really. So, the airgapped docker image names look like this in the k3s-images.txt file:
docker.io/rancher/klipper-helm:v0.7.1-build20220407
docker.io/rancher/klipper-lb:v0.3.5
docker.io/rancher/local-path-provisioner:v0.0.21
...

The trick is, you can't keep the docker.io bit because that's actually the registry name. In my script above, I was erroneously stripping out the rancher bit in the image names and prepending my registry name before it, producing names like this:
localhost:5000/docker.io/klipper-helm:v0.7.1-build20220407

Obviously, this is horribly wrong. What they actually need to look like are this:
localhost:5000/rancher/klipper-helm:v0.7.1-build20220407

The docker command line tools don't make this obvious because they really really want you to use docker.io as the registry. If you use podman instead, it's much more explicit about this kind of thing.
If you end up using my script to automate the loading using the docker tooling, you'll want this patch to fix it:
diff --git a/ansible/roles/registry/files/push-airgapped-images.sh b/ansible/roles/registry/files/push-airgapped-imas.
sh
index 690274f..6735e85 100755
--- a/ansible/roles/registry/files/push-airgapped-images.sh
+++ b/ansible/roles/registry/files/push-airgapped-images.sh
@@ -16,17 +16,19 @@ function try

 images=$(cat k3s-images.txt |sed 's,^docker.io/,,')

-try docker load -qi k3s-airgap-images-amd64.tar
+try docker load -qi k3s-airgap-images-arm64.tar

 for fullname in $images; do
-    imagename="${fullname#rancher/}"
-    tagname="${registry}/${imagename}"
+    tagname="${registry}/${fullname}"
     try docker tag "${fullname}" "${tagname}"
     try docker push "${tagname}"
 done

+exit
+
 for imageid in $images; do
     base_imagename=${imageid%:*}
     image_hash=$(docker images |grep $base_imagename |awk '{ print $3 }')
     try docker rmi --force $image_hash
 done


Working with WebCore

Not a bug but a note...
when using webcore, the switches (only tested these) dont show in the main actuator list.
You need to look in the 'Compatibiliy Group 3 -> switches' section.

Are you talking about the tasmotaswitchsensor and other device types that I created
I guess I could add the actuator capability - anyway atomicState doesnt work for device handlers so still working on updating the device handler - essentially if you are hitting the refresh - sometime you will have to refresh a 2nd time to get all updates synced up.

they now all appear in the default area like other switches.


Fedora build problems

On Fedora 28 :
CC=/usr/bin/clang CXX=/usr/bin/clang++ ./build.sh debug
... works (though bluegl doesn't work on 32 bit systems).
If I do :
./build.sh debug
... I get "Could NOT find Threads (missing: Threads_FOUND)" .  Dunno why.
Also, if I do
./build.sh -m debug
... I get : c++: error: unrecognized command line option ‘-stdlib=libc++’
This is GNU 8.1.1

Make sure you have pthread-dev installed. Note that we do not support building on desktop 32 bit systems, only 64 bit systems. We do support 32 bit systems for Android.
Which version of clang are you using? Could you please paste the output of cmake right after you run the build script?

I don't see pthread-dev . I see /usr/include/pthread.h , which is part of glibc-headers
clang is 6.0.1
Here's the output of ./build.sh debug :
Warning: JAVA_HOME is not set, skipping Java projects
Building debug in out/cmake-debug...
-- The C compiler identification is GNU 8.1.1
-- The CXX compiler identification is GNU 8.1.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake/Modules/FindThreads.cmake:205 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
third_party/libgtest/tnt/CMakeLists.txt:26 (find_package)
-- Configuring incomplete, errors occurred!
See also "~/filament/out/cmake-debug/CMakeFiles/CMakeOutput.log".
See also "~/filament/out/cmake-debug/CMakeFiles/CMakeError.log".

Oh wait ... looking at cmake-debug/CMakeFiles/CMakeError.log :
cc: error: unrecognized command line option ‘-fcolor-diagnostics’

-fcolor-diagnostics is only supported by clang, not gcc. We do not support building with gcc.

OK ... I though you did both gcc & clang.  Now I see in the readme that you don't do gcc.
Thanks

Is it possible to make  a cmake rule that clearly states that gcc is not supported (instead of obscure errors about pthreads) ?

Yep, that would be wise :)


Run 'npm audit fix' to update vulnerable dependencies

Changes

Ran npm audit fix
Modified the start:firefox command because it wasn't working

merged into master


Sort Analyzer autocomplete suggestions by applicable types

The first results should be types which are most relevant to the current assignment.
For example, if I'm passing an argument which annotated with type Foo to a function, and I type new  and ask for suggestions from the analyzer, I should first see suggestions for constructors of types which will implement Foo.
Originally filed as https://github.com/dart-atom/dartlang/issues/143

Duplicate of https://github.com/dart-lang/sdk/issues/26254 which is now fixed.


Option to store only columns that have changed in item_versions

When browsing a list of item versions like this one it's difficult to tell at a glance which rows have changed since the previous version:

It would be neat if there was a mode that could only store values in the versions table for columns that have changed since the last version.

This could also help shrink the size of the overall database (see also #12).
One catch: does null mean "this value did not change" or does it mean "this value was set to null"?
I'm going to see if I can address that using a bitmap - an additional column, probably called _changed (though I'm worried that might be confused for a timestamp) which stores an integer which represents which columns changed since the previous item.
A custom Datasette plugin could read that bitmap and use it to show extra information when the table is displayed.

I started my prototype by writing a _changed column that contained a JSON list of columns, rather than a bitmap: ["AcresBurned", "PercentContained", "ConditionStatement", "Updated", "PersonnelInvolved", "CrewsInvolved", "Helicopters", "Engines", "Dozers"]
I really like this, but it's very verbose and consumes a lot of table space.
So... how about turning that into a lookup table? So the JSON value becomes a key in another table?
This would save on space, and display reasonably well. Running a query for "all versions that had an updated Dozers column" would be possible but would involve a join against that table plus some JSON shenanigans, which wouldn't be particularly simple for users.
Probably simpler than dealing with a bitmap column though!

Alternative formats for the _changed column:

JSON array of column string names - verbose but easy to query
Integer that foreign keys to a string that's a JSON array, using table.lookup() in sqlite-utils
JSON array of column offset integers
Single integer bitmap of column offset integers
,3,5,7, style string, which allows for where _changes like '%,3,% queries - could even add a trigram index?
JSON array of integers that refer to column names stored in a lookup table
a full many-to-many relationship

Maybe the many-to-many relationship would be the best way to handle this? I'm slightly put off because Datasette doesn't have facet by many-to-many yet - I mostly implemented that a few years ago but never finished it, see https://github.com/simonw/datasette/issues/551

After sweating over this decision for a full week, I'm going to try the m2m table and see how well it works.
I may try turning it back into a JSON list for one of the views.

That commit is my initial prototype of a _changed column which stores a full JSON list of column names.
I'm going to try using m2m instead next.

Since item_versions currently uses a compound primary key I'm ending up with a m2m table that relates against a compound foreign key, which doesn't work great with Datasette yet. Code looks something like this:
                    db["item_versions"].insert(
                        item_version,
                        pk=("_item", "_version"),
                        alter=True,
                        replace=True,
                        column_order=("_item", "_version", "_commit"),
                        foreign_keys=(
                            ("_item", "items", "_id"),
                            ("_commit", "commits", "id"),
                        ),
                    )

                    if changed_columns:
                        # Record which columns changed in the changed m2m table
                        for column in changed_columns:
                            db["changed"].insert({
                                "_item": item_id,
                                "_version": version,
                                "column": db["columns"].lookup({"name": column})
                            }, pk=("_item", "_version", "column"), foreign_keys=(
                                ("column", "columns", "id"),
                                ("_item_id", "items", "_id"),
                            ))

I think I'll switch item_versions to having a regular integer primary key instead to make the m2m table more predictable.

The changed table looks like this:
CREATE TABLE [changed] (
   [item_version] INTEGER REFERENCES [item_versions]([id]),
   [column] INTEGER REFERENCES [columns]([id]),
   PRIMARY KEY ([item_version], [column])
);

This makes it difficult to facet by item in the Datasette interface because you have to first go through item_version - I can fix that by adding a changed_details view or similar that makes that join for you.

% sqlite-utils create-view ca-fires-m2m-2.db changed_details '
select
  items._item_id,
  items._id as _item_pk,
  item_versions._version,
  columns.name
from
  changed
  join item_versions on changed.item_version = item_versions.id
  join columns on changed.column = columns.id
  join items on item_versions._item = items._id'

That prototype worked, but highlighted that without foreign key links on views this interface isn't as useful as I'd like it to be:

I'm tempted to address that problem using a Datasette render_cell() plugin, which looks out for a table (actually a view) with that name and columns of that name and turns them into hyperlinks to the right place.

Building a Datasette plugin that imports an existing Click CLI tool and re-registers it is proving hard - Click doesn't really want you to do that. I tried this:
from datasette import hookimpl
from git_history.cli import file as git_history_file

@hookimpl
def register_commands(cli):
    cli.command(name="git-history")(git_history_file.callback)

But when I run this:
 % datasette git-history --help
Usage: datasette git-history [OPTIONS]

  Analyze the history of a specific file and write it to SQLite

Options:
  --help  Show this message and exit.

The options are all missing - which means that the command doesn't actually work. Will need to research this pattern separately.

Figured out a pattern that works: https://til.simonwillison.net/datasette/reuse-click-for-register-commands

Here's the full plugin, adding hypelinks to that view:
from datasette import hookimpl
import markupsafe
from git_history.cli import cli as git_history_cli

@hookimpl
def register_commands(cli):
    cli.add_command(git_history_cli, name="git-history")

foreign_keys = {
    # table, column => other-table
    ("changed_details", "_item_version_pk"): "item_versions",
    ("changed_details", "_item_pk"): "items",
}

@hookimpl
def render_cell(value, column, table, database, datasette):
    if (table, column) in foreign_keys:
        other_table = foreign_keys[(table, column)]
        return markupsafe.Markup(
            '{label}'.format(
                href=markupsafe.escape(
                    "{}/{}".format(
                        datasette.urls.table(database, other_table), value
                    )
                ),
                label=value,
            )
        )

Created another view to join _commit_at date to item_versions:
% sqlite-utils create-view ca-fires-m2m-2.db item_versions_details '
select commits.commit_at as _commit_at, commits.hash as _commit_hash, item_versions.*
from item_versions join commits on item_versions._commit = commits.id'

I shipped Datasette 0.59.4 with fixes for a couple more bugs that came up with the _item_id underscore prefixed column names: https://docs.datasette.io/en/stable/changelog.html#v0-59-4

OK, I like the m2m version of this. I'm going to add documentation and tests and ship it.

I'm tempted to make this the default, and have an option for recording full duplicate records in the item_versions table.

If this was the dddsult behaviour, what would the option to store full records instead of only the changes be called?
I think --full.

This is mostly done, I still need to add documentation and tests - and maybe switch the default behaviour.

I'm testing this against sf-tree-history now.
(git-history) git-history % time git-history file sf-trees-changed.db ../sf-tree-history/Street_Tree_List.csv --repo ../sf-tree-history --changed --id TreeID --csv
  [------------------------------------]  1/247    0%

I turned on WAL mode first so that I could run Datasette against it while it processes:
sqlite3 sf-trees-changed.db vacuum
sqlite-utils enable-wal sf-trees-changed.db

It's still on the first commit and Datasette already shows this:

That's because this CSV file has ~190,000 rows in it, so just the first commit will create 190,000 item versions and 190,000 * 18 = 3,420,000 rows in the item_changed table.
Be interesting to see what the remaining 248 commits do to the database size.

I restarted it to try out an optimization, but then realized there's a nasty bug: the _version resets to 1 if you restart it even if there are already commits in the database.

This option - which reduces the number of times we call db["column"].lookup() by a huge amount - seems to have made a solid difference to performance.
diff --git a/git_history/cli.py b/git_history/cli.py
index 9be75c3..b294623 100644
--- a/git_history/cli.py
+++ b/git_history/cli.py
@@ -170,6 +170,17 @@ def file(
     item_id_to_last_full_hash = {}
     item_id_to_previous_version = {}

+    column_name_to_id = {}
+
+    def column_id(column):
+        if column not in column_name_to_id:
+            id = db["columns"].lookup(
+                {"namespace": namespace_id, "name": column},
+                foreign_keys=(("namespace", "namespaces", "id"),),
+            )
+            column_name_to_id[column] = id
+        return column_name_to_id[column]
+
     for git_commit_at, git_hash, content in iterate_file_versions(
         resolved_repo,
         resolved_filepath,
@@ -325,24 +336,21 @@ def file(

                     if changed_columns:
                         # Record which columns changed in the changed m2m table
-                        for column in changed_columns:
-                            db[changed_table].insert(
+                        db[changed_table].insert_all(
+                            (
                                 {
                                     "item_version": item_version_id,
-                                    "column": db["columns"].lookup(
-                                        {"namespace": namespace_id, "name": column},
-                                        foreign_keys=(
-                                            ("namespace", "namespaces", "id"),
-                                        ),
-                                    ),
-                                },
-                                pk=("item_version", "column"),
-                                foreign_keys=(
-                                    ("item_version", version_table, "_id"),
-                                    ("column", "columns", "id"),
-                                    ("namespace", "namespaces", "id"),
-                                ),
-                            )
+                                    "column": column_id(column),
+                                }
+                                for column in changed_columns
+                            ),
+                            pk=("item_version", "column"),
+                            foreign_keys=(
+                                ("item_version", version_table, "_id"),
+                                ("column", "columns", "id"),
+                                ("namespace", "namespaces", "id"),
+                            ),
+                        )

         else:
             # no --id - so just correct for reserved columns and add item["_commit"]

Now at 52 minutes estimated remaining:
(git-history) git-history % time git-history file sf-trees-changed.db ../sf-tree-history/Street_Tree_List.csv --repo ../sf-tree-history --changed --id TreeID --csv
  [####--------------------------------]  34/247   13%  00:52:39

It broke!
  [#####-------------------------------]  40/247   16%  00:57:55Error: Commit: 3fb63a99dfab8a75c83d341c67afc9abf484e0c4 - every item must have the --id keys. These items did not:
[
    {
        "1": "9",
        "DPW Maintained": "DPW Maintained",
        "Pinus radiata :: Monterey Pine": "Palm (unknown Genus) :: Palm Spp",
        "10 10th Ave": "97 12th St",
        "Median : Cutout": "Median : Cutout",
        "Tree": "Tree",
        "DPW": "DPW",
        "": "",
        "3": "20",
        "16": "3X3",
        "5992719.40718": "6007166.5822125",
        "2114869.71245": "2109668.7679164",
        "37.7866097734767": "37.7731533608049",
        "-122.468982710496": "-122.418629924651",
        "(37.7866097734767, -122.468982710496)": "(37.7731533608049, -122.418629924651)",
        "11": "19",
        "6": "2",
        "54": "28853"
    },
...

https://github.com/simonw/sf-tree-history/commit/3fb63a99dfab8a75c83d341c67afc9abf484e0c4 is the commit in question - it says "sort CSV before storing" - so maybe I need to only process commits AFTER that point?

Documentation: https://github.com/simonw/git-history/blob/416566bf868aba2033db19fe09631d8162d4bd9b/README.md#track-the-history-of-individual-items-using-ids