Please Make This Usable In Windows, Please?

Using Cygwin in Windows to try go through the steps to install.  On attempt 30 or so I think.
Relatively new to the frameworks and packages used in its development, so this is causing headaches.
Finally found out how to get node running with Windows+Cygwin, then tried the run.sh, which tells me to run ./install-multirust.sh
First attempt:
Submodule path 'test/multirust-v1': checked out '2eaf9067314307320d27be510e1ac3949a21eee9'
blastoff: installing
./install-multirust.sh: line 121: sudo: command not found
blastoff: failed to install multirust
Alright fine, open the script and get rid of all the sudo and try again:
Submodule path 'test/multirust-v1': checked out '2eaf9067314307320d27be510e1ac3949a21eee9'
blastoff: installing
install: uninstalling component 'multirust'
install: creating uninstall script at /usr/local/lib/rustlib/uninstall.sh
install: installing component 'multirust'
build/work/multirust-0.7.0/install.sh: line 384: uname_value: unbound variable
blastoff: failed to install multirust
Bah!
I like the readme instructions:
Eve relies on TypeScript, Rust Nightly, and multirust. You'll want these installed, though our run.sh will attempt to install them for you.
bash run.sh
we're working on a nice experience for windows, but we suggest trying to use cygwin and doing the above for now
I am using cygwin and doing the above (probably wrong) and getting nowhere.
The last run.sh attempt:
$ ./run.sh

Installing commit hooks...done.
Checking dependencies...module.js:340
throw err;
^

Error: Cannot find module 'C:\home\User\Projects\Eve\ui\node_modules\typescript\bin\tsc'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:289:25)
at Function.Module.runMain (module.js:467:10)
at startup (node.js:136:18)
at node.js:963:3
x Eve requires tsc version "1.6.2" but "" is installed. Please reinstall using:
cd ui && npm install && cd .. before continuing.
Can you add Cygwin instructions for all of us (or maybe just me?) that can't piece together the install?  I can't see what I am missing here.

Don't worry, it's not just you; when we released Eve Version 0, Rust support on Windows was very tenuous, and we honestly didn't test on Windows at all. Today Rust support for Windows is much better, but the Version 0 graph editor is still stuck at Rust 1.3. If you'd like to try, you can upgrade to Rust 1.5 and try to resolve any issues that comes along with that. In fact, you might not need to use Cygwin at all, as rust 1.5 links against native windows libraries (I don't know how far you'll get with the packages though). Then you can use node natively in windows to compile the UI.
One thing I'd like to point out is that multirust isn't a hard dependency; it was just a convenient way to switch between Rust versions.
Ultimately, our work on the graph editor released with Version 0 has concluded; We're focusing our development on experimental\cardwiki, which you'll be happy to know as of yesterday has a build script that works in Windows! Give it a try and let us know how it works out. We'll be coming out with more docs and better instructions for the next release, so sit tight!

@Dro2000 in case you want to check the development on experimental\cardwiki here's my hacky way to get it running

download zip

extract only the experimental\cardwiki folder

edit run.sh

--- D:\Desktop\wikidata\cardwiki\run.sh	Thu Jan 14 11:09:12 2016
+++ D:\Desktop\wikidata\cardwiki\run.sh	Thu Jan 14 21:32:26 2016
@@ -88,1 +88,1 @@
-node_modules/tsify/node_modules/typescript/bin/tsc --watch -m commonjs 2>&1 | tag "typescript" "$blue" &
+node_modules/.bin/tsc --watch -m commonjs 2>&1 | tag "typescript" "$blue" &
4. `bash run.sh` from `cmd` or maybe just `./run.sh` on cygwin (i use MSYS2)
5. after it bundle's `runtimeTest` kill it
6. move content from `bin/src/` to `bin/`
7. `node bin/server.js`

Our new node-based run script just landed, which adds experimental support
for windows. Can you try running
node run.js -v --width 80 and then visit the editor at
<path\to\eve>\experimental\cardwiki.editor.html and confirm if it works
for you? Bundling may take a while, so keep an eye on the status indicators
and wait til they turn green. You don't need the server to be running
(that's another experiment for multi-user, which isn't quite ready for
primetime, though should still work).
Cheers,
Josh
On Thu, Jan 14, 2016 at 12:38 PM, Valerij Primachenko <
<EMAIL_ADDRESS>wrote:

@Dro2000 https://github.com/Dro2000 in case you want to check the
development on experimental\cardwiki here's my hacky way to get it running

download zip
extract only the experimental\cardwiki folder

edit run.sh
--- D:\Desktop\wikidata\cardwiki\run.sh Thu Jan 14 11:09:12 2016+++ D:\Desktop\wikidata\cardwiki\run.sh Thu Jan 14 21:32:26 2016@@ -88,1 +88,1 @@-node_modules/tsify/node_modules/typescript/bin/tsc --watch -m commonjs 2>&1 | tag "typescript" "$blue" &+node_modules/.bin/tsc --watch -m commonjs 2>&1 | tag "typescript" "$blue" &

bash run.sh from cmd or maybe just ./run.sh on cygwin (i use MSYS2)
after it bundle's runtimeTest kill it
move content from bin/src/ to bin/
node bin/server.js

—
Reply to this email directly or view it on GitHub
https://github.com/witheve/Eve/issues/333#issuecomment-171772755.

the instruction i posted were intended to be used with last properly working ui version (or pref d26af08a via Eve talk), which was before run.js landed. Should have wrote that i guess :)
as to run.js
i need to install livereload manually even after npm i
also i need to create the bin folder myself
then the script succeeds and i can use the editor.html

Thanks for the information!  I still cannot get this to work due to my inexperience with Linux and the frameworks used.  I have a windows box and a remote linux box connection and have tried a few times in both messing around but there are always errors or missing packages or something doesn't install or work correctly.
I am sure I could get it working if I was super dedicated on a mission to try this out but as it is still beta and full of warnings to not use it for anything production so I am not that dedicated..... yet.
I will be watching this project closely though and very much look forward to a "click-and-run" demo when/if it becomes available.
So far from the few bits I have seen this looks like an excellent concept, I just hope it comes to be at some point.
As a tongue in cheek side rant prompted by this experience:
I hear Windows getting bashed all the time since it is not as "cool" as Linux but I find I can focus on the work I want to do.  As soon as you hit Linux you need to become a IT/OS/framework/programming/package/vim/vi/bash/script/makefile/etc/etc professional to even just get something to run.  I have tested many a Windows demo (and C/C++/C# git clone VS projects) with typically nothing more than a double-click or a build button but wow this Linux stuff is crazy.  I know I can be faulted for resisting Linux (as it is the flavor of the decade now) but I do not have as much interest in becoming a command-line OS and package configuration god as I do with actually using software to accomplish something.  From my brief time so far in Linux I find you spend ALOT of time becoming the former rather then achieving the latter.
  And its great because just like a good actor you can find some good commands with a healthy amount of scrolling text to really drive home the "this is just like a movie/tv show" effect.  (Or break out VIM and do some editing in a source file mid-stream, unbelievable!)  I am guessing this is part of the attraction to Linux.  Windows is so boring with things just mostly working out of the box, with intuitive visual menus and buttons.  (I know I know, there are exceptions and I have had to go through some painful troubleshooting but at least you could click-search your way to a solution, a blank flashing prompt requires intimate knowledge of the internal workings to even start to figure anything out)
It reminds me of when I was looking at a bunch of LISP code and thought it was a ridiculous language due to horrible readability to people not familiar with it, I was told that its probably because LISP coders enjoy the puzzle of the language itself, not just the puzzle of coding a good solution to accomplish something.  Seems similar to Linux, people must enjoy the puzzle of the OS itself and its not enough to just run an application, you really need a challenge there to feel acceptably accomplished.  (Nothing is as good as when you have to really work hard for it, then it becomes much more satisfying on a psychological level)
And you can really draw this out to a more general level.  You can become like one of those telephone psychics and really surprise people.  You notice someone going to fill up with gas at a pump but they back into the pump stall with the gas cap facing the wrong side so they have to drag and weave the hose in a complex way to avoid scratching the paint!?  You are pretty safe to assume that once approaching them and they say they program for a living that they are probably involved with Linux and a language similar to LISP.  They crave a challenge for the task at hand, no matter how mundane.
And vice versa, if someone works with Linux and you have to do a real-world task strap yourself in because you are not taking the easy direct way to achieve it.

i went so far and packaged two runnable versions of Eve for you @Dro2000
recent (a3690de9 based) version here
and older (67a0d14e based) version here which i prefer myself
sometimes Eve will fail to load, the only way i found to fix it is to clean the storage by running
delete localStorage["local-eve"]

in browsers console
:exclamation: this will delete all the data you have saved in Eve

Wow thanks!  This is awesome!  Thank you for doing that, I have tried both versions @vprimachenko  has uploaded in Firefox and they do not render correctly.  There is a couple of boxes I can type text into but no save, and if I close that box there is nothing.  Only a few numbers about the version in the upper right hand corner of the page.
The newer version has a title bar with a magnifying glass search but does not do anything when I click on it.  I see #pet and Pet is test data but nothing else.
I am thinking I am really going to need to get a Linux box up or a VM going to try the install steps again.  It does not look like there is a way for me to get this working in Windows/Cygwin with my level of knowledge.

Alright I tried downloading and running again and get the following with the latest experimental card-wiki (in Cygwin):
$ ./run.sh
Updating node_modules...
npm WARN package.json<EMAIL_ADDRESS>No README data
Symlinking vendor for node execution...
Starting watchers...
17:16:17 [typescript] src/server.ts(4,27): error TS2307: Cannot find module 'compression'.
17:16:18 [typescript] 5:16:18 PM - Compilation complete. Watching for file changes.
17:16:18 [richTextEditorTest] TypeScript error: src/richTextEditor.ts(7,1): Error TS2304: Cannot find name 'require'.
17:16:18 [richTextEditorTest] TypeScript error: src/richTextEditor.ts(8,1): Error TS2304: Cannot find name 'require'.
17:16:18 [richTextEditorTest] TypeScript error: src/richTextEditor.ts(95,7): Error TS2345: Argument of type '{ mode: string; lineWrapping: boolean; autoCloseBrackets: boolean; viewportMargin: number; extraK...' is not assignable to parameter of type 'EditorConfiguration'.
17:16:18 [richTextEditorTest] Object literal may only specify known properties, and 'autoCloseBrackets' does not exist in type 'EditorConfiguration'.
17:16:18 [editor] TypeScript error: src/richTextEditor.ts(7,1): Error TS2304: Cannot find name 'require'.
17:16:18 [runtimeTest] TypeScript error: src/richTextEditor.ts(7,1): Error TS2304: Cannot find name 'require'.
17:16:18 [richTextEditorTest] TypeScript error: src/utils.ts(7,21): Error TS2304: Cannot find name 'exports'.
17:16:19 [runtimeTest] TypeScript error: src/richTextEditor.ts(8,1): Error TS2304: Cannot find name 'require'.
17:16:19 [editor] TypeScript error: src/richTextEditor.ts(8,1): Error TS2304: Cannot find name 'require'.
17:16:19 [runtimeTest] TypeScript error: src/richTextEditor.ts(95,7): Error TS2345: Argument of type '{ mode: string; lineWrapping: boolean; autoCloseBrackets: boolean; viewportMargin: number; extraK...' is not assignable to parameter of type 'EditorConfiguration'.
17:16:19 [editor] TypeScript error: src/richTextEditor.ts(95,7): Error TS2345: Argument of type '{ mode: string; lineWrapping: boolean; autoCloseBrackets: boolean; viewportMargin: number; extraK...' is not assignable to parameter of type 'EditorConfiguration'.
17:16:19 [queryParserTest] TypeScript error: src/richTextEditor.ts(7,1): Error TS2304: Cannot find name 'require'.
17:16:19 [runtimeTest] Object literal may only specify known properties, and 'autoCloseBrackets' does not exist in type 'EditorConfiguration'.
17:16:19 [queryParserTest] TypeScript error: src/richTextEditor.ts(8,1): Error TS2304: Cannot find name 'require'.
17:16:19 [runtimeTest] TypeScript error: src/utils.ts(7,21): Error TS2304: Cannot find name 'exports'.
17:16:19 [editor] Object literal may only specify known properties, and 'autoCloseBrackets' does not exist in type 'EditorConfiguration'.
17:16:19 [queryParserTest] TypeScript error: src/richTextEditor.ts(95,7): Error TS2345: Argument of type '{ mode: string; lineWrapping: boolean; autoCloseBrackets: boolean; viewportMargin: number; extraK...' is not assignable to parameter of type 'EditorConfiguration'.
17:16:19 [editor] TypeScript error: src/utils.ts(7,21): Error TS2304: Cannot find name 'exports'.
17:16:19 [queryParserTest] Object literal may only specify known properties, and 'autoCloseBrackets' does not exist in type 'EditorConfiguration'.
17:16:19 [queryParserTest] TypeScript error: src/utils.ts(7,21): Error TS2304: Cannot find name 'exports'.

try running npm install first

Ah looks like there was something else wrong that Corey just fixed. Try pulling and then run:
npm install
npm run dev

and then go to http://localhost:3000/

I get:
"KILLING PROCS" in the Cygwin terminal when I try this.  Moving the mouse around in the cygwin terminal causes random input to scroll across as well.  Always 3 characters at a time and the rate of input is proportional to the mouse movement.  Maybe this is a feature of some module, not sure.
"Unable to connect"
when navigating to the localhost:3000
There were a bunch of errors during the npm install, the number of lines appearing to be related to the errors was > 50 so will have to investigate later
This time when npm run dev the windows firewall exception request dialog popped up which is new and exiciting.  progress!

Eve will work in Windows going forward. At the very least, we will have a Docker container, but I develop in Windows full time, so I am invested in making sure it works here.

I also request something that works in Windows.  Please make something that runs like software (from the installs I have tried it is a complex convoluted process and I have not been able to use the software as of yet….)
From: Corey Montella<EMAIL_ADDRESS>Sent: July 6, 2016 12:14 PM
To: witheve/Eve
Cc: Darren Rauckman; Mention
Subject: Re: [witheve/Eve] Please Make This Usable In Windows, Please? (#333)
-- Sent by an external sender --
Eve will work in Windows going forward. At the very least, we will have a Docker container, but I develop in Windows full time, so I am invested in making sure it works here.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/witheve/Eve/issues/333#issuecomment-230857966, or mute the threadhttps://github.com/notifications/unsubscribe/AP6a50sdOAeMPam74dWmLmcR8jGgwFm4ks5qS_BQgaJpZM4HEBSD.

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.

Dro2000,
As a Windows user myself, I feel your pain. Do you use Docker at all? I myself have not until now, so I'm curious how Windows users perceive it. I've found that for the user side, the experience was pretty seamless on Windows.


nominatim: Add pyosmium to the dependencies

This package is required to perform updates of nominatim (using replication --once)
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 283861 run on x86_64-linux 1

1 package built:

nominatim


BREAKING CHANGE: Switch Toast component to v3 design tokens #DS-1446

Description

Additional context

Issue reference

coverage: 78.763% (+0.07%) from 78.69%
when pulling 7392dad6e0c90eee60f7e2bab41e80248478d7e2 on feat/v3-toast
into d2d3f545dc14ae213a4336206301e558367268b1 on integration/BC-v3-design-tokens.


fix: remove accidental quotes in filename

Relates to: #147

:tada: This PR is included in version 1.13.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add table gcp_vertex_ai_endpoint

Integration test logs

Logs
Add passing integration test logs here

Example query results

Results
Add example SQL query results here (please include the input queries as well)

Will be addressed in https://github.com/turbot/steampipe-plugin-gcp/pull/513


LintDiff support request by Tarun Sharma: incorrect RequiredPropertiesMissingInResourceModel

Please see here

Link to the pipeline where error is seen - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2604647&view=ms.vss-test-web.build-test-results-tab

@raych1 confirmed in the Teams thread that LintDiff appears to report a false positive.

Some details of the bogus error:
<html>
<body>
<!--StartFragment-->

Rule
--
❌ RequiredPropertiesMissingInResourceModel

<!--EndFragment-->
</body>
</html>Rule
❌ [RequiredPropertiesMissingInResourceModel](https://github.com/Azure/azure-openapi-validator/blob/main/docs/required-properties-missing-in-resource-model.md)
Messges
Model definition 'HyperVSite' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L12646)
Model definition 'ImportSiteResource' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L13613)
Model definition 'MasterSite' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L13883)
Model definition 'ServerSiteResource' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L15271)
Model definition 'V20200201DeleteImportMachinesJob' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L17321)
Model definition 'VMwareSite' must have the properties 'name', 'id' and 'type' in its hierarchy and these properties must be marked as readonly.
Location: [Microsoft.OffAzure/stable/2022-10-27/migrate.json](https://github.com/Azure/azure-rest-api-specs/blob/81aece4a388def0c7cec404a915e0879b78df64f/specification/migrate/resource-manager/Microsoft.OffAzure/stable/2022-10-27/migrate.json#L17679)

Another potential occurrence of this problem:

https://github.com/Azure/azure-rest-api-specs/pull/24086#issuecomment-1560515497
CI check for the above
Build log for the above


The problems in streaming decode are in pruned_ transducer_ stateless5

I used my own data to train a streaming model. The recognition effect is poor when decoding.
There are two obvious problems, one is to delete words at the end, and the other is to insert multiple words in the middle.
training：

decoding：

result：

What should I do to reduce these mistakes！I look forward to your reply. Thank you！！！

one is to delete words at the end,

Could you add some tail paddings to your utterances and retry?

the other is to insert multiple words in the middle

This problem is much harder to fix in your case since the ground truth contains 3 contiguous 零.
Does your training data contain data patterns like this?

should be HYP not HYF.
Yes, this is  the kind of case where E2E models have trouble.

the other is to insert multiple words in the middle

This problem is much harder to fix in your case since the ground truth contains 3 contiguous 零. Does your training data contain data patterns like this?

When I set avg to 1, the insertion error decreases a lot

the other is to insert multiple words in the middle

This problem is much harder to fix in your case since the ground truth contains 3 contiguous 零. Does your training data contain data patterns like this?

When I set avg to 1, the insertion error decreases a lot

What is your original setting?

one is to delete words at the end,

Could you add some tail paddings to your utterances and retry?

I added 1s silence after the testset, and almost didn't delete at the end of the sentence.
Another question is, what is the possible reason for many deletions in the middle of sentences?

What is your original setting?

The above reply contains the original configuration for decoding. It was written incorrectly. The avg used to be 2, but now it is 1

one is to delete words at the end,

Could you add some tail paddings to your utterances and retry?

I added 1s silence after the testset, and almost didn't delete at the end of the sentence.
Another question is, what is the possible reason for many deletions in the middle of sentences?

Have you listened to the audios of theses two utterances? Do they look normal?


Prepare release for Azure.ResourceManager.Grafana 1.0.0-beta.1

API change check for Azure.ResourceManager.Dashboard
API changes are not detected in this pull request for Azure.ResourceManager.Dashboard

/azp run prepare-pipelines

/azp run net - grafana - ci

API change check for Azure.ResourceManager.Grafana
API review is created for Azure.ResourceManager.Grafana. You can review APIs here.


ReferenceSet name not set

If we run
$ python client_dev.py referencesets-search http://localhost:8000
R1JDaDM4LXN1YnNldA==    None

This output shows the ID and name of an object. We expect the name of a reference set to be non-null, so this is a bug.
Edit ga4gh/datamodel/references.py. Find the AbstractReferenceSet.toProtocolElement method and insert the line
ret.name = self.getLocalId()

After saving the file and reloading the server, we should now see
$ python client_dev.py referencesets-search http://localhost:8000
R1JDaDM4LXN1YnNldA==    GRCh38-subset

Bug fixed!
To make sure the bug stays fixed, we must add some test cases.
Open tests/unit/test_simulated_stack.py and update the verifyReferenceSetsEqual method. Verify the tests work by running nosetests -vs tests/unit/test_simulated_stack.py. (To make double sure that the tests  work correctly, comment out the line you added above, and check that the test now fails.)

PR incoming


Rename initate to initiate

Fix https://github.com/ilyabirman/Likely/issues/7

Thanks :)

Yay, first breaking change! looking forward for likely 3.0

I didn't actually mentioned that likeky.initate or likely.initiate was part of public API. As I understand, Birman's initial intention of Likely was "simply install and forget".
Even 0.X Likely didn't had any public API (because to initiate dynamically likely widget, you had to invoke socialLikes method on jQuery element * gasp *). I'm planning onto writing documentation and making Likely even more modular (hello from OOP), which will lead to... Likely 3.0 (probably)

Birman's initial intention of Likely was "simply install and forget".

yeah, it was okay in the pre-npm era, but know we are using it in scripts directly. Looking forward for more modular likely! Great job 👍

yeah, it was okay in the pre-npm era, but know we are using it in scripts directly

That's sooo embarrassing (I'm about initate), lol

Great job 👍

Thanks :)

I decided to bring back the initate name. Yeah, it's misspelling of initiate, but there's couple of reasons why I did it:

To avoid major release (Likely is only 2.0.4, there's more to come)
To teach myself a lesson

That's pretty much it. @gangstaJS sorry for bothering you :smile:

huh, you still can rename initate to initiate, but keep initate for backwards compatibility until next major release. i think it will be better

I didn't thought of that. That's genius! Thanks for the suggestion! :smile:

@volter9 don't worry bro. It would be great to make from Likely react component.


Docker fix

Describe your changes :

Worked on improving and optimizing the local docker

Type of change :

[x] Improvement

Frontend Preview (Screenshots) :
For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have performed a self-review of my own.
[x] I have tagged my reviewers below.
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] My changes generate no new warnings.
[ ] I have added tests that prove my fix is effective or that my feature works.
[x] All new and existing tests passed.

Reviewers

@harshach @shahsank3t

Reviewed and Tested. Waiting for additional documentation for Airflow UI credentials.


Multi-view ReFit

Thank you for the excellent job you have done. Can you provide more code details about Multi-view ReFit, thank you very much.

Thanks for reaching out! Please take a look at this script as well as the model it uses:
https://github.com/yufu-wang/ReFit/blob/main/scripts/eval_mv.py
We cannot release the annotation for h36m at this point, but the multiview fitting code should work when the input's batch dimention is the number of views. We are working to release more details about this. Thank you for your patient!

Thanks for reaching out! Please take a look at this script as well as the model it uses: https://github.com/yufu-wang/ReFit/blob/main/scripts/eval_mv.py We cannot release the annotation for h36m at this point, but the multiview fitting code should work when the input's batch dimention is the number of views. We are working to release more details about this. Thank you for your patient!

Thanks for your answer. I will attempt the solution you provided.


Docs: inherit_mode

[x] Wrote good commit messages.
[N/A] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[N/A] Added tests.
[N/A] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[N/A] The PR relates to only one subject with a clear title
and description in grammatically correct, complete sentences.
[x] Run rake default or rake parallel. It executes all tests and RuboCop for itself, and generates the documentation.

Requested changes made, thanks


paper request

Hi，
would you like send this paper <RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search>to me?

Paper will be soon released as the camera-ready version.


refacator discrete tokenizer

What does this PR do?

Move all discrete tokenizers to discrete_tokenizer folder
update LJSpeech/quantization with new quantization HF repo and merge all hubert,wavlm, and wab2bev2 into one file
update discrete_ssl with new quantization HF repo

Before submitting

[ ] Did you read the contributor guideline?
[ ] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ ] Did you make sure to update the documentation with your changes? (if necessary)
[ ] Did you write any new necessary tests? (not for typos and docs)
[ ] Did you verify new and existing tests pass locally with your changes?
[ ] Did you list all the breaking changes introduced by this pull request?
[ ] Does your code adhere to project-specific code style and conventions?

PR review

Reviewer checklist

[ ] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[ ] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified
[ ] Confirm that the changes adhere to compatibility requirements (e.g., Python version, platform)
[ ] Review the self-review checklist to ensure the code is ready for review

@TParcollet  this is the PR for the discrete tokenizer... even the other models are dependant on other libraries..so they are not exactly SB homemade model

Thank you @poonehmousavi


Clamd Permissions Fix for Debian AppArmor

🗣 Description
Resolve AppArmor configuration issue which prevents clamd and freshclam from running.
💭 Motivation and context
On Debian GNU/Linux 11 (bullseye), Clamd and Freshclam are not configured correctly by default with latest versions.
Apparmor Version: 2.13.6
ClamAV Version: 0.103.10
Error: Can't open /var/log/clamav/freshclam.log in append mode (check permissions!).
Defect Notes: https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1842695
🧪 Testing
Change was incorporated into internal Playbook
✅ Pre-approval checklist

[X] This PR has an informative and human-readable title.
[X] Changes are limited to a single goal - eschew scope creep!
[X] All relevant type-of-change labels have been added.
[X] I have read the CONTRIBUTING document.
[X] These code changes follow cisagov code standards.
[X] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[ ] Tests have been added and/or modified to cover the changes in this PR.
[ ] All new and existing tests pass.

@krimsonkla - In order to test these changes you'll probably need to create a separate Molecule scenario where apparmor.service is started.  As an example, cisagov/ansible-role-clamav#53 is a previous PR where a new Molecule scenario was added.

@jsf9k I'll try to find some time to work out the molecule testing. Having never used molecule, I started down that path last evening but struggled to get molecule to work in my nix-shell.

@krimsonkla - If you're not accepting someone's changes, please let the reporter resolve the comments.  They are the person in the best position to determine whether their concerns have been addressed.


alertmanager with pushbullet

Hi,
I'm looking for a way for alertmanager to push alert using pushbullet (https://www.pushbullet.com/) do you know how is it possible ?
thx for your help

@gfediere I am not aware of any. Maybe @stuartnelson3 knows more.
If there is none I would suggest you use the Alertmanager webhook configuration and write a small proxy.
In general, please open up usage questions on the Prometheus users Google group

Hi @mxinden
thx for your return


Update started.md

Update import lines
The import lines from the doc were returning an error, I just took the (working) ones from the README.md to put them back here

According to https://github.com/vibora-io/vibora/blob/master/vibora/init.py you can import responses from the vibora module.


@cycle/http 14.5.0 isolation issue (a regression from 14.2.0)

Encountered an issue when upgrading @cycle/http to 14.5.0. The HTTP responses don't make it to a response stream. This turned out to be isolation issue appeared after this commit: https://github.com/cyclejs/cyclejs/commit/e0dea73
Code to reproduce the issue:
https://jsbin.com/hinifocufu/edit?html,js,output
Expected behavior:
HTTP responses should work just fine through multiple levels of isolateion
Actual behavior:
HTTP response stream never emits when isolated more than 1 once with different isolation scopes.
Versions of packages used:

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Some insights:
The problem is that arrayEqual assumes the 2 arrays of namespaces (the one in request and that in isolated source) are exactly the same - i.e., absolutely identical, with elements going in exactly the same order, while this request._namespaces.push(...) here actually makes the array reverse (perhaps, because it happens in isolateSink and sinks are "outputs" and are collected on the way back, thus resulting in the reverse order of the request namespaces).
Looks like it should either:

be a req._namespaces.unshift(...) here; OR
request.namespace should be passed in a reverse order here

The issue obviously doesn't appear when there's no isolate or when there's only one isolate(...) call (because in that case there's just one element in the _namespaces array).
Hopefully, I'm not missing something important, guys!

Looks like a legit bug. Thanks for the detailed report!

@staltz maybe replace _namespaces Array with a Map (or better with simple object) and then assign props to it instead of push, checking presence also will be quicker than indexOf and other array methods.

@whitecolor, the order of namespaces is important. I introduced this bug with the fix of a bug that was exactly this: Wrong isolation due to ignoring of order

the order of namespaces is important.

Why? I don't get it.

the order of namespaces is important.

I see that issue https://github.com/cyclejs/cyclejs/issues/685 about the same scope value passed to isolate. Why not just make it unique (by adding to the string something)? And get rid of a little bit complicated array-based logic. Just as a variant.

Adding to the string would have to be random, and this random can be the same. Thus it doesnt fix the bug, it just makes it very unlikely. I'd rather do a proper fix and avoid nondeterminism

Adding to the string would have to be random, and this random can be the same.

No, you may just check if the key already exists in _namespaces object, you prefix it with just say _, no need to use anything random. or you may add innter isolate counter value, and increment it as it is done for implicit scope.

Still not a nice solution. I would avoid Math.random(). The data structure that suits the best here is a stack. You push the inner namespaces on top, and pop on the exit

@jvanbruegge
No need to use random, though you probably right about a stack here.


The emoji "flag_gb" should be renamed to "flag_uk".

The emoji flag_gb contains the flag of Northern Ireland, which is not in Great Britain.
The flag this emoji has should rather be named to "flag_uk" which is the correct name for it.

http://unicode.org/Public/emoji/13.0/emoji-test.txt
1F1EC 1F1E7              ; fully-qualified     # 🇬🇧 E0.6 flag: United Kingdom

1F3F4 E0067 E0062 E0065 E006E E0067 E007F  ; fully-qualified     # 🏴 E5.0 flag: England 1F3F4 E0067 E0062 E0073 E0063 E0074 E007F  ; fully-qualified     # 🏴 E5.0 flag: Scotland 1F3F4 E0067 E0062 E0077 E006C E0073 E007F  ; fully-qualified     # 🏴 E5.0 flag: Wales


Parsing issues

author, as well type are frequently not parsed, even with very common news sites. One example is the verge: https://www.theverge.com/2017/12/2/16727166/apple-ios-11-2-features-release

thanks for letting me know! just pushed out a release that fixes this.

Thanks! May I ask what exactly is determining the type? In the link above, I'd consider it to be an article, but slurp categorizes it as post.

all of the metadata is self reported by the website, slurp just collects it

I see. Continuing with a different feature request then, since this is solved


Terminal API should throw an event when a terminal has completed (and is waiting to close)

From @dbaeumer:

an event when a task executed in a terminal has finished (e.g. the terminal process existed)

What I actually want to do is the following:
this.terminalService.createInstance(`Task: ${task.name}`, undefined, ['/C', 'dir'], false);

dir is an example here and '/C' is currently specific to cmd.exe.
What I would like to see happening here is that the cmd.exe is started, dir is executed and then cmd.exe terminates. When cmd.exe terminates I would like to get an event.
Regarding the '/C' option: any change to have this abstracted in the terminal itself. Having that knowledge on the client side is not very nice.

What currently happens is the following:

the terminal is started
dir is not executed
the terminal stays alive

I changed it to
this.terminalService.createInstance(Task: ${task.name}, 'C:\\WINDOWS\\system32\\cmd.exe', ['/C', this.configuration.command], false); after some debugging but now no terminal shows up

Did you call Terminal.show() immediately after? API terminals are launched in the background. You could try point to C:\\Windows\\sysnative\\cmd.exe and/or use an environment var to find C:\\Windows. Sysnative is a weird symlink thing that points to system32 or syswow64. Try that first and maybe rm -rf node_modules/node-pty node_modules/pty.js and try reinstall deps?
It would be good to encapsulate the bash and cmd command execution commands inside the TerminalService, however I'm not sure something like that should be exposed in the Terminal API. In general I don't want to encourage using a custom shell/args unless the specific use case calls for it (like tasks).

I got it working passing in true to waitOnExit. My code looks like this now:
let terminal = this.terminalService.createInstance(`Task: ${task.name}`, 'C:\\WINDOWS\\system32\\cmd.exe', ['/C', this.configuration.command], true);
this.terminalService.setActiveInstance(terminal);
this.terminalService.showPanel(false);

What I actually would like to have is a listener instead of the waitOnExit. But we already discussed this.

Regarding:

It would be good to encapsulate the bash and cmd command execution commands inside the TerminalService, however I'm not sure something like that should be exposed in the Terminal API. In general I don't want to encourage using a custom shell/args unless the specific use case calls for it (like tasks).

As said we keep the task framework in the core for now. So it would allow us to experiment with some API here before thinking about making it official.

@dbaeumer I guess I'll expose some event on ITerminalInstance exit for now which when used with waitOnExit to allow reuse of the terminal (via another ITerminalInstance call to start up a new process while retaining the old output).

I closed this as it's possible internally now via ITerminalInstance.onExit


Error with Pyspark GBTClassifier

@QuentinAmbard
I just installed Shap from PyPi (0.32.0) and running a version of your test still produces the same error - shown below. Is there something that I am missing in the use of Shap with a pyspark model?
import pyspark
print(pyspark.__version__)
import shap
print(shap.__version__)
import sklearn.datasets
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.ml.feature import VectorAssembler, StringIndexer
from pyspark.ml.classification import RandomForestClassifier, DecisionTreeClassifier, GBTClassifier
import pandas as pd

iris_sk = sklearn.datasets.load_iris()
iris = pd.DataFrame(data= np.c_[iris_sk['data'], iris_sk['target']], columns= iris_sk['feature_names'] + ['target'])[:100]
spark = SparkSession.builder.config(conf=SparkConf().set("spark.master", "local[*]")).getOrCreate()

col = ["sepal_length","sepal_width","petal_length","petal_width","type"]
iris = spark.createDataFrame(iris, col)
iris = VectorAssembler(inputCols=col[:-1],outputCol="features").transform(iris)
iris = StringIndexer(inputCol="type", outputCol="label").fit(iris).transform(iris)

classifier = GBTClassifier(labelCol="label", featuresCol="features")
model = classifier.fit(iris)
explainer = shap.TreeExplainer(model)
X = pd.DataFrame(data=iris_sk.data, columns=iris_sk.feature_names)[:100] # pylint: disable=E1101
shap_values = explainer.shap_values(X)

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-31-f47b3a56c25f> in <module>
     23 explainer = shap.TreeExplainer(model)
     24 X = pd.DataFrame(data=iris_sk.data, columns=iris_sk.feature_names)[:100] # pylint: disable=E1101
---> 25 shap_values = explainer.shap_values(X)

/mnt1/anaconda3/lib/python3.7/site-packages/shap/explainers/tree.py in shap_values(self, X, y, tree_limit, approximate, check_additivity)
    283
    284         if check_additivity and self.model_output == "margin":
--> 285             self.assert_additivity(out, self.model.predict(X))
    286
    287         return out

/mnt1/anaconda3/lib/python3.7/site-packages/shap/explainers/tree.py in predict(self, X, y, output, tree_limit)
    785             import pyspark
    786             #TODO support predict for pyspark
--> 787             raise NotImplementedError("Predict with pyspark isn't implemented")
    788
    789         # see if we have a default tree_limit in place.

NotImplementedError: Predict with pyspark isn't implemented

@QuentinAmbard
Have anyone else ran this code successfully?

That's almost the code from the unit test, so yes it should run without error.
I'll try to debug that this week, maybe there is an issue with 0.32.0 ...

This is also happen to me. It used to work with RandomForest in version 0.30.2.
I'll try to revert and check again ..

Hello, I`m having the same problem with Shap version:  shap-0.32.1
I also tried previous version but I get the same error since the commit which added GBT and RF.
I made a test to check the unit test and copy all function: test_pyspark_classifier_decision_tree()
And run it in my environment and I get the same error.
Could be that the build was done without unit testing?

Hello, I`m having the same problem with Shap version: shap-0.32.1
I also tried previous version but I get the same error since the commit which added GBT and RF.
I made a test to check the unit test and copy all function: test_pyspark_classifier_decision_tree()
And run it in my environment and I get the same error.
Could be that the build was done without unit testing?

@caspiDoron You may try version 0.30.2. it works for me.

Hello, I`m having the same problem with Shap version: shap-0.32.1
I also tried previous version but I get the same error since the commit which added GBT and RF.
I made a test to check the unit test and copy all function: test_pyspark_classifier_decision_tree()
And run it in my environment and I get the same error.
Could be that the build was done without unit testing?

@caspiDoron You may try version 0.30.2. it works for me.

Thanks but it seems to work only for DT, Random forest failing on the tests: AssertionError: SHAP values don't sum to model output for class0!
GBT is not supported which is the one i use...

I just re-run the unit test and something is broken indeed.
As a workaround you can set check_additivity=False when computing the shap_values
It's a new check that has been added and calls the predict function.
I suspect this hasn't been catch in the unit tests because spark isn't in the env and the test is ignored in this case.

Thank you @QuentinAmbard it is working with this workaround.

Great!
I suggest we do the following:

Create a small fix to disable check_additivity for spark models (I'll commit that soon as a fix to this issue)
Make sure the tests are launched with the spark lib in the env to prevent from this kind of issues (will create a new issue to fix that)
More long term / viable: implement the prediction with spark (I'll create a new feature too)

Thanks for checking into this @QuentinAmbard! I just pushed an updated tolerance check for additivity to address #887, but I suspect this might be a true error that this new check uncovered. Happy to help work through it on the PR

Hi. I can run the example code properly.
But I'm not fully understand how shap_values works actually.
Can anyone please explain why shap_values takes 'X' as data in from of features in each column (i.e. sepal_length, sepal_width, petal_length, petal_width, separately in each column), while GBTClassifier model actually takes features in 1 column (named 'features').
Why shap_values can understand the difference between when the model was trained (features in 1 column) and when the model was to be explained?
Thank you sir.

shap_values takes a pandas Dataframe containing one column per feature.
GBTClassifier is a spark classifier taking a spark Dataframe to be trained. Spark works with 1 column containing an array with all the features you are using (that's what is doing the VectorAssembler)
Once the model is trained shap will explain it using shap_values(...).
You have to convert your data into a pandas dataframe to  explain it. If your dataset is too big you can easily create a spark Pandas UDF to run the shap_values in a distributed fashion.

shap_values takes a pandas Dataframe containing one column per feature.
GBTClassifier is a spark classifier taking a spark Dataframe to be trained. Spark works with 1 column containing an array with all the features you are using (that's what is doing the VectorAssembler)
Once the model is trained shap will explain it using shap_values(...).
You have to convert your data into a pandas dataframe to explain it. If your dataset is too big you can easily create a spark Pandas UDF to run the shap_values in a distributed fashion.

Thanks @QuentinAmbard. I still wonder how shap_values knows that each column in Pandas Dataframe equal to which element of Spark Dataframe (when the model was trained).

I'm using the index of the features, I assume the order of the pandas column must be the same as the features added in the vector assembler of your spark dataframe. Probably worth mentioning it in the documentation.
https://github.com/slundberg/shap/blob/master/shap/explainers/tree.py#L951

Hi All,
Is there any solution for NotImplementedError: CategoricalSplit are not yet implemented" in pyspark?

Using iris example i  try to parallelize shap values calculation in this way:
iris_shap=iris.drop('type','features','label').repartition(10)
X_columns=iris_shap.columns
explainer = shap.TreeExplainer(model)

def calculate_shap(rows,X_columns,explainer):
  a=pd.DataFrame(rows,columns=X_columns)
  shap_values = explainer.shap_values(a)
  return [Row(*( [float(f) for f in shap_values[i]])) for i in range(len(shap_values))]

iris_shap.rdd.mapPartitions(lambda j:calculate_shap(j,X_columns,explainer)).toDF(X_columns)

if model is sklearn.ensemble.GradientBoostingClassifier no problem but when is equal to pyspark.ml.classification.GBTClassifier obtain this error:
PicklingError: Could not serialize object: Py4JError: An error occurred while calling o135.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
	at py4j.Gateway.invoke(Gateway.java:274)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

Any suggestion?

The explainer can't be serialized probably because we are keeping spark references inside. I'll try to have a look.
As workaround you can recompute the explainer in each partition maybe ?

i try passing model to function calculate_shap and compute the explainer inside partition but obtain the same error

@QuentinAmbard
I am trying to get shap to work with a pyspark GBT classifier. I got my features as a numpy array X and then tried (as in the example):

model = pyspark.ml.classification.GBTClassificationModel.load("/path/to/trained/model")
explainer = shap.TreeExplainer(model)
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
sv = explainer.shap_values(X)

It gave the following error:

Traceback (most recent call last):
File "", line 1, in
File "/lib/python3.7/site-packages/shap/explainers/tree.py", line 304, in shap_values
assert self.model.fully_defined_weighting, "The background dataset you provided does not cover all the leaves in the model, "
AssertionError: The background dataset you provided does not cover all the leaves in the model, so TreeExplainer cannot run with the feature_perturbation="tree_path_dependent" option! Try providing a larger background dataset, or using feature_perturbation="interventional".

I did not provide a background dataset, so I don't understand why it wants me to provide a larger one.  Also, the matrix X contains my entire training dataset, so I don't understand how it could not cover all the the leaves in the model. Am I doing something obviously wrong?
Then, when I tried using feature_perturbation="interventional", it gave a different error:

explainer = shap.TreeExplainer(model, data=X)
Traceback (most recent call last):
File "", line 1, in
File "/lib/python3.7/site-packages/shap/explainers/tree.py", line 151, in init
self.expected_value = self.model.predict(self.data).mean(0)
File "/lib/python3.7/site-packages/shap/explainers/tree.py", line 972, in predict
raise NotImplementedError("Predict with pyspark isn't implemented. Don't run 'interventional' as feature_perturbation.")
NotImplementedError: Predict with pyspark isn't implemented. Don't run 'interventional' as feature_perturbation.

Also, if running the predictions with spark is complicated to implement, it might be worth adding the ability of the user to supply the expected predictions for validation.

You should get this error when your tree is built with a leaf without data inside. If you get this error, I assume you are using shap on a model built with a small data size?
Can you open another issue to track the implementation of the predictions with spark to make it works with interventional ?

You mean that not all paths to the leaf are the same length, so some of what would otherwise be intermediate nodes have no children?
Also, I created Issue #1192 to track spark predictions.

@QuentinAmbard
Are there news about the error of @amandolesi? Any new suggestion?
Thank you.

Hi All,
Is there any solution for NotImplementedError: CategoricalSplit are not yet implemented" in pyspark?

I run into the same problem, did you find out the solution to it?

I didn't had time to search what's causing this exactly these last weeks. I suspect there is a reference to spark kept somewhere and it breaks the serialization of the tree explainer with a spark model.  I'll have a look when I got some time, but it shouldn't be a big deal, especially if the serialisation is working with other models.

@QuentinAmbard
Hello Quentin, to recap and also give you some additional feedback, I performed some tests using a local standalone instance of spark.
As you mentioned a serialization error, I tried pickling a 'pyspark.ml.classification.RandomForestClassificationModel' object, basically a fitted pyspark random forest and I got a Py4J error, the same that @amandolesi reported above.
In explainer/tree.py, TreeExplainer class, row 695:
elif "pyspark.ml" in str(type(model)):
            assert_import("pyspark")
            self.original_model = model

so this serialization problem propagates. I tried commenting out "self.original_model = model" and I was then able to pickle the TreeExplainer object with a PySpark model. Of course it is a workaround but predictions are not implemented with PySpark yet, so commenting that line for the time being should not be an issue, what do you think about it?

Thanks @guidiandrea ! Absolutely that's what I had in mind too, but still haven't find time to do the change :/ The original_model was indeed kept in order to implement predictions (https://github.com/slundberg/shap/issues/1192) but I think we should find another way to avoid breaking serialisation with spark models.
Would you like to do the PR ?

Here you are:
https://github.com/slundberg/shap/pull/1307
Thank you @QuentinAmbard!

@slundberg I think we can now close this issue as everything should be solved with #1313

Looks like this should be closed @allard-jeff

I am still having `NotImplementedError: CategoricalSplit are not yet implemented" error. I am using shap==0.39.0 and Spark 3

@chengyin38
Does shap.TreeExplainer(pipeline_model.stages[-1]) work?

@chengyin38  The issue is that Shap can not handle categorical splits. So, in the Pyspark pre-processing you really need to drop the meta data from the data frame that pyspark will implicitly use to determine a feature is a categorical variable. This is done it seems as df= df.rdd.toDF(). String indexing without categorical splits might not be an optimal approach to the modeling however.

@QuentinAmbard I am finding that with Shap 0.39.0 this error continues with pyspark GBT.  The weird issue is that this error only seems to happen when a saved GBT is loaded. If I use the original model in memory, the error does not occur. Any ideas?
    assert self.model.fully_defined_weighting, "The background dataset you provided does " \
AssertionError: The background dataset you provided does not cover all the leaves in the model, so TreeExplainer cannot run with the feature_perturbation="tree_path_dependent" option! Try providing a larger background dataset, or using feature_perturbation="interventional".

Any idea how to fix this?
Model type not yet supported by TreeExplainer: <class ‘sparkdl.xgboost.xgboost_core.XgboostRegressorModel’>
I am trying to run this type of code:
xgboost = XgboostRegressor(**params)
pipeline = Pipeline(stages=[stringIndexer, vecAssembler, xgboost])
pipelineModel = pipeline.fit(trainDF)
explainer = shap.TreeExplainer(pipelineModel.stages[-1])

@QuentinAmbard I am finding that with Shap 0.39.0 this error continues with pyspark GBT. The weird issue is that this error only seems to happen when a saved GBT is loaded. If I use the original model in memory, the error does not occur. Any ideas?
    assert self.model.fully_defined_weighting, "The background dataset you provided does " \
AssertionError: The background dataset you provided does not cover all the leaves in the model, so TreeExplainer cannot run with the feature_perturbation="tree_path_dependent" option! Try providing a larger background dataset, or using feature_perturbation="interventional".

any updates with this problem?

@QuentinAmbard I am finding that with Shap 0.39.0 this error continues with pyspark GBT. The weird issue is that this error only seems to happen when a saved GBT is loaded. If I use the original model in memory, the error does not occur. Any ideas?
    assert self.model.fully_defined_weighting, "The background dataset you provided does " \
AssertionError: The background dataset you provided does not cover all the leaves in the model, so TreeExplainer cannot run with the feature_perturbation="tree_path_dependent" option! Try providing a larger background dataset, or using feature_perturbation="interventional".

Any progress with this problem? @QuentinAmbard


Wiki/User Manual

Nice project. Is there a user manual/wiki on how to use the app though?

https://github.com/MACEPA/EpiSample-Old/wiki
On Wed, Nov 2, 2016 at 12:01 PM, Jason Rogena<EMAIL_ADDRESS>wrote:

Nice project. Is there a user manual/wiki on how to use the app though?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/MACEPA/EpiSample/issues/1, or mute the thread
https://github.com/notifications/unsubscribe-auth/ALH2ihTrIjWFc_q6cNl3b1FwvaWK6em6ks5q6FFxgaJpZM4Km_s8
.

https://github.com/maxramirez84/EpiSample/wiki


Added appveyor file and SFML 2.4.0

Added appveyor support to ensure proper build of everything. It will try to install everything found in ports. (Running as I write this comment : https://ci.appveyor.com/project/Ninetainedo/vcpkg)
Added missing CONTROL files for TCL, TK and COCOS2D.
Added support for SFML 2.4.0
Added support for Sery 1.0.0

This is an impressive set of changes, but could you split them into individual PR's?
The libraries with missing CONTROL files (tcl, tk, cocos2d) likely don't work; they should probably be removed unless you've additionally fixed the portfiles.


Add the ability to create "lockfiles"

There's a very interesting new project - conda-lock which allows creating an explicit spec file  (conda list --explicit) without having to first create the environment.
i.e. you can create the lockfile without having to download and install all the packages.
It would be great if conda-lock could optionally use mamba for much faster (and parallel?) solutions.

I'm not sure where the functionality should be implemented but I'm just opening an issue here in case conda-lock requires any functionality not yet implemented in mamba
xref: https://github.com/mariusvniekerk/conda-lock/issues/20

It appears that mamba implements all the functionality (json output / dry-run) for conda-lock to work.
I guess this issue can be closed then modulo any thoughts on implementing the functionality natively in mamba

Hi! I wanted to check whether there were any plans for mamba to support this natively, or whether we should continue looking to conda-lock.
For context, at work we've been looking into creating a wrapper around conda/mamba to handle some of the intricacies of our internal workflow, and lockfiles are definitely a feature that we'd want to support. Should we primarily engage with the conda-lock project, or is mamba planning to take on some of this "environment management" work?

Hi @alanhdu we currently don't have any immediate plans to support the creation of lockfiles from within mamba.
If you are interested in contributing or sponsoring this development we could accelerate it.
I think the functionality is straight forward enough to be part of core-mamba.

That makes sense! I'll check in at work about carving out some time to contribute here, although we might just opt for building on top of conda-lock (my Python is significantly better than my C++). I'll let you know if we have any updates here!

conda-lock supports both mamba and micromamba now, so I think we can close this.
I am not super motivated to add conda-lock functionality into mamba itself right now, because I don't have a use-case myself, and conda-lock already exists and does an excellent job.


Add coerce methods to convert DateTime, Time and Date to Continuous

Further context: https://github.com/alan-turing-institute/MLJModels.jl/issues/234
This requires implementation of
coerce(::Arr{T}, ::Type{<:Union{Missing,Continuous}};
    verbosity::Int=1, tight::Bool=false) where T<:Union{Missing,DateTime}

in analogy with similar implementation at https://github.com/alan-turing-institute/MLJScientificTypes.jl/blob/b65844b14a0335ef1f936e39e07b0036acd75cb7/src/convention/coerce.jl#L52 (for coercing arbitrary Real data,  eg, Int, to Continuous element scitype). Here Arr is an alias for AbstractArray valid in that code block, which is where the new implementations should live.
Similar for Time and Date, although it may be possible to combine the implementations into one, or at least suck out the common code in a new function dispatched on the actual type.
For convenience, the coerce doc-string is copied below.
    coerce(A, ...; tight=false, verbosity=1)

Given a table `A`, return a copy of `A` ensuring that the scitype of the
columns match new specifications.
The specifications can be given as a a bunch of `colname=>Scitype` pairs or
as a dictionary whose keys are names and values are scientific types:

coerce(X, col1=>scitype1, col2=>scitype2, ... ; verbosity=1)
coerce(X, d::AbstractDict; verbosity=1)

One can also specify pairs of type `T1=>T2` in which case all columns with
scientific element type subtyping `Union{T1,Missing}` will be coerced to the
new specified scitype `T2`.

## Examples

Specifiying (name, scitype) pairs:

using CategoricalArrays, DataFrames, Tables
X = DataFrame(name=["Siri", "Robo", "Alexa", "Cortana"],
height=[152, missing, 148, 163],
rating=[1, 5, 2, 1])
Xc = coerce(X, :name=>Multiclass, :height=>Continuous, :rating=>OrderedFactor)
schema(Xc).scitypes # (Multiclass, Continuous, OrderedFactor)

Specifying (T1, T2) pairs:

X  = (x = [1, 2, 3],
y = rand(3),
z = [10, 20, 30])
Xc = coerce(X, Count=>Continuous)
schema(Xfixed).scitypes # (Continuous, Continuous, Continuous)

@venuur If you are still happy to help out, please assign yourself to this issue. You might want to wait for #23 to merge, for then you already have Dates in the Project.toml.
Please make any PR to the dev branch, as per contributing guidelines
Don't hesitate to ask questions here or on slack.

Happy to help. I should get around to it later in the week.

I have two initial questions.
First, what does the argument tight mean?
Second, I originally viewed this as a fit-transform type transformer, like OneHotEncoder. The reasoning is that the linearization can then be extrapolated correctly when running prediction, continuing the example from the original issue. Suppose the training features looks like the following
dt_feature | linearized_dt_feature
2018-01-01 | 1
2018-01-02 | 2
2018-01-03 | 3

If we then want to predict for the dates 2018-01-04, 2018-01-05, then the correct linearization is the following:
dt_feature | linearized_dt_feature
2018-01-04 | 4
2018-01-05 | 5

Here's the implementation of the fit-transform variant for reference from my current forecasting code:
# Linearize dates to fit and predict linear trends correctly.
MLJModelInterface.@mlj_model mutable struct DateLinearizer <: MLJModelInterface.Unsupervised
    features::Vector{Symbol} = Symbol[]
end

function MLJModelInterface.fit(model::DateLinearizer, verbosity::Int, X)
    fitresult = [minimum(getproperty(X, c)) for c in model.features]
    cache = nothing
    report = nothing
    return fitresult, cache, report
end

function MLJModelInterface.transform(model::DateLinearizer, fitresult, X)
    let X = copy(X)
        for (c, min_dt) in zip(model.features, fitresult)
            dts = getproperty(X, c)
            setproperty!(X, c, @. Float64(Dates.value(dts - min_dt)))
        end
        X
    end
end

Forgot to actually ask my question in the second part. Do you think this still makes since as a method of coerce given the use case I described, or am I conflating two different problems?

No, you're right. You have learned parameters here and so a transformer makes better sense.


AKS Periscope install failure

Hi, I'm trying to install AKS Periscope on a private cluster with egress through a firewall.   Diagnostics settings for the cluster are configured and point to a storage account.   When I try to run az aks kollect, I get the following error:

azureuser@jbl-vm1:~$ az aks kollect --name spoke1-aks --resource-group spoke1-vnet-rg
No storage account specified. Try getting storage account from diagnostic settings
This will deploy a daemon set to your cluster to collect logs and diagnostic information and save them to the storage account spoke1akslogs as outlined in http://aka.ms/AKSPeriscope.
If you share access to that storage account to Azure support, you consent to the terms outlined in http://aka.ms/DiagConsent.
Do you confirm? (y/N): y
Getting credentials for cluster spoke1-aks
Merged "spoke1-aks-admin" as current context in /tmp/tmpgfmwgfz_
Starts collecting diag info for cluster spoke1-aks
Cleaning up aks-periscope resources if existing
serviceaccount "aks-periscope-service-account" deleted
serviceaccount "default" deleted
configmap "containerlogs-config" deleted
configmap "kubeobjects-config" deleted
configmap "nodelogs-config" deleted
daemonset.extensions "aks-periscope" deleted
secret "azureblob-secret" deleted
secret "default-token-tgdh6" deleted
clusterrolebinding.rbac.authorization.k8s.io "aks-periscope-role-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "aks-periscope-role-binding-view" deleted
clusterrole.rbac.authorization.k8s.io "aks-periscope-role" deleted
diagnostic.aks-periscope.azure.github.com "aks-periscope-diagnostic-aks-nodepool1-36173997-vmss000000" deleted
diagnostic.aks-periscope.azure.github.com "aks-periscope-diagnostic-aks-nodepool1-36173997-vmss000002" deleted
customresourcedefinition.apiextensions.k8s.io "diagnostics.aks-periscope.azure.github.com" deleted
Deploying aks-periscope
The command failed with an unexpected error. Here is the traceback:
'NoneType' object has no attribute 'replace'
Traceback (most recent call last):
File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 608, in execute
raise ex
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 666, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 659, in _run_job
six.reraise(*sys.exc_info())
File "/opt/az/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 636, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 306, in call
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.6/site-packages/azure/cli/core/init.py", line 493, in default_command_handler
return op(**command_args)
File "/home/azureuser/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 1442, in aks_kollect
normalized_fqdn = mc.fqdn.replace('.', '-')
AttributeError: 'NoneType' object has no attribute 'replace'

Cool, I tried replicating this but not able to generate the error mentioned above, unless you have some old version of az cli running. The error mentioned is not specific to Periscope but failure in the underlying aks-preview, I tried looking little deep and the failure in your machine is caused by NoneType object  here this: https://github.com/Azure/azure-cli-extensions/blob/master/src/aks-preview/azext_aks_preview/custom.py#L1442
:bulb: Can you please share the specific steps how you are able to get the below error?
I have a feeling update az cli if its running on old version might help along with https://github.com/Azure/aks-periscope#user-guide removing and adding the aks-extension again.
cc: @JunSun17 , for anything extra which can be added.
Thanks heaps,

I tested from the same VM using another cluster that is not private (public API server endpoint) and I was able to get periscope installed.  However, I still can't deploy to a private cluster with egress through a firewall.  When I look at the pods in the aks-periscope namespace, they are in a error state.  To reproduce, start with a private cluster and try installing periscope.

Update:
I think CLI has a gap in support of private cluster.
As far as I can see by playing around with it, seem like even though via a jumpbox the private-cluster is not supported from periscope or the aks-preview command via az-cli
I will reach-out to few folks to get clarity, and update the document in this case that is the case.
Essentially from the cli point of view this jumpbox scenario on private DNS Zone is not getting correct qualified name hence the cli errors our in case of normal cluster this is not the case hence it works.
Surely there must be some network complexity in play here, will keep this thread updated further if I dig any information further for the storage.
Thanks,

Thanks @Tatsinnit for checking on this.  As part of my testing, I also noted that I needed to open my firewall to allow access to "aksrepos.azurecr.io" on port 443.  This is not mentioned here - https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic .

Just keeping this thread update:
Cool, to dig a little deeper, I have also opened an associated bug against az-cli-extension just to get full understanding, should be able to get to the bottom soon. Thanks.
Detail: https://github.com/Azure/azure-cli-extensions/issues/1679

@Tatsinnit Thanks for your help.  It looks like the fix for this was merged yesterday - https://github.com/Azure/azure-cli-extensions/pull/1685.  Not sure when a new release will be pushed but I will test as soon as it is available.

👍 All good @dstampfli , while playing with the whole thing I was able to replicate couple of other things which were relating to private cluster and team here collaborated across to fix all of those thing in 1 go.
Gist:

I have tested this internal with our test acr and it worked form my jumpbox in same subnet as private-cluster.
Once we have an official acr I can share documentation for direct deployment which you should be able to test.
Fyi: I am unsure as to when the cli-extension repo release cycle is to use it via Kollect command (will update latter on this).

Thanks

#TLDR
💡 Note: The new acr image for periscope is out. With regards to the az-cli release this will go to release S-169 and S-170 which to me seems like in few week. (I will check what these time line means from az-cli release.
#DETAILED
To unblock any other user for private-cluster , I did this to test our periscope level changes and it worked for me.
Use Direct Kubectl apply deployment command:

When user create private-cluster they should be aware how it works, I find this doc https://docs.microsoft.com/en-us/azure/aks/private-clusters and this section helpful https://docs.microsoft.com/en-us/azure/aks/private-clusters#options-for-connecting-to-the-private-cluster when I tried to replicate the above.
Once user create a JumpBox VM in the same Subnet (mine was ubuntu) as their private cluster, which will guarantee that your Private-cluster is visible to that VM.
Within that Jumpbox as sudo user, git clone the aks repo: https://github.com/Azure/aks-periscope
Open deployment/aks-periscope.yaml file and add Base64 Encoded Account Name and Account Key this is positional here https://github.com/Azure/aks-periscope/blob/master/deployment/aks-periscope.yaml#L108 (Save and close the file in your jump box)
Then run this command: https://github.com/Azure/aks-periscope/blob/master/docs/appendix.md For Example from my cloned repo I ran like this: kubectl apply -f aks-periscope.yaml

You will see output from periscope run and then check your storage account, I was successfully able to see a new container name > along with latest logs from my private-cluster

💡 Hiya: @dstampfli Can you you please check now, I think you need to update your az cli version and command should be hooked from az cli thanks.

@Tatsinnit I tested on 2 private clusters and it is working now!!!  I tested using az aks kollect with a storage account configured in the RG's diagnostic settings.  Thank you for your help!

Thank you so much for this loop back sounds awesome!! Closing this issue. 😊🙏


Game Request: Pavlov VR

Module Request
Game Information:

Pavlov VR Dedicated Server
Available on Steam
Linux Only
No console input or RCON. Log is dumped into a file you can just tail for console.
http://wiki.pavlov-vr.com/index.php?title=Dedicated_server
It does require some pre-reqs installed to run. Not sure if you can handle that in AMP or not.
I confirm:

[x] that I have searched for an existing module request for this application.
[x] that the server is an official server distributed by the games publishers/developers.
[x] that the server is available to the general public.
[x] that the server can be run headless (without a GUI).

Can the log file be changed? If not then this is non-viable.

I don't quite understand what you are asking for. The headless server doesn't support any input at all as of right now. Server log is dumped into pavlovserver/Pavlov/Saved/Logs/Pavlov.log and is essentially read-only. Admin commands are done in-game until they develop rcon input or something.

Without a way to tell the server to log to a particular file and without RCON or anything similar this is probably something best handled by the generic module then. In its current state the server looks non viable for AMP otherwise.

Fair enough. I'll await for documentation on the generic module then!


How to lead to a daily screen with time and events when clicked on a date with events

Hello, I am making this app where a user can log feelings whenever he / she wants. The log is written into firebase using a JSON structure like date -> time -> log. I would like to show certain feelings when a user clicks on a specific date. For example clicking on 7/16/17 would lead to a daily screen, like a table view with times of the logs (11.15 AM Happy, 1 PM Hopeful, 6 PM Strong). How can I achieve this through this library?
I looked over every tutorial but could not find much related to adding events. Thanks a lot for this amazing API and I would be really happy if you could reply ASAP as I am working on my app in a bootcamp :)

just wondering if you looked at the last tutorial on (this play list](https://www.youtube.com/watch?v=wyh_DVFeH_w&list=PLpqJf39XekqyUG7dxcqGO0JNprryysv9Q)?
Its called Common Questions # 1
It think it has an example on how to load things from a server and displaying them?
If this vid tutorial does not answer your question, then can you let me know on which part you are stuck? I can help further.

I did, and it does have an example, however in that example there is only one event per day, and I will have multiple logs in each day. I am having trouble getting my head around how to implement that, maybe by leading to a different screen? Is there a way to make the calendar lead to a daily view when clicked on a cell?

Update: I fully set up my calendar (the part from the tutorial) and the months do not seem to update as rapidly. If you could help out with that as well, that would be amazing :)

Hello I think I should be good on other issues, however right now the problem is that my scrolling does not work as yours do. I have to manually scroll everywhere whereas in your app, if you scroll between months, it automatically goes to that month. In addition, when I set scroll to today's date, it shows it in between maps. I noticed there was another thread related to this issue, however I couldn't find an answer there. Thanks a lot for the help in advance

ok i'm @ work right now.
But i can take a quick look if you provide a striped down version of your porject (with no secret company code in it).
If you provide a zipped file of the project i can take a quick look, else i can only look more into it after work in 7hrs

Okay, I will work on it myself until the evening and I will send you the source code for the calendar setup once I come back to home. I think I will be able to fix problems other than the scrolling. Thank you so much for this support!

Hello, sorry I couldn't email you last night, I will send the email you the source code for the calendar part right now

I sent you an email<EMAIL_ADDRESS>

i checked out the file.
This isnt an error.
Based on the image below, please select/deselect the following values of the JTAppleCalendarView on your storyboard.

I changed it and it works perfectly now, thank you so much for taking the time out of your day to help :)

Awesome. Cheers mate 🍻


[BACKEND] Make getStackPointer as interface of the TargetInfo to generalize getSharedMemoryBase usage

This decoupling allow different getStackPointer implement which is useful to support some new operation like tl.gather for out of tree backend.

Closing this PR as discussion.


App homescreen updates

[ ] Minor fix to icon to get rid of white underline
[ ] Fix title so that it is 3 lines: Establish; Guinness-o-Metric; Reforestation Trials
[ ] If possible change 3 home-screen buttons so they are vertically arranged rather than horizontal

@whmacken could you remake the icon to 1024 x 1024 pixels please?


Use gh instead of a dedicated action for working with releases

Creation of new draft releases and uploading release artifacts is now done using the GitHub CLI instead of some action.
The previous, official action is deprecated and instead of choosing an action by someone from the internet, it feels better to use the official CLI tool that's installed on the runners already.
Closes #16

Closes #9 as well


Support C-ish infrastructure

It looks like the time has come. Parts of this are being discussed at #339 but it deserves its own issue, which is: how to support optimizations in a faster language such as C, C++, or Cython? I think we could simply have Cython be a dependency, but that requires a C compiler like gcc to be present on the system so making it be optional might be more involved.

I have been doing some research on this, and here is a brief summary.
Cython is a superset of of Python, and its source code needs to go through 2 stages of compilation:

Compile Cython source code into platform independent C/ C++;
Compile generated C/ C++ source into a shared library with a standard C/ C++ compiler;

The standard approach for those 2 stages is:

cythonize from Cython;
distutils from standard Python library;

Eventually it goes back to priority of speeding up in scientific computations, which I'm not sure yet. @babbush
References:

Cython: A Guide for Python Programmers (link).
scipy source code.

I think this is not currently urgent and it would be a very large project, so I'm going to close the issue. But thank you for this research Sheila.


FPGA: Update README to include note about emulator environment variables

Description
The FPGA Emulator has modified how pipe depth is emulated. If a design wishes to increase the host buffer capacity (in emulation) of a host pipe to larger than the buffer on the device, CL_CONFIG_CHANNEL_DEPTH_EMULATION_MODE will have to be set to ignore-depth.
This change updates the README's of designs that do exactly that, and were previously relying on the emulator defaulting to this behaviour.
External Dependencies

N/A

Type of change
Please delete options that are not relevant. Add a 'X' to the one that is applicable.

[X] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

These are just changes to the README. No code was touched.

@justin-rosner I thought we were setting the 'infinite host pipe host buffer depth' behaviour to be default:
https://hsdes.intel.com/appstore/article/#/14019216094
why do we need to set an environment variable?

@justin-rosner I thought we were setting the 'infinite host pipe host buffer depth' behaviour to be default:
https://hsdes.intel.com/appstore/article/#/14019216094
why do we need to set an environment variable?

We made that change for the simulator, but when testing the latest sycl/rel resource it appears that the emulator team made the opposite decision. We will have to file a case to the emulator team for the 2025.1 release to have the default once again be infinite pipe depth .

The change that caused this behaviour was reverted, so we no longer need this change.


Use the nydus-snapshotter by default

This PR is a rework of https://github.com/confidential-containers/operator/pull/263, but I don't want to lose that public reference of what's been done.
Please, take a look at each  commit for more info on what it's doing and why.

/test

/test

/test

/test

/test

/test-kata-qemu-tdx

/test-kata-qemu-tdx

/test

We might see more SEV/SNP failures if other PRs (or the baseline) have run given that they will leave the nodes in a bad state. Should work if we rerun a few times tho.

Cool, thanks!
Right now I'm re-working the uninstall as it made the jobs related to uninstall / reinstall break.

/test

/test

/test

/test


[Simplex] What is the best command to reproduce LightGCN-CCL and YouTubeNet-CCL?

Hello,
I want to reproduce results for  LightGCN-CCL  and YouTubeNet-CCL.
What is the best way? Which command should I use?

You can refer the YouTubeNet here and modify the loss in the config: https://github.com/reczoo/RecBox/tree/v0.0.3/model_zoo/YouTubeNet
Currently we did not provide LightGCN-CCL.


Shrink derive output size

This PR has a series of commits that shrink the size of the code generated for derive(Serialize, Deserialize), particularly for large structs.
Here are some instruction count results for some synthetic benchmarks.

Benchmark
Profile
Scenario
% Change
Significance Factor?

big300
check
full
-31.80%
159.00x

big300
debug
full
-30.15%
150.75x

big300
opt
full
-29.99%
149.96x

big200
check
full
-29.44%
147.19x

big200
debug
full
-26.94%
134.69x

big200
opt
full
-26.83%
134.15x

big100
check
full
-25.50%
127.48x

big100
debug
full
-21.02%
105.11x

big100
opt
full
-20.73%
103.64x

derive-serde
check
full
-6.50%
32.50x

derive-serde
opt
full
-3.97%
19.85x

derive-serde
debug
full
-3.76%
18.82x

big{1,2,3}00 each contain a single struct with {100,200,300} u32 fields that is annotated with derive(Serialize, Deserialize).
derive-serde contains a variety of structs and enums. It is identical to https://github.com/rust-lang/rust/blob/master/src/test/ui/deriving/deriving-all-codegen.rs except with the builtin trait derives replaced with derive(Serialize, Deserialize).

What the table headers mean?

Any thoughts on whether this change is a good one? There are 32 positive emoji reactions on the description. I'm happy to make adjustments (e.g. moving that method to a free function), but it's not yet clear to me if this change is wanted by the serde developers. Thanks!

Summary: the wins in terms of size reduction in a "real world" project that I tested are unfortunately very small.

I was very happy to see this PR as I recently found out that generated serde code makes up a big part of the final binary of a project I'm working on as part of my day job. It's a web application with Rust backend that performs various tasks and uses serde derive for lots of different stuff. (It's even open source, if anyone's interested: link). So I thought it might be a valuable data point to see how this PR performs with a "production project".
I know that this PR is mainly motivated by compile times, but I thought checking it for binary output size is interesting too.
Unfortunately, the results were not as great as I has hoped for. I tested serde 1.0.139 vs. this branch (f8cc37c8), using [patch]. I performed a release build with these adjustments:
[profile.release]
debug = 1
codegen-units = 1
lto = "thin"

[profile.release.package."*"]
debug = false

We need basic debug info to better debug errors in production. And it allows me to check per symbol size! Also, some custom --remap-path-prefix flags are set and the resulting binary is compressed with objcopy --compress-debug-sections.
But first, the final file sizes (the -stripped files had strip applied to them):
23602168  smaller-serde
23595960  with-1.0.139
14708592  smaller-serde-stripped
14720880  with-1.0.139-stripped

With this patch, the binary including debug info is even larger than with 1.0.139. That was unexpected! At least the stripped one shows a small win. I tried only checking the size of serde functions. I did that via this command chain:
llvm-nm-10 --print-size --size-sort --radix=d <binary>
    | rg -F <filter>
    | cut -d ' ' -f 2
    | awk '{s+=$1} END {print s}'

Here is the result matrix for the two variables <filter> and <binary>. Note that my llvm-nm-10 did not demangle the symbols, that's the reason for the .. there.

Filter:
serde
serde..de..Deserialize
serde..ser..Serialize

with-1.0.139
809,524
525,434
56,330

smaller-serde
796,166
510,409
56,330

That's a roughly 13KiB reduction = 98.3% of the original size. A win is a win, but I had hoped for more and I feel like there are a lot of size optimization that could still be done.
I also looked at the biggest functions to see what changed there.

20 largest serde functions
with-1.0.139:
 6834 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 6936 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 7143 <tobira::search::confique_partial_meili_config::_::<impl serde::de::Deserialize for tobira::search::confique_partial_meili_config::PartialMeiliConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 7536 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 7595 <meilisearch_sdk::errors::_::<impl serde::de::Deserialize for meilisearch_sdk::errors::ErrorCode>::deserialize::__FieldVisitor as serde::de::Visitor>::visit_bytes
 7771 <tobira::config::theme::confique_partial_theme_config::_::<impl serde::de::Deserialize for tobira::config::theme::confique_partial_theme_config::PartialThemeConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 7977 <toml::de::MapVisitor as serde::de::Deserializer>::deserialize_any
 8026 <tobira::config::general::confique_partial_general_config::_::<impl serde::de::Deserialize for tobira::config::general::confique_partial_general_config::PartialGeneralConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
10939 <serde::de::impls::OptionVisitor<T> as serde::de::Visitor>::visit_some
11060 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
13154 <tobira::auth::confique_partial_auth_config::_::<impl serde::de::Deserialize for tobira::auth::confique_partial_auth_config::PartialAuthConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
13461 <tobira::search::event::_::<impl serde::de::Deserialize for tobira::search::event::Event>::deserialize::__Visitor as serde::de::Visitor>::visit_map
15906 <toml::de::ValueDeserializer as serde::de::Deserializer>::deserialize_struct
17591 meilisearch_sdk::tasks::_::<impl serde::de::Deserialize for meilisearch_sdk::tasks::TaskType>::deserialize
19177 tobira::sync::harvest::response::_::<impl serde::de::Deserialize for tobira::sync::harvest::response::HarvestItem>::deserialize
20449 <toml::de::MapVisitor as serde::de::MapAccess>::next_value_seed
22081 meilisearch_sdk::tasks::_::<impl serde::de::Deserialize for meilisearch_sdk::tasks::Task>::deserialize
24697 <serde::__private::de::content::ContentDeserializer<E> as serde::de::Deserializer>::deserialize_any
26080 <juniper::http::GraphQLResponse<T> as serde::ser::Serialize>::serialize
95307 <tobira::config::confique_partial_config::_::<impl serde::de::Deserialize for tobira::config::confique_partial_config::PartialConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map

smaller-serde
 6605 <tobira::sync::confique_partial_sync_config::_::<impl serde::de::Deserialize for tobira::sync::confique_partial_sync_config::PartialSyncConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 6607 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 7143 <tobira::search::confique_partial_meili_config::_::<impl serde::de::Deserialize for tobira::search::confique_partial_meili_config::PartialMeiliConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 7212 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 7444 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
 7595 <meilisearch_sdk::errors::_::<impl serde::de::Deserialize for meilisearch_sdk::errors::ErrorCode>::deserialize::__FieldVisitor as serde::de::Visitor>::visit_bytes
 7744 <tobira::config::theme::confique_partial_theme_config::_::<impl serde::de::Deserialize for tobira::config::theme::confique_partial_theme_config::PartialThemeConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 7937 <tobira::config::general::confique_partial_general_config::_::<impl serde::de::Deserialize for tobira::config::general::confique_partial_general_config::PartialGeneralConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
 7977 <toml::de::MapVisitor as serde::de::Deserializer>::deserialize_any
10923 <serde::de::impls::OptionVisitor<T> as serde::de::Visitor>::visit_some
11031 <core::marker::PhantomData<T> as serde::de::DeserializeSeed>::deserialize
13239 <tobira::auth::confique_partial_auth_config::_::<impl serde::de::Deserialize for tobira::auth::confique_partial_auth_config::PartialAuthConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map
13324 <tobira::search::event::_::<impl serde::de::Deserialize for tobira::search::event::Event>::deserialize::__Visitor as serde::de::Visitor>::visit_map
15929 <toml::de::ValueDeserializer as serde::de::Deserializer>::deserialize_struct
17396 meilisearch_sdk::tasks::_::<impl serde::de::Deserialize for meilisearch_sdk::tasks::TaskType>::deserialize
20435 <toml::de::MapVisitor as serde::de::MapAccess>::next_value_seed
21953 meilisearch_sdk::tasks::_::<impl serde::de::Deserialize for meilisearch_sdk::tasks::Task>::deserialize
26080 <juniper::http::GraphQLResponse<T> as serde::ser::Serialize>::serialize
32282 tobira::sync::harvest::response::_::<impl serde::de::Deserialize for tobira::sync::harvest::response::HarvestItem>::deserialize
93552 <tobira::config::confique_partial_config::_::<impl serde::de::Deserialize for tobira::config::confique_partial_config::PartialConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map

A few things:

The tobira::sync::harvest.... function is significantly larger with this path (19177 -> 32282). The struct definition is here.
I'm not sure why there are several PhantomData as DeserializeSeed impls in there.
Some functions stayed exactly the same size.
The largest function is quite a lot larger than everything else, so it might be worth checking that out specifically. But I don't have time to dig into that now. I can roughly say that it's a a struct with 9 fields of type Option<X> where X is different per field and is always another struct with several Option fields. If anyone is interested in digging into that, I'm happy to help (as the confique thingy is by me).

Ok, I hope this investigation is useful at all. As someone with 800KiB of serde function in my release artifact, I'm very interested in improvements in this area ^_^

Thank you Lukas, your writeup is enormously helpful in evaluating this PR. One thing I would want to look into is whether any of the "small serde" redesigns (miniserde, deser, …) are viable for your use case, or even do at all better on code size.
In response to @nnethercote's last comment: it is not unwanted, but it is unsolicited, and an unsolicited PR (i.e. not in response to a "help wanted" label or an issue discussion which I participate in) is always less likely to appear at the top of my priorities. The way to get a PR reviewed is by being the most impactful thing for me to dedicate time to according to my opinion, and my sense has been this is not that, since there are multiple other serde PRs that have been queued longer and which I would rather land over this. Recently though my biggest chunk of time is going to a major serde_yaml revamp since that crate has been neglected much worse and longer than this crate.
You called out how impressive the emoji reactions to this PR are, but that is not a signal I look at in deciding where to invest time. It is predominantly a measure of social media reach over anything else, and prioritizing PRs from whoever has the highest twitter followers, rather than whichever is the most impactful contribution to the project, is not a productive direction I think.

I made some tests with rust-analyzer (it mostly uses serde via lsp-types) and the results are a mixed bag, too:
# run cargo clean, then cargo build --release

# v1.0.139, incremental = true, opt-level = 3
   text	   data	    bss	    dec	    hex	filename
25289491	1451288	   3784	26744563	19816f3	target/release/rust-analyzer

# nn, incremental = true, opt-level = 3
   text	   data	    bss	    dec	    hex	filename
25296787	1448360	   3784	26748931	1982803	target/release/rust-analyzer

# v1.0.139, incremental = true, opt-level = "s"
   text	   data	    bss	    dec	    hex	filename
22104455	1663648	   3784	23771887	16abaef	target/release/rust-analyzer

# nn, incremental = true, opt-level = "s"
   text	   data	    bss	    dec	    hex	filename
22115079	1660808	   3784	23779671	16ad957	target/release/rust-analyzer

Notice how the text increase is somewhat offset by data, but only in the opt-level = 3 case. I haven't specifically checked the size of the serde functions, but they add up to 1 144 089 bytes in the third case above.

Still, I think this is worth pursuing. serde is in the top 10 of most downloaded crates, so any compile time or code size wins will add up across the ecosystem. I'd like to thank nnethercote for putting in the effort to optimize not only the Rust compiler, but also many of the popular third-party crate.

@LukasKalbertodt, @lnicola: thanks for the data. Your results are expected, though I see now this wasn't clear from my original description. The change I made shrinks the size of the Rust code that the front end has to process, but the new functions are marked inline, so by the time inlining happens (either at the MIR level or at the LLVM IR level) the final code will be very similar to the current. Which means this PR will help compile times but have little effect on binary size. If you removed the inline annotations from the new functions things might be different, but ser/deser speed might also be slower.
For binary size, my measurements show that Deserialize::visit_map is easily the biggest function. E.g for the stress test of a struct with 100 u32 fields (which is big but within the realms of possibility) the generated visit_map is 2833 lines of code before this PR, and 2133 lines of code after this PR.
AFAICT, the generated visit_map is able to handle struct fields being present in any order, even though the generated Serialize::serialize always writes the fields in declaration order. If visit_map could assume the fields were in declaration order it could be shorter.

AFAICT, the generated visit_map is able to handle struct fields being present in any order, even though the generated Serialize::serialize always writes the fields in declaration order. If visit_map could assume the fields were in declaration order it could be shorter.

AFAIK, the visitor is driven by the deserializer which can mean that it is driven by the order of the fields in the wire format. Hence it needs to be able handle any order to be able to deserialize the output of systems other than Serde.

From the other hand a new struct attribute could be added, for example, #[serde(strict_order)] which would allow only the order that is defined in the struct

Something that did likely also help would be an attribute to only allow named structs and enums (for json, toml, ...) or index based structs and enums (for bincode, ...), while disallowing the other for the type it is applied to. In the common case where you only need to serialize/deserialize a single format, that would cut the size of the derived Deserialize impl in half I think.

There should be one single non-generic DeserializeSeed type that is used for deserializing all fields of all data structures, because none of that logic is different depending on whose fields they are.

Something like this?


FriendlyEats doesn't imports firebase.js

Ideally, we should update our sample to only import the dependencies as needed:

firebase-app.js
firebase-auth.js
firebase-firestore.js

As is, we are importing 777 KB of all of Firebase.

@samtstern @abehaskins


Fix Audio Book page navigation Error

Related Issue
In Audio Book page header navigation showing error page not found.
Fixes:  #3703
Description
As you can see in below demo video in Audio Book page header navigation showing error page not found. So i fized the navigation issue and add correct link.
Type of PR

[x] Bug fix
[x] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
Before
https://github.com/user-attachments/assets/4bc66ed7-efa3-4551-89a4-02ceb006a76d
After fixing the navigation page error not found
https://github.com/user-attachments/assets/1abc0c3d-6437-4b85-bc87-62e52b922dd1
Checklist:

[x] I have made this change from my own.
[ ] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

when we click on [about] in navbar . we get [ page not found ] . Correct that link too .

@abhi03ruchi  @sailaja-adapa  Now i have also fix About link issue page not found error, in Audio Book page.

resolve conflicts , then will merge pr .

@abhi03ruchi  I hace solve the merge conflict issue. Now you can ckeck.


Make changeset directory names human readable so they're easier to remember

Libraries I found with some quick searching:

https://www.npmjs.com/package/human-readable-ids
https://github.com/jjt/hashwords
https://github.com/davidmarkclements/hash-phrase

Merged