TS 4.1 regression in type assertion of generic mapped type

TypeScript Version:  4.1.2
Search Terms: CFA, narrowing, generic, mapped type, conditional type, type assertion
Code
interface A { x: number }

declare function isA(a: unknown): a is A;

type FunctionsObj<T> = {
  [K in keyof T]: () => unknown
}

function g<
  T extends FunctionsObj<T>,
  M extends keyof T
>(a2: ReturnType<T[M]>) {
  if (isA(a2)) {
    // a2 is not narrowed
    a2.x // error, but should be ok
  }
}

Expected behavior:
Type of a2 should be narrowed to ReturnType<T[M]> & A
It works this way in TS 4.0
Actual behavior:
Type of a2 remains only ReturnType<T[M]>
Playground Link: https://www.typescriptlang.org/play?ts=4.1.0-beta#code/JYOwLgpgTgZghgYwgAgILIN7IB4C5kgCuAtgEbTIC+AUNQCYQIA2cUKMhICYwA9iMmABnVAAo4+TgGsQvAO4gAlPjiChaANy0wATwAOKAGKdufEEIDypAFYAeACoA+ZAF5M1ZMgDaAaUECpCB1eGGR7AF18UUVXZ2lZBWoaag4uHn5kAHNbDzDkCGxIEDp1YzSzSxsHRwAaXIBZfMKIYvVA4ND7akdxACZ8ACUIMEIoEHt9CAcvevDHGIxc4FDRYTE4XsUF3M8NgDpsZAB6I-yoKF4oXJoaIA

I tried to reduce the repro, but it reproduces only when all these present:

conditional type that is being narrowed ReturnType<>
generic mapped type that is constraints by itself T extends FunctionsObj<T>
field accessed with generic type T[M]

@typescript-bot run repros

Here's the problem, but without control flow involved:
interface A { x: number }

declare function isA(a: unknown): a is A;

type FunctionsObj<T> = {
    [K in keyof T]: () => unknown
}

function g<
    T extends FunctionsObj<T>,
    M extends keyof T
>(a2: ReturnType<T[M]>, x: A) {
    x = a2; // shouldn't be allowed, but is
}

This is a bug in our subtyping rules, not control flow (which would undoubtedly be easier to fix).
@typescript-bot run repros

Looks like the issue is the circular-ish constraint of T. When we're relating the source and the target, we so a series of simplifications on the source for our (admittedly inaccurate) distributive constraint, which look like
ReturnType<T[keyof T]>

Looks like the root cause is our change in 4.1 to isDeeplyNestedType - it is now erroneously flagging a target side of the comparison, which doesn't change, as "deeply nested". Since the source side of the relation goes through about 6 transforms with the target side remaining unchanged, both sides get flagged as "deeply nesting", resulting in a Maybe result which eventually allows the assignment. I think this can be remedied by having separate source/target stack depths, so we don't increase the target stack depth when we're only transforming the source for a comparison.


FE setup - header - footer

Fonts & Icons set up
Footer & header UI implementation
Vuetify theme set up  and use global config to adjust Vuetify's behavior in alignment with the ZT Figma design

Hi @yuwu9145, Looks great! Thank you for making the changes.  We had a couple changes to request:

Could we move the Notebook/App toggle to the center of the App Bar (header)? We want to maximize the amount of vertical space available to the user.
Could we reduce the height of the footer by about maybe 1/3?  Same reason as above and decreasing the text size is fine as well.  If it looks better when the text is smaller to reduce padding between the elements in the footer that is fine as well


This fixes bug in TZ build that fails with perms

Sets USER to root in the dockerfile for portion of docker build
for dpkg and apt command section
Signed-off-by: Eric Van Hensbergen<EMAIL_ADDRESS>

Is this still meant to be a draft or is it complete?

I was waiting for the test to complete.  There are some other fixes but
they are unrelated to the original issue so I'll mark this as complete for
now and submit the others elsewhere.
On Fri, May 14, 2021 at 7:50 AM Dominic Mulligan @.***>
wrote:

Is this still meant to be a draft or is it complete?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/veracruz-project/veracruz-docker-image/pull/10#issuecomment-841223303,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAQ7WGGPKYMWVT2PVJDVILTNUMCXANCNFSM44226SDQ
.

+1 reached, merging.


Dockerise this?

Would it be possible to dockerise this application?
It sounds really useful, but I can't go around installing python & pip on our production boxes. But if it were available as a container I could pull that & run it inside docker.
I'm guessing it would need the docker socket to be mounted to give it access to the docker server? Or it could accept JSON piped from docker inspect via the cli and then output the reverse-engineered docker run command and exit.
$ docker inspect my_running_ctr | docker run assaflavie/runlike
Am I missing anything / are there other reasons that this wouldn't work?

It'll work. I don't like mounting the docker socket on containers in general... but if you can't get python/pip on your servers, that's probably the way to go. It would be a 3 line Dockerfile basically...
But, also, this hasn't been updated in a while, so trust but verify (especially if you're using relatively recent docker features). Currently it knows the most basic docker flags (the ones that have been around for ~ 2 years), like port publishing, volumes, etc.
I should really get around to updating this.

I've pushed this as an image now.
So you can do something like:
docker run \
    -v /var/run/docker.sock:/var/run/docker.sock \
    assaflavie/runlike -p SOME-CONTAINER


Building Singularity image crashes with "not an empty directory"

Description/Reproduction:
When following instructions given here to create Singularity/Apptainer image:
pip3 install spython
cd algorithmic-efficiency/docker
spython recipe Dockerfile &> Singularity.def
singularity build --fakeroot mlcommons.sif Singularity.def

The last command (i.e. building the Singularity image) crashes with the following error:
Setting up algorithmic_efficiency repo
+ branch=main
+ framework=both
+ git_url=https://github.com/mlcommons/algorithmic-efficiency.git
+ git clone https://github.com/mlcommons/algorithmic-efficiency.git
fatal: destination path 'algorithmic-efficiency' already exists and is not an empty directory.
+ cd /algorithmic-efficiency
+ git checkout main

Explanation:
Observe that the automatically generated .def includes the following line:
%files
scripts/startup.sh /algorithmic-efficiency/docker/scripts/startup.sh

When executed, this line copies the file from the first path (on host machine) into the second path (inside container). And for this, it recreates the full path, thus creating an algorithmic-efficiency directory.
Then, further down the line, git will crash when attempting to clone into that directory, since it already exists.
Fix:
Note how, if we were to git clone successfully, the startup.sh file would end up in the exact same place anyway. Further note that none of the in-between commands depends on that file in any way. Therefore, this issue should be fixed by either of the following:

Suppress the %files command by regex matching
Recursively delete algorithmic-efficiency prior to git clone
Tell the converter to not generate the %files command in the first place, in a programmatical way (cleanest)

To confirm this, manually suppressing the %files command indeed allowed compilation to continue. To pursue the cleanest fix, I created a Python script that mimics the spython command, with the only difference that it does not produce a %files comamnd. Using the created file allowed to continue the installation process without issues, so I'll propose a PR with the fix.
I'll copypaste the file in a commend for further reference.

Alright, so PR https://github.com/mlcommons/algorithmic-efficiency/pull/553 is the proposed fix.
For reference, the singularity_converter.py script (present in the PR) is:
"""
This script is a modification of the
``spython recipe Dockerfile &> Singularity.def`` command, implemented here:
github.com/singularityhub/singularity-cli/blob/master/spython/client/recipe.py
It converts the Docker recipy to Singularity, but suppressing any %files
command. Usage example:
python singularity_converter.py -i Dockerfile -o Singularity.def
"""

import argparse
#
import spython
from spython.main.parse.parsers import get_parser
from spython.main.parse.writers import get_writer

# globals
ENTRY_POINT = "/bin/bash"  # seems to be a good default
FORCE = False  # seems to be a good default
#
parser = argparse.ArgumentParser(description="Custom Singularity converter")
parser.add_argument('-i', '--input', type=str,
                    help="Docker input path", default="Dockerfile")
parser.add_argument('-o', '--output', type=str,
                    help="Singularity output path", default="Singularity.def")
args = parser.parse_args()
INPUT_DOCKERFILE_PATH = args.input
OUTPUT_SINGULARITY_PATH = args.output

# create Docker parser and Singularity writer
parser = get_parser("docker")
writer = get_writer("singularity")

# parse Dockerfile into Singularity and suppress %files commands
recipeParser = parser(INPUT_DOCKERFILE_PATH)
recipeWriter = writer(recipeParser.recipe)
key, = recipeParser.recipe.keys()
recipeWriter.recipe[key].files = []

# convert to string and save to output file
result = recipeWriter.convert(runscript=ENTRY_POINT, force=FORCE)
with open(OUTPUT_SINGULARITY_PATH, "w") as f:
    f.write(result)


Fixed README (example usage)

Thanks, this got addressed as part of #93
https://github.com/ipfs-shipyard/ipfs-geoip#usage


facility link snapping

Adds CLI method for snapping activity facilities to a network geometry.
It can be invoked as: pam snap-facilities <path_population_in> <path_population_out> <path_network_geometry> -f <link_id_field>
pam snap-facilities --help

Usage: pam snap-facilities [OPTIONS] PATH_POPULATION_IN PATH_POPULATION_OUT
                           PATH_NETWORK_GEOMETRY

  Snap facilities to a network geometry.

Options:
  -f, --link_id_field TEXT  The link ID field to use in the network shapefile.
                            Defaults to 'id'.
  --help                    Show this message and exit.

For each activity, the method will identify the closest link in the shapefile, and add it as a link=.. attribute to the activity element. Finally, the updated plans file will be saved to a new xml file.
Checklist
Any checks which are not relevant to the PR can be pre-checked by the PR creator.
All others should be checked by the reviewer(s).
You can add extra checklist items here if required by the PR.

[X] CHANGELOG updated
[X] Tests added to cover contribution
[ ] Documentation updated

Thanks @Theodore-Chatziioannou , picking this up now

Hi @Theodore-Chatziioannou,
I've tested the new pam snap-facilities function with 100 agents' plan for Londinium, and it worked quite well!
When I tried it on TE pop, I noticed it perform quite slow when scaling up to larger populations. It took about 30 minutes of runtime for 0.01% of the population for TE pop.
I feel the primary slowdown seems to come from the way distances are calculated for each activity to all network links. Maybe some vectorized operations or something similar would help to potential speed-ups to process?

Previously, the snap_facilities_to_network performed slowly when it needed to work with big population inputs.
Now it utilizes cKDTree for efficient nearest neighbour search and batch processing of activity locations. This update improves performance in snapping facilities to the network geometry now. @Theodore-Chatziioannou

thanks @syhwawa

your change has broken the CI - can you fix the linting? I believe this due to using single quotes
the code can be simpler. There is no need to two separate loops, one collecting the data and another applying it, saving the intermediate data in activity_points and activities_info. The updated locations can be assigned directly in the first loop.

Thanks for the comments @Theodore-Chatziioannou
The function has been updated to remove the separate loops for collecting and applying data. Now, the updated locations are queried and assigned directly within the first loop. And the lint error has been fixed.


Peer Review Feedback (from Rahul Kuriyedath)

Hi Group 30,
Please find below my feedback for the work you have done until milestone 3. :

I faced the below error when I ran ‘make all’ in the terminal:

rahulkuriyedath@Rahuls-MBP:~/Desktop/Labs/522-Group_30-Rockstars$ make allpython src/download_data.py --url=https://data.strathcona.ca/api/views/c9fr-ivqf/rows.csv?accessType=DOWNLOAD \
	--out_file=data/2018_Property_Tax_Assessment.csv
python src/data_cleaning.py --in_file=data/2018_Property_Tax_Assessment.csv \
	--out_file1=data/2018_Property_Tax_Assessment_clean.csv \
	--out_file2=data/2018_Property_Tax_Assessment_clean_train.csv \
	--out_file3=data/2018_Property_Tax_Assessment_clean_test.csv
python src/eda_script.py --in_file1=data/2018_Property_Tax_Assessment_clean.csv \
	--in_file2=data/2018_Property_Tax_Assessment_clean_train.csv --output_file=results/
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 25605 entries, 0 to 25604
Data columns (total 12 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   YEAR_BUILT  25597 non-null  float64
 1   ASSESSCLAS  25605 non-null  object
 2   BLDG_DESC   25597 non-null  object
 3   BLDG_FEET   25605 non-null  int64
 4   GARAGE      25605 non-null  object
 5   FIREPLACE   25605 non-null  object
 6   BASEMENT    25605 non-null  object
 7   BSMTDEVL    25605 non-null  object
 8   ASSESSMENT  25605 non-null  int64
 9   LATITUDE    25605 non-null  float64
 10  LONGITUDE   25605 non-null  float64
 11  AGE         25597 non-null  float64
dtypes: float64(4), int64(2), object(6)
memory usage: 2.3+ MB
SyntaxError: Unexpected token ']'
    at Function (<anonymous>)
    at Object.field (/Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-util/build/vega-util.js:110:7)
    at Object.getField [as parse] (/Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:132:47)
    at parseParameter (/Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:80:18)
    at /Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:65:42
    at Array.map (<anonymous>)
    at parseParameters (/Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:65:17)
    at parseOperatorParameters (/Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:220:9)
    at /Users/rahulkuriyedath/opt/miniconda3/envs/571/lib/vega-cli/node_modules/vega-runtime/build/vega-runtime.js:320:7
    at Array.forEach (<anonymous>)
python src/housing_assessment_prediction.py \
	--in_file1=data/2018_Property_Tax_Assessment_clean_train.csv \
	--in_file2=data/2018_Property_Tax_Assessment_clean_test.csv \
	--out_file1=results/validation_table.csv --out_file2=results/test_score.csv --out_file3=results/coefficients_table.csv
Rscript -e "rmarkdown::render('doc/strathcona_housing_price_predict_report.Rmd')"

processing file: strathcona_housing_price_predict_report.Rmd
  |....                                                                  |   5%
  ordinary text without R code

  |.......                                                               |  11%
label: setup (with options)
List of 1
 $ include: logi FALSE

Quitting from lines 12-16 (strathcona_housing_price_predict_report.Rmd)
Error in library(reticulate) : there is no package called 'reticulate'
Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> library

Execution halted
make: *** [doc/strathcona_housing_price_predict_report.Rmd] Error 1

In the readme.MD file in the root directory, in the ‘about’ section the first paragraph seems to be talking about breast cancer prediction while your project title says housing price prediction. Probably this was just a miss, and you could easily correct it :)

In the dependency section, it might be nice to include all the dependencies in a yaml file as it will make the process of reproducing the environment easier.

There is no section in the readme that tells how I should run the make file commands. You could add a line in the readme something like - ‘to reproduce this project, run the command make all in the terminal ‘.

For the plots scatter.png and barchart.png, the render looked a little funny in my local system (it had a dark background with dark text which made it a little difficult to read). But I previewed the same on the github repo and it looked good, just thought I will give you a heads up in case you want to look into it.

In the strathcona_housing_price_predict_report.md, I see an empty references section but it is not populated with any details

I hope the above points help you in polishing what otherwise looks like a great project! Please let me know if you need any clarifications.

Hi Rahul,
We appreciate your great feedback and kind words. I was also struggling with the reticulate library. It should be all fixed with the new docker image. We will also address all your other comments.
Thank you for your time and for collaborating to this project! =)


Fix Windows regresion on OgreLightVisual

Related with this regression issue https://github.com/ignitionrobotics/ign-rendering/issues/226
Signed-off-by: ahcorde<EMAIL_ADDRESS>

Weird to see the heightmap warning on Windows, it had been addressed on #180. I think the pragmas can be extended to wrap the whole file.

@chapulina should I fix this here ?

@chapulina should I fix this here ?

That would be great, thanks!

@j-rivero or @chapulina  any idea about this one:
D:\Jenkins\workspace\ign_rendering-pr-win\ws\ign-rendering\ogre2\src\Ogre2Light.cc(284,1): fatal error C1001: Internal compiler error. [D:\Jenkins\workspace\ign_rendering-pr-win\ws\build\ignition-rendering5\ogre2\src\ignition-rendering5-ogre2.vcxproj]

any idea about this one:

That looks very strange, I retriggered the build :thinking:

@osrf-jenkins retest this please

@osrf-jenkins retest this please

@osrf-jenkins retest this please


pkgs/stdenv/generic/make-derivation.nix: add a bug reference to stric…

…tDepsByDefault TODO
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Related to https://github.com/NixOS/nixpkgs/issues/178468


Oracle Linux 9 Installation Failure

Environmental Info:
K3s Version:
v1.23.8+k3s1
Node(s) CPU architecture, OS, and Version:
Linux ol9.mshome.net 5.15.0-0.30.19.el9uek.x86_64 #2 SMP Wed Jun 15 22:59:49 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
Single node installation
Describe the bug:
K3s does not install
Steps To Reproduce:
curl -sfL https://get.k3s.io | sh -
Expected behavior:
k3s is installed
Actual behavior:
Installation script fails during installation of k3s-selinux package with:
Problem opening package k3s-selinux-1.1-1.el8.noarch.rpm
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Additional context / logs:

Backporting

[ ] Needs backporting to older releases

Can you try with curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=testing sh -

Jul 07 21:54:06 ol9.mshome.net k3s[1998]: time="2022-07-07T21:54:06+04:00" level=fatal msg="extracting data: open /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f-tmp/bin/containerd-shim-runc-v2: permission denied"

Did you get the k2s-selinux package installed successfully? It looks like selinux is denying some very basic startup operations.

yes, k3s-selinux package is installed
[max@ol9 ~]$ rpm -qa | grep k3s
k3s-selinux-1.2-1.el8.noarch

You might just run k3s-uninstall.sh and then reinstall now that you've got the selinux package installed. That's probably easier than trying to manually relabel everything with chcon.


Error message " Cannot read property 'checked' of null"

I am getting data from a web service and trying to bind those to ngx-treeview, however I was able to loop trough the list and mapped to TreeViewItem[]. now all the data are displaying properly and check buttons also working ,
but at the time page loading, im getting above error.
I think thats happening because I have assigned data within the service subscriber but I have no idea how to correct, my code is bellow, if anyone can help me, it would be very much appreciate.
   public getDataList() {         this._mydataService.getAllVehicleDetails().subscribe(data => {             this.items = this.loadTreeviewItems(data)         }, error => { this.handleErrorResult(error) });
and the loop is bellow
GenerateChildItems(child) {         const sublevel: any[] = [];         var childrens: TreeviewItem[];         console.log("1")         child.forEach((childObj) => {             if (childObj.subItems != undefined) {                 if (childObj.subItems.length > 0) {                     childrens = this.GenerateChildItems(childObj.subItems);                 }             }             sublevel.push(new TreeviewItem({                 text: childObj.title,                 value: { id: childObj.id, imagePath: childObj.image },                 checked: true,                 children: childrens             }))             this.myItemList.push({ id: childObj.id, path: childObj.icon })         });         return sublevel;     }     public loadTreeviewItems(result): TreeviewItem[] {         const returnObj: TreeviewItem[] = [];         if (result != null || result != undefined) {             result.forEach((mainObj) => {                 console.log("mainObj", mainObj)                 if (mainObj.subItems.length > 0) {                     var childItems = this.GenerateChildItems(mainObj.subItems);                     const mydata = new TreeviewItem({                         text: mainObj.title,                         value: { id: mainObj.id, imagePath: mainObj.image },                         checked: false,                         children: childItems                     });                     this.myItemList.push({ id: mainObj.id, path: mainObj.icon })                     returnObj.push(mydata)                 }             });         }         return returnObj;     }

Please download my demo => setup your data & try, if getting error, please push code here. Thanks.

Hi Leovo,
your demo is working perfectly, its occurred after i bind data from my servicec

How could I help you? I can't find root cause without your example.

Hi Levo,
I re build treeview according to your demo, somehow error was gone, may be a my mistake, Im sorry
and thank you

lol


second layer data pack

I tried to use the example second layer data pack and it kinda works. I can choose a second origin but i can only use one power. for example i chose phantom and enderman and could only use the pearls. Is there something i have to add or change in the datapack like with a secondary power or something?
Thanks for reading

Closing this because you opened the same issue on the example pack repository, where it is more relevant.


Experiment with improving the parallelization of the layout optimizer

It may be possible to improve the parallelizability of optimize_layout without changing the results from the single-threaded case. This is done by precomputing a "graph" of dependencies between observations and spinning off threads if we observe a series of observations that don't depend on each other.

Iterate across observations within each epoch as usual. At each observation, find its neighbors and also hit the RNG to get the negative samples. The combined set of neighbors and negative samples defines the "dependencies".
Check if the dependencies depend on any other queued observations in other threads.

If yes and it's just one thread, add it to the task queue for that thread.
If yes and it's multiple threads, close the task queues for all threads. Put this observation on hold.
If not, add it to the task queue of the thread with the shortest task queue.

If all task queues are closed, actually execute the compute by spinning up one thread per non-empty task queue.
Restart this process with the on-hold observation, or finish if no observation was held.

Hopefully, the task queues build up to a healthy size and we don't have to spin up too many threads. At around 10 us per thread and 500 epochs, we'd be aiming for ~100 start/stops to keep the thread overhead under 10 seconds.
Note that this also necessitates holding onto the identities of the negative samples. This shouldn't take a lot of memory, but it may be wise to set a buffer size (e.g., 10 million negative samples) that closes all task queues and triggers execution.

It is done: e98d91118ca2ce1f679cdd616b140e228b3faa51 (#14). Code is not for the faint-hearted.
tl;dr it probably wasn't worth it.
The final implementation differs quite a bit from the above description. In the merged version, we iterate across observations at usual. For each observation, we check if it, its neighbors, or its negative samples were accessed in previous observations. If not, we send the task of updating that observation's embeddings (and the associated updates to its neighbors) to a waiting thread. This is repeated for the next observations until all threads are occupied or we get an observation that needs to access coordinates that are currently being updated by an active thread. We then wait for all active threads to finish, reset the access logs, and repeat this process until we run out of observations.
I distribute update tasks for individual observations rather than building a queue of tasks because the chance of conflicting accesses is too high with multiple observations. It's basically the birthday problem - if you take even a small group of observations, you'll probably run into at least one conflict, which requires us to stop all new update tasks until the existing threads finish their jobs. By distributing individual observations, we reduce the frequency of conflicts between threads. This frequency is further reduced when you have larger dataset because the graph becomes increasingly sparse, so it's unlikely that multiple threads will be working on the same things at the same time.
Having said that, the need to distribute tasks for individual observations means that we need to make inter-thread communication as fast as possible. Obviously this requires persistent threads as I'm not going to spin up a new thread per observation. Unfortunately, mutexes don't cut it for inter-thread synchronization; there's too much stalling when we're distributing millions of jobs. Instead, we use a busy-wait approach where each thread polls an atomic bool to check whether it can start working (or if it's finished). This is very fast and allows threads to pick up jobs as soon as they are distributed. The downside is that (i) we get CPU usage of 100% for each thread, which is pretty wasteful w.r.t. energy consumption if they're not actually doing anything, and (ii) the user better have enough available CPUs, as any thread contention will destroy performance when the busy-waits start interfering with actual compute.
So, with that design (and after resolving the many race conditions that keep on popping up), we get the following results. I'll use the yaumap package for testing on real data, specifically the MNIST digits dataset from snedata. To keep things simple, I already computed the nearest neighbors so that we can see the impact of parallelization on the layout optimization specifically, instead of getting swamped by the runtime of NN detection.
res <- readRDS("nn.rds")
library(yaumap)
system.time(out <- umap_from_neighbors(res$index, res$distance, num_threads=4, num_epochs=200))
##    user  system elapsed
##  27.076   0.019  24.238

system.time(out2 <- umap_from_neighbors(res$index, res$distance, num_threads=4, parallel_optimization=TRUE, num_epochs=200))
##    user  system elapsed
##  69.423   0.021  17.416

identical(out, out2)
## [1] TRUE

Same results, but ~30% faster, hooray. Keep in mind that this is with 400% CPU usage due to the busy-waits, so you can see it's not very cost-effective. But it is at least faster, so if cost is not a concern, and you have the cores...
The biggest bottleneck in the parallel version is the pseudo-random number generation. This step is done only in the main thread; the negative samples are then stored for distribution to each worker thread. I reckon it's responsible for about 10 seconds of the runtime, which is reasonable given that we're generating around a billion random numbers (200 epochs, 70,000 observations, 15 neighbors, about 5 negative samples on average = 1.05 billion numbers). Performance could definitely be improved by switching to a better PRNG and/or discrete sampling function, e.g., replacing MT with PCG32 and replacing my aarand::discrete_uniform with GCC's no-modulo method. If you're particularly wild, you could even create one RNG stream per observation and then you could sample within the worker thread; this would require PRNGs with a small state (i.e., not MT), and you would need to accept different results from single-threaded runs.
An interesting tidbit is that the RNG is not a major contributor to runtime in the serial version. Replacing the RNG with pre-specified numbers only exhibits modest decreases in runtime (<0.5 seconds) in the single-threaded code. I speculate that this is because the compiler reorders some operations in the update iterations across observations; specifically, I'm guessing that it kicks off the instructions to do the random number sampling for the next iteration while the updates are being performed for the current iteration, running them in a kind-of-parallel way and eliminating the cost of the RNG.
Tagging @jlmelville, who might be interested in this story given our adventures in jlmelville/uwot#83.

I am slightly in awe of the heroics that have gone into making this scheme work. I knew when I read the daunting initial description that this isn't something I would have conceived of or been able to implement in a way that was competitive with the standard asynchronous approach, let alone be faster.
@LTLA how much of the decisions are fundamental to the inter-thread communications versus compiler or architecture-specific performance around e.g. mutexes vs busy-waiting? From your write-up you seem ambivalent about the outcome: is this a dead end (but presumably an intellectually satisfying accomplishment) or something that could be made more CPU-efficient with e.g. more exotic threading constructs?

how much of the decisions are fundamental to the inter-thread communications versus compiler or architecture-specific performance around e.g. mutexes vs busy-waiting?

Hm. I did most of my development on an old Intel Mac with clang during the day, and kept on going on my slightly-less-old Ubuntu machine running GCC in the evening. While the absolute timings obviously differ across machines, the relative difference between the various strategies within the same machine are pretty consistent (IIRC). So it may indeed be something more fundamental, though perhaps things might be different on a non-Intel architecture.

is this a dead end (but presumably an intellectually satisfying accomplishment) or something that could be made more CPU-efficient with e.g. more exotic threading constructs?

Actually, I don't believe the threading is the issue. If I disable the threads and don't distribute any jobs to any workers at all, the umap_from_neighbors() call still takes 15-16 seconds! At this point, the only thing it's doing during the iterations is sampling random numbers, plus some checks for conflicting accesses. So, even the best threading approach would just save us 1-2 seconds; hence my focus on the PRNG performance in my above essay.
With a sufficiently fast PRNG, it should be possible to distribute jobs to workers quicker, and then we'd probably get a speed-up that matches up better with CPU usage. The PCG site suggests a few PRNGs that are more than twice as fast as the 64-bit MT I'm using, so that could be a good place to start. I guess I could allow people to template in an alternative PRNG themselves; I'm a bit reluctant to stray outside of the standard library for default uses of umappp.
If we are willing to change the algorithm a little, it is possible to parallelize the RN generation across observations by using PRNGs that support multiple streams. Each observation would get its own RNG instance at a different stream; each worker would then be asked to generate negative samples; control would return back to the serial section to check for conflicting accesses; and then the update tasks would be distributed for observations without any conflicts. This adds an extra round of inter-thread communication to shift RN generation into each worker, and should greatly mitigate the associated bottleneck in the serial section that is present in the current implementation.
Of course, that's even more complicated than the current version. And the complexity of the code is already a major problem - these damn race conditions are extremely painful to resolve. It's for this reason that I wouldn't recommend implementing this in uwot - the benefits are too modest to justify the loss of hair.


sha512 hash is not the same for the windows exe

My bad, it is a typo, it's supposed to say SHA256, I shall fix it.


GVRBitmapTexture javadoc update

The previous pattern of calling recycle() immediately after construction
is actually incorrect now and would lead to black textures or other
unexpected results.

do we have any samples that use this and call recycle() afterwards?  if so, we should also change them.

Found only one: gvrsixaxissensortest_v1_5_0. Will make a pull request for it. It is also not a real sample and to me another candidate for either removal and to be moved under Framework/Tools/.


pipenv: patch pipenv to point to python that has virtualenv

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @kaali

This could also be fixed by creating a python.withPackages with all runtimeDeps, and then patching the shebangs to point to that Python version.

Fixes #71771 and #71178 with pipenv

So it's OK now?


Electron reports error when running sharp

Possible install-time or require-time problem

[x] I have read the documentation relating to installation.
[x] I have ensured that the architecture and platform of Node.js used for npm install is the same as the architecture and platform of Node.js used at runtime.

Are you using the latest version of sharp?

[x] I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.
If you are using another package which depends on a version of sharp that is not the latest, please open an issue against that package instead.
Is this a problem with filesystem permissions?
If you are using npm v6 or earlier and installing as a root or sudo user, have you tried with the npm install --unsafe-perm flag?
If you are using npm v7 or later, does the user running npm install own the directory it is run in?
If you are using the ignore-scripts feature of npm, have you tried with the npm install --ignore-scripts=false flag?
What is the complete output of running npm install --verbose --foreground-scripts sharp in an empty directory?

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

I installed Sharp in Electron and got an error after running it under the windows system, as shown below.
The error is only reported in Windows system, but it is normal on Mac.

this is repo: https://github.com/dylan-7/sharp
electron/main/index.ts
import sharp from 'sharp'
ipcMain.handle('compress', async (_, arg) => {
  try {
    const base64 = arg.replace(/^data:image\/[jpeg|jpg]+;base64,/, '')
    const result = await sharp(Buffer.from(base64, 'base64')).jpeg().toBuffer()
    return `data:image/jpeg;base64,${result.toString('base64')}`
  } catch (e: any) {
    console.log(e)
    return e
  }
})

thank you for any help

Given you're using electron-builder, are you using asar / asarUnpack? Did you see https://github.com/lovell/sharp/issues/678#issuecomment-275956028 ?

Given you're using electron-builder, are you using asar / asarUnpack? Did you see #678 (comment) ?

I configured asar, but still got an error. You can look at the code https://github.com/dylan-7/sharp/blob/main/electron-builder.json5
  "asar": true,
  asarUnpack: ["**/node_modules/sharp/**/*"],

I have also encountered this problem. Do you have a solution

I have the same problem using electron-forge to package a windows.squirrel install.
I am sure it is to do with the way that electron forge decides to package the sharp module.
I have tried both using nm and yarn. The app runs fine in dev using electron-forge start.
I have searched the documentation for some hints on how to setup a basic electron App using Sharp on a windows platform. I have found lots of people with the same issue, but no examples of a solution.
I am happing to provide a git example if I could get some guidance on what might be required.
Best regards
Jeorme

@dylan-7 Were you able to make any progress with this?

v0.33.0 is now available and relies entirely on your choice of package manager for installation, so this class of error goes away. There may of course still be problems, but the first place to look if someone does go wrong is the package manager logic.

@philkunz Please upgrade npm; you'll need at least v9.6.5.
https://github.com/lovell/sharp/blob/4d049ee8f54f12faf452492217db981ed26d086e/npm/darwin-arm64/package.json#L33

@lovell I'm at 9.6.7


Does not support saves.

Some games (e.g. Pokemon) allow the player to save their progress and continue at a later time. Unfortunately, gbemu does not seem to remember these states.
How to reproduce

Start a game that supports saves (e.g. any Pokemon game)
Save game
"Restart" (in my case, I closed the tab, opened a new tab, loaded the same rom again)
On the "new game"/"resume game" screen there is no option to resume the game

How to fix
I think this is due to the fact that saves are not actually saved within the ROM but in an additional memory region (most/all emulators generate an additional file for each save). Maybe, each time the game is saved, the backend could create this save file (associated to the hash of the rom). Whenever the user starts the same rom later on load the associated save.
I dont know much about the technical things in the background of emulators, so I dont know how much effort this would be.

Thanks for the suggestion. A simple button to save SRAM data to localstorage might be doable in a couple days. I'll try to add it next weekend.

it could be very useful also to be able to export and import the .sav files


Configuration file format too old

When I run the docker and see its log, it showed:
https://pastebin.com/fpYsvWz9
It's a warning, but then "stopping the Postfix mail system".  Not sure if it is normal or not.  Self-test is passed, but test mails I sent seems never to be delivered.

It seems that there's some problem with your environment.
I do not understand what happened, maybe other users could give some information about this?

https://forum.artixlinux.org/index.php/topic,793.0.html
No real affect, just eliminate the warning messages.  SMF worked fine now.

Glad to hear that!


Complete 1.1.0 release

Template from gist https://gist.github.com/milutz/7712c37320ec062291abe435ac6bf0d5

[x] Create a branch for the release, typiclly that should be in the format "X.Y" X=major, Y=minor - patch shouldn't be in the branch name (multiple patch releases might come off the same release branch)

git checkout -b 1.1
git push -u origin 1.1

[x] Make sure ES software defaults are set to the desired ES release (for the compose install)

grep ELASTIC_TAG docker-compose.yml
grep 'ARG ELASTIC_TAG' plugin/Dockerfile
grep version plugin/package.json

[x] Build the kibana plugin builder for that ES version

git clone https://github.com/samtecspg/kibana-plugin-build.git
cd kibana-plugin-build.git
./build-by-version.sh 6.2.1

[x] Switch back to Conveyor, specifically the plugin directory and run the builder container there

cd ../conveyor/plugin
docker run --rm -it -v `pwd`:/kibana/plugin samtecspg/kibana-plugin-dev:6.2.1 bash

[x] (in the build container) run the build and exit the container

cd plugin
yarn install
yarn build
exit

[x] Update the project docker-compose.yml to pull the images for this release

cd ..
sed -i -- "s/repo-head/1.1.0/" docker-compose.yml

[x] Get the changes checked back in then tag to kick off a build

git add docker-compose.yml plugin/build/conveyor-6.2.1.zip
git commit -m "Updates for 1.1.0 release"
git tag -a v1.1.0 -m "Release tag for 1.1.0 release"
git push
git push origin v1.1.0

[x] (after docker cloud is done with the build) do a compose up and test

docker-compose up

[x] build the docker-compose zip file

./package.sh

Abandoning 1.1.0 release because of bugs

[ ] Make a new release on github off the release tag, upload zip made by the packaging script
[ ] In master update the readme to point to the new release
[ ] Update SPG Conveyer website to point at new release

@wrathagom Best method to template the release process was via a Gist

Because of "first flow create can cause kibana index issue" problem, 1.1.0 is not being completed.   Moving on to 1.1.1 release:

[ ] Get the changes checked back in then tag to kick off a build

git tag -a v1.1.1 -m "Release tag for 1.1.1 release"
git push
git push origin v1.1.1

[ ] (after docker cloud is done with the build) do a compose up and test

docker-compose up

[ ] build the docker-compose zip file

./package.sh

[ ] Make a new release on github off the release tag, upload zip made by the packaging script
[ ] In master update the readme to point to the new release

git checkout master
git pull
vi README.md
vi docs/_getting_started/basic-installation.md
git add README.md docs/_getting_started/basic-installation.md
git commit -m "Updating readme & docs to point and newest release"

[ ] Update SPG Conveyer website to point at new release

Closing (without website update) because moving on to 1.2.0 release


Nuxt 3. Failed to resolve component. Hydration node mismatch

Check that this is really a bug

[X] I confirm

Reproduction link
https://stackblitz.com/edit/nuxt-starter-ctjmny?file=app.vue
Bug description
I tried to use new Swiper Element API(WebComponents) in Nuxt 3 project and got Vue warnings in terminal/console. Everything is working so, but I don't think that it is a normal behaviour.

Expected Behavior
No warnings
Actual Behavior
Warnings
Swiper version
9.1.1
Platform/Target and Browser Versions
macOS Chrome 111
Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
[ ] Make sure this is a Swiper issue and not a framework-specific issue

Would you like to open a PR for this bug?

[ ] I'm willing to open a PR

I'm facing the same issue. I'm using Quasar CLI with Vite + Vue 3. Any solution to get rid of this warning?

Error message clearly states what needs to be done


array operator add clone function (#4749)

What type of PR is this：

[ ] bug
[ ] feature
[ ] enhancement
[ ] others

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

run starrocks_fe_unittest

run all


Extension issue

Issue Type: Bug
Extension Name: python
Extension Version: 2020.9.114305
OS Version: Windows_NT x64 10.0.19041
VSCode version: 1.50.1

:warning: We have written the needed data into your clipboard. Please paste! :warning:

Thanks for the bug report, but unfortunately we need some more information from you in order to triage this issue appropriately. Please edit your opening comment by providing a filled-in copy of our issue template. Once you are done please leave a comment to that effect.


fix contractor currentPeriod tracking bug

This PR fixes a bug in the contractor, where the allowance period was not being accurately tracked.

I've also noticed that something wasn't right there today (I was trying to tackle #2308) and I said it on #core-dev. I was waiting for an answer to code it because I'm not sure about the expected behaviour.
I thought it was more like:
if c.currentPeriod == 0 {
    c.currentPeriod = c.blockHeight
}

plus a few more lines to stop c.ProcessConsensusChange(...) to move it from 0.

merge conflicts after I merged the other PR

fixed up the merge conflicts

@avahowell can you update this? Would like to see this merged into RC2.

@DavidVorick updated


Frequency_vectorizer added

I fixed it and committed the changes. Thanks for your review and catching the mistakes.

Okay, I see, thanks. Let's not merge our own PR and not commit directly on main in the future.


chore: upgrade vue

Codecov Report

Merging #2400 into dev will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff          @@
##            dev   #2400   +/-   ##
====================================
  Coverage   100%    100%
====================================
  Files        20      20
  Lines      1144    1144
====================================
  Hits       1144    1144

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ec8f14...7e956c8. Read the comment docs.


Close returns an error instead of crashing if the server is not running

Hi Brian. Thanks for your PR.

For sure!


Improve search results

I'm having a hard time improving the Elastic results. I use simple_query_string (#17) when the user has a special character to enable powerful search, and I use multi_match with phrase_prefix as a default.
{
  query: {
    bool: {
      must: [
        {
          multi_match: {
            fields: ["text", "title", "description", "name"],
            type: "phrase_prefix",
            operator: "OR",
            query: searchTerm,
          },
        },
      ],
    },
  },
}

The results are okay for on word queries, but when multiple words are used in one query, it doesn't seem to use the OR operator correctly, resulting in fewer hits for sentences, instead of more.
My reasoning for the above query:

Multi_match because I want to search across multiple fields.
phrase_prefix because I want to hit "watertekort" when I search for "water"
OR operator because it should use the OR operator between words.
The bool_prefix type seems reasonable, but we're running 7.0 and that doesn't support it.

You might want to try or instead of OR? ;-) (since apparently it is still using and)
Also boosting could also be used to improve search resuts, Ie title^4 to give more importance to matches in titles etc.

Thanks @breyten!
or vs OR give the same results. The problem had to do with phrase_prefix, and it disappeared when I switched to the best_fields type.
However, now the "water" "watertekort" auto complete prefix thingy doesn't work. Maybe something with N-grams / Compound words?


S3FileInfo Exists with space in the key name

I have tried every encoding I can think of for the key name and Exists always returns false. I have tried the following:
Leaving spaces un-encoded
Replacing with %20
Replacing with +
Replacing with %2B
In each case S3FileInfo.Exists returns false. I am using v<IP_ADDRESS>.
Here is some sample code:
public Stream GetFileContents(IAmazonS3 s3Client, string bucketName, string key) {
    var fileInfo = new S3FileInfo(s3Client, bucketName, key);
    if (fileInfo.Exists) {
         return fileInfo.OpenRead();
    }

    return null;
}

I'm taking a look

I haven't been able reproduce the error. Here is the test code I wrote but it always passes.
public void FileWithSpacesInName()
{
    var key = "The Space.txt";
    Client.PutObject(new PutObjectRequest
        {
            BucketName = bucketName,
            Key = key,
            ContentBody = "Test Data"
        });

    // Wait for eventual consistency of S3 before checking if the object exists
    Thread.Sleep(TimeSpan.FromSeconds(2));

    S3FileInfo fileInfo = new S3FileInfo(Client, bucketName, key);
    Assert.IsTrue(fileInfo.Exists);
}

I'm wondering if the encoding issue happened when you put the object in S3. Can you give some details on how the object was uploaded?

I swear this didn't work yesterday. Now it works great. I know you walked over to the S3 team and had them fix this issue. Thanks for looking and sorry to waste your time.


Role of 'time_steps'

Hi Qiuqiang,
def train(args):
    model_type = args.model_type

    time_steps = 10
    freq_bins = 128
    classes_num = 527

   # Hyper parameters
    hidden_units = 1024
    drop_rate = 0.5
    batch_size = 500

Can you please explain the role of time_steps in the code? I guess it is present because the Google audioset dataset comprises of 128 dimensional embedding sampled at 1Hz and each audio sample is approximately of 10 seconds length, so that would account for 10 time_steps.
Am I on right track ? If yes, then how do you account for the audio clips that are less than 10 seconds in duration (say 6 seconds or so)?
Thanks in advance.

Hi Sourav,
You are right. The time_steps=10 comes from 10 seconds audio clip. For those audio clips shorter than 10 seconds, they are padded with silence to 10 seconds.

Best wishes,
Qiuqiang

From: Sourav Sahoo<EMAIL_ADDRESS>Sent: 28 May 2019 06:24
To: qiuqiangkong/audioset_classification
Cc: Subscribed
Subject: [qiuqiangkong/audioset_classification] Role of 'time_steps' (#5)
Hi Qiuqiang,
def train(args):
model_type = args.model_type
time_steps = 10
freq_bins = 128
classes_num = 527

Hyper parameters
hidden_units = 1024
drop_rate = 0.5
batch_size = 500

Can you please explain the role of time_steps in the code? I guess it is present because the Google audioset dataset comprises of 128 dimensional embedding sampled at 1Hz and each audio sample is approximately of 10 seconds length, so that would account for 10 time_steps.
Am I on right track ? If yes, then how do you account for the audio clips that are less than 10 seconds in duration (say 6 seconds or so)?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqiuqiangkong%2Faudioset_classification%2Fissues%2F5%3Femail_source%3Dnotifications%26email_token%3DADFXTSO7YSNG45W7XEZZIVLPXS6XXA5CNFSM4HP72SLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GWEIEOQ&data=02|01|q.kong%40surrey.ac.uk|8d3e0d3ab6bd4e30e78108d6e32cb7e7|6b902693107440aa9e21d89446a2ebb5|0|0|636946178530839878&sdata=lPV3zxFoGNDJSommVM4GezwsHt14B4EcSGb3DwDmfsY%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADFXTSIW6LFP55S5YCJVX33PXS6XXANCNFSM4HP72SLA&data=02|01|q.kong%40surrey.ac.uk|8d3e0d3ab6bd4e30e78108d6e32cb7e7|6b902693107440aa9e21d89446a2ebb5|0|0|636946178530849873&sdata=aeQU4RQ1Z5Td6dd%2FuDPrlj72RmC2RtqBldSIb8F5oj0%3D&reserved=0.


oneshot prune fails for a model

Hi author,
When I applied oneshot prune to a  model, following message rose up:
File "/usr/local/lib/python3.6/dist-packages/TinyNeuralNetwork-<IP_ADDRESS>30112164819+393a92189ec1ec6f65b6f17995674bc624bb4d47-py3.6.egg/tinynn/prune/oneshot_pruner.py", line 125, in prune
self.register_mask()
File "/usr/local/lib/python3.6/dist-packages/TinyNeuralNetwork-<IP_ADDRESS>30112164819+393a92189ec1ec6f65b6f17995674bc624bb4d47-py3.6.egg/tinynn/prune/oneshot_pruner.py", line 145, in register_mask
sub_graph.calc_prune_idx(importance, self.sparsity)
File "/usr/local/lib/python3.6/dist-packages/TinyNeuralNetwork-<IP_ADDRESS>30112164819+393a92189ec1ec6f65b6f17995674bc624bb4d47-py3.6.egg/tinynn/graph/modifier.py", line 2845, in calc_prune_idx
assert len(leaf.dim_changes_info.constraints_i) == 1, leaf.unique_name()
AssertionError: output_0_f
The len was 2 instead of 1 when errors happened. Also, I could not find 'output_0_f' in the model python script. Appreciate your time, thanks.

https://github.com/alibaba/TinyNeuralNetwork/commit/f3edbbff9ce5b51a3039d0f6545b134b0067435d
https://github.com/alibaba/TinyNeuralNetwork/commit/c72ae04da2d09465bbf321233e4746fb125f1d7c
bug fixed


How to setup in Railway?

:question: Question

Hi there, I've trying to setup a private server on railway's website through a Dockerfile. Here is it:
FROM binwiederhier/ntfy
EXPOSE 80
EXPOSE 443
COPY server.yml /etc/ntfy/server.yml
ENTRYPOINT ["ntfy", "serve"]

And here is my server.yml:
base-url: "https://ntfy-server-production.up.railway.app"

Everything seems to be working when I send the deploy command through my terminal.

But when I try to access I'm getting a server error.

Any suggestions on what I'm missing?

Trying by curl also happens to give the same error
curl -s https://ntfy-server-production.up.railway.app/test/json

I found the solution: I must to provide the port environment variable in the project's settings on Railway's dashboard
Thanks anyway.

If you'd like, you could provide an example with screenshots for the examples page: https://docs.ntfy.sh/examples/

@binwiederhier, is a YouTube video valid one?
https://youtu.be/L49wM84ivzE

@p-samuel Awesome!! I'll add it to the examples page.

Wait, where did the video go? I watched it the other day and now it's gone.

@binwiederhier , sorry, I've uploaded again because the audio was too bad.
https://youtu.be/auJICXtxoNA


Log votes for very long unconfirmed elections

Have you tested this on the live net? Is it a reasonable amount of logging?

This is arbitrary value thats hsould be enough to discover possible issues.Most nodes never have such issues so it's hard to test in live network.


Unnecessary elaboration about not being assignable to type parameters

async function foo<T>(x: T): Promise<T> {
  let yaddable = await getXOrYadda(x);
  return yaddable;
}

interface Yadda {
  stuff: string,
  things: string,
}

declare function getXOrYadda<T>(x: T): T | Yadda;

Here, you'll get an error like
Type 'T | Yadda' is not assignable to type 'T'.
  'T' could be instantiated with an arbitrary type which could be unrelated to 'T | Yadda'.
    Type 'Yadda' is not assignable to type 'T'.
      'T' could be instantiated with an arbitrary type which could be unrelated to 'Yadda'.

Proposed
Either
Type 'T | Yadda' is not assignable to type 'T'.
    Type 'Yadda' is not assignable to type 'T'.
      'T' could be instantiated with an arbitrary type which could be unrelated to 'Yadda'.

or
Type 'T | Yadda' is not assignable to type 'T'.
  'T' could be instantiated with an arbitrary type which could be unrelated to 'T | Yadda'.

I created a pull request that makes the proposed change (#42952) but had one small question about the behavior in scenarios related to the one discussed here.


Mark current position in stack

Stack:

#155
#156
#154
#153
#152
#151
#150
#149 ⮜
#148
#147

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #156


Update to latest nvbench

Description

The changes involving spdlog and fmt have led to RAPIDS using a newer version of fmt while also avoiding using CPM when a conda package is preinstalled. Previously we were force downloading fmt for nvbench compatibility in cudf while using a separate version in spdlog via rmm. Now that we need to make these consistent, we need a version of nvbench that is compatible with the latest fmt. This PR bumps the nvbench hash past my nvbench PR that includes the necessary compatibility changes.
Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.
[x] The documentation is up to date with these changes.
[ ] The cmake-format.json is up to date with these changes.
[ ] I have added new files under rapids-cmake/

[ ] I have added include guards (include_guard(GLOBAL))
[ ] I have added the associated docs/ rst file and update the api.rst

/merge


ERROR: No matching distribution found for unifi-protect-backup

Unifi Protect Backup version: N/A
Unifi Protect version: 2.6.17
Python version: Python 3.8.10
Operating System: Ubuntu 20.04.5 LTS
Are you using a docker container or native?: Native

Description
Following the README, I cannot install unifi-protect-backup via pip.
What I Did
$ pip install unifi-protect-backup
ERROR: Could not find a version that satisfies the requirement unifi-protect-backup (from versions: none)
ERROR: No matching distribution found for unifi-protect-backup

Never mind, python version was off.


Error message presented to the user gives no useful information when adding a tool with invalid url.

Electron build -  81ca1d7
Steps to reproduce the problem:

Chat with Tildy
Use "Add Tool from URL" option and add invalid url like github.com/trash.

User is presented with the following error message:

Expected Behavior:
Error message presented to the user should be more meaningful.

There is also another case where I see the same error message being presented to the user when uploading of knowledge file failed because of a DNS issue

Errors seen in console:
2024-09-04T19:48:05.825Z [server] [ERROR] Error: Command failed: /Users/sangeethahariharan/acorn/desktop/bin/knowledge ingest --prune --dataset nde6ei /Users/sangeethahariharan/Library/Application\ Support/acorn/Acorn/threads/nde6ei/knowledge
2024/09/04 12:47:45 INFO Created dataset id=nde6ei
2024/09/04 12:47:45 INFO Pruned files count=0 basePath="/Users/sangeethahariharan/Library/Application Support/acorn/Acorn/threads/nde6ei/knowledge"
2024/09/04 12:48:05 ERROR Failed to add documents error="couldn't add document '908f16fc-b8f4-4e43-936e-09cd5354ab6d': couldn't create embedding of document: couldn't send request: Post \"https://gateway-api.gptscript.ai/llm/embeddings\": dial tcp: lookup gateway-api.gptscript.ai on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:56756-><IP_ADDRESS>:53: i/o timeout"
2024/09/04 12:48:05 failed to add documents: couldn't add document '908f16fc-b8f4-4e43-936e-09cd5354ab6d': couldn't create embedding of document: couldn't send request: Post "https://gateway-api.gptscript.ai/llm/embeddings": dial tcp: lookup gateway-api.gptscript.ai on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:56756-><IP_ADDRESS>:53: i/o timeout

    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1105:16)
    at Socket.<anonymous> (node:internal/child_process:457:11)
    at Socket.emit (node:events:519:28)
    at Pipe.<anonymous> (node:net:338:12)
    at Pipe.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 1,
  killed: false,
  signal: null,
  cmd: '/Users/sangeethahariharan/acorn/desktop/bin/knowledge ingest --prune --dataset nde6ei /Users/sangeethahariharan/Library/Application\\ Support/acorn/Acorn/threads/nde6ei/knowledge',
  stdout: '',
  stderr: '2024/09/04 12:47:45 INFO Created dataset id=nde6ei\n' +
    '2024/09/04 12:47:45 INFO Pruned files count=0 basePath="/Users/sangeethahariharan/Library/Application Support/acorn/Acorn/threads/nde6ei/knowledge"\n' +
    `2024/09/04 12:48:05 ERROR Failed to add documents error="couldn't add document '908f16fc-b8f4-4e43-936e-09cd5354ab6d': couldn't create embedding of document: couldn't send request: Post \\"https://gateway-api.gptscript.ai/llm/embeddings\\": dial tcp: lookup gateway-api.gptscript.ai on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:56756-><IP_ADDRESS>:53: i/o timeout"\n` +
    `2024/09/04 12:48:05 failed to add documents: couldn't add document '908f16fc-b8f4-4e43-936e-09cd5354ab6d': couldn't create embedding of document: couldn't send request: Post "https://gateway-api.gptscript.ai/llm/embeddings": dial tcp: lookup gateway-api.gptscript.ai on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:56756-><IP_ADDRESS>:53: i/o timeout\n`,
  digest: '1602320271'
}

@cjellick - The first issue should be solved already with #483
@sangee2004 - Do you have steps to reproduce for the second issue? I've tried uploading files, and I'm not able to get that DNS Error

@ryanhopperlowe The issue with knowledge file upload happens randomly. I dont have reproducible steps to get to this state.

There is also another case where I see the same error message being presented to the user when uploading of knowledge file failed because of a DNS issue

completely separate bug from "Error message presented to the user gives no useful information when adding a tool with invalid url."
you can open a separate issue from the second one. the original issue should be addressed.

Logged https://github.com/gptscript-ai/desktop/issues/508 for tracking issue seen with error message not being shown to user when knowledge file addition fails.

Tested with electron build - 0f259cd7e
Proper error message is presented to the user.


build_library: Update qemu script to run on arm64 machines

I used a condensed indentation in the case statements to avoid the set commands from wrapping so much.  I can change it to match what was there before if you like.

Changed case to "${VM_BOARD}+$(uname -m)".
Added new patch 	build_library: Add missing die function.


Removing sync provider causes the plugin to throw an error about incorrect credentials on boot`

Describe the bug
If you delete a Github sync provider, then whenever you boot the plugin, it shows an error asking you to enter github credentials. You can't get directly past this error, you need to follow the prompt which takes you to settings, then go back out to the main token panel. This process also removes any existing token sets from the UI, and replaces them with a single empty set called global (which I think might be the default onboarding set).
The plugin also maintains the push/pull from github controls at the bottom despite the sync being removed.
To Reproduce
Steps to reproduce the behavior:

Add a Github sync provider
Delete that provider
Close and re-open the plugin

Expected behavior
It feels like deleting a Github sync (assuming no other sync providers are active as in it is the only sync provider users have) should let the plugin fall back to it's default behaviour of local storage without throwing any errors about credentials.
Screenshots or Screencasts

Is there a workaround for this?

@keeganedwin thank you that fixed it for me

Fixed in 1.38 🚢


react query v5 support

Is your feature request related to a problem? Please describe.
React Query v5 has been released: https://tanstack.com/query/latest/
Describe the solution you'd like
Any plans to support v5? :)
Describe alternatives you've considered
No response
Is your feature request related to a problem? Please describe.
No response

+1. In need of this for a massive project if we want to migrate to rq5.

+1 with SolidJS Support

duplicate https://github.com/dotansimha/graphql-code-generator-community/issues/318


Extend nested filter input-types to allow filter-depth configuration per relation.

Extend nested filter input-types (https://github.com/TriPSs/nestjs-query/pull/143) to allow filter-depth configuration per relation.
@FilterableRelation('relation', () => Relation, {
  // This one does not work yet
  filterDepth: 2,
})

Describe the solution you'd like
Requires further customization in the /nestjs-query/packages/query-graphql/src/types/query/filter.type.ts file. At Object.keys(filterableRelations).forEach((field) => { the filterDepth of the relation should be considered (only if set), not the filterDepth of the parent element.
// Example-A.entity.ts
@ObjectType('EntityA')
@FilterableRelation('relation', () => relation)
class EntityA { /* ... */ }

// leads to
// input EntityAFilter {
//    # should generate depending on the `filterDepth` of `EntityA`
//    relation: EntityAFilterRelationFilter
// }

// Example-B.entity.ts
@ObjectType('EntityB')
@FilterableRelation('relation', () => relation, { filterDepth: 2 )
class EntityB { /* ... */ }

// leads to
// input EntityBFilter {
//     # should generate a filter type of depth 2
//     relation: EntityBFilterRelationFilter
// }

This is not a critical feature for me personally, so I will look at this in the future when I have more resources available again.

any updates regarding this

Nope, sorry. Still not really a critical feature for me. But feel free to create a PR, if you need this feature 😄 @qashlan

@hedwiggggg  sure, will put sometime into this.


Faster geometry serialization

faster .geojson export
faster regular .json export with geometry columns
faster geometry ingestion

More to come later.

soda-fountain #350 SUCCESS
This pull request looks good


docs: update iOS documentation/recategorize

Just missed the 1.0 release for this, but here are some additions to the configuration doc and a bit of general nav cleanup to separate the platforms. Android quickstart coming soon.

:tada: This PR is included in version 1.0.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update get-networkcontrollerserviceinsertion.md

Added descriptions and example

@AnirbanPaul,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Not able to Delete ATWS Account Alerts

I only see the following cmdlets:

Get-ATWSAccountAlert
New-ATWSAccountAlert
Set-ATWSAccountAlert

I don't see a way to remove or overwrite an account alert despite the Autotask API supporting it. https://www.autotask.net/help/developerhelp/Content/APIs/SOAP/Entities/AccountAlertEntity.htm
Am I missing something?
Carter

I think this is something that has been added which is likely why it is missing from this powershell wrapper. Could this be added to the roadmap? This would save me having to write a function using the REST API which is where my comfort level is.

You are right. A rebuild of the module is long overdue. The SOAP API is supposedly in feature freeze, but support for removing accountalerts have definitely been added. I am Pestering the updated version tonight. Expect a release this week.

Wow I didn't expect a reply so quick, much less a new release this week!  I will eagerly await this release.
Thanks so much!

Module have been updated with a fix for this and updated to API version 1.6.11.


Issue running detect.py

Thanks for share to project.
I try to run the program and get an error:
'Hardswish' object has no attribute 'inplace'
under the procedure:
def fuseforward(self, x):
return self.act(self.conv(x))
I follow the steps and  install  through requirements.txt.
I was not able to find a solution through web search. Any idea or suggestion would be greatly appreciated.
Env:
windows 11,
Python 3.8
cuda-11.3.0
graphic card GeForce RTX 3090
Thanks in advance.

Also was having the same issue as above!
Please let me know if you have any tips here as I would love to run your model!
Env:
windows 10
Python 3.10
Cuda-11.3.0
Graphic card GeForce GTX 1080 x2

Hi everyone,
First of all, I apologize for the delay in answering to this issue. The people behind the original code of this repository have left the employment of the City of Montreal a while ago, and I am by no mean an expert on Python, even less so the Machine Learning libraries used in this project.
I have been able to reproduce the same issue that you all experience, so I can confirm there is indeed something wrong with the code. However, I was not able to find a fix for it.
My search on Github has led me to this open issue (with this exact stack trace) https://github.com/pytorch/pytorch/issues/46971 that seems to indicate that it would be a problem with the fact that the pre-trained models that we provide have a dependency on a specific older version of pytorch (through the use of the yolov5). I have tried every pytorch version between 1.6.0 and 1.11.0 in the requirements.txt and they all exhibit the same problem.
I know for a fact that this code ran successfully in the past, so it might only be a matter of getting the exact versions of the dependencies correct. Unfortunately, I don't have access to any working environment to get those version numbers. It makes me sad that the requirements.txt file doesn't specify strict version numbers as this would have prevented such an issue.
One potential solution would be to re-train the models with a current, up to date, version of pytorch, and then run the detect.py on it afterwards, but I have no idea how to do that.
I'm sorry that I can't be of more help. I will leave this issue open, in case somebody expands on my investigation and comes up with a working fix for it.


Steps to build a CSDMS Workbench component

Here are the steps to build a component for the CSDMS Workbench from a library containing a model or a tool.
To help explain these steps, consider a library foo packaged in a Python module foo that exposes a class Foo.

Wrap the library with a Basic Model Interface (BMI). For the case of the foo library, create a class BmiFoo and include in the foo module or in a separate bmi_foo module.

Learn more about BMI from its documentation
See examples of wrapping a simple model with a BMI in C, C++, Fortran, and Python
See examples of wrapping data sources with soilgrids, nwis, and gridmet

[Optional, but recommended] Create a conda-forge package for the BMI library. For the foo library, the conda-forge package could be foo or bmi-foo.

The conda-forge documentation on contributing packages is essential reading
Examples of conda-forge packages that wrap a model or data source with a BMI include hydrotrend, permamodel, gipl, and gridmet_bmi

Run the babelizer over the BMI library to create a component that can be imported into pymt. Babelizing the BmiFoo class would produce the pymt_foo module exposing the Foo class.

Learn more about the babelizer from its documentation, and about pymt from its documentation
Examples of components include pymt_hydrotrend, pymt_permamodel, pymt_gipl, and pymt_gridmet

[Optional, but recommended] Create a conda-forge package for the pymt component. For example, pymt_foo.

As mentioned above, the conda-forge documentation on contributing packages is essential reading
Examples of conda-forge packages of CSDMS Workbench components include pymt_hydrotrend, pymt_permamodel, and pymt_gipl

@mcflugen @gantian127 @gregtucker: Please feel free edit these steps as you see fit, and add any discussion as comments below!

Nice! How about including links for people to learn about the steps, e.g., (Learn about BMI) <= link to bmi.readthedocs.org, and (Learn how to create a conda-forge package) <= link to whatever pages on conda-forge, etc.

Nice! How about including links for people to learn about the steps, e.g., (Learn about BMI) <= link to bmi.readthedocs.org, and (Learn how to create a conda-forge package) <= link to whatever pages on conda-forge, etc.

@gregtucker Done! I've added links to docs and examples.

@gregtucker Done! I've added links to docs and examples.


Get it working with jQuery AJAX

Hi,
Great job on this! Feels much more solid than more established solutions like Chosen and Select2.
I'm having some issue with making it work with my jquery ajax request in WordPress.
I have a select element where I want it to fetch results via AJAX when user starts to search.
I've initiated the choices element on page load and then I have this:
choices_general.passedElement.addEventListener('search', function(event) {
			var nonce = $(choices_general.passedElement).data( 'nonce' );
			choices_general.ajax( function( callback ) {
				console.log('Do ajax');
				$.ajax({
					type : 'post',
					dataType : 'json',
					url : sk_search.ajax,
					data : {
						action: 'search_general',
						nonce: nonce,
						s: s,
					},
					success: function( response ) {
						console.log( response );
						//callback( response.data, 'value', 'label' );
					},
					error: function( response ) {
						console.log( response );
					}
				});

			});
		});

It makes it's way into the choices_general.ajax function, then jquery throws this error:
Uncaught RangeError: Maximum call stack size exceeded at Function.type (jquery.js?ver=1.12.4:2)
Seems it gets trapped in an infinite loop somehow?
Would be grateful for insight from people smarter than me!

I will answer myself in case anyone else comes here with similar questions.
I ended up reading up on promises and the fetch API and instead got it working with the "WordPress AJAX way".
Here's how I do it. Calling to an WordPress Ajax action and passing along a searchstring and a security nonce.
choices_general.passedElement.addEventListener('search', function(event) {
			var nonce = $(choices_general.passedElement).data( 'nonce' );
			var formData = new FormData();
			formData.append('action', 'search_general');
			formData.append('nonce', nonce);
			formData.append('s', 'test');

			choices_general.ajax( function( callback ) {
				fetch(sk_search.ajax, {
					method: 'post',
					body: formData
				})
		        .then(function( response ) {
					return response.json();
		        }).then(function( data ) {
					console.log( data );
					callback( data, 'value', 'label' );
				})
		        .catch(function( error ) {
		            console.log( error );
		        });

			});
		});


Update WorldAnchorManager and StabilizationPlaneModifier for Unity 2020

Overview

Compiles out calls to legacy XR.WSA APIs on 2020.1+, where the APIs no longer exist.
Adds support for XR SDK's SetFocusPlane API in StabilizationPlaneModifier, to add back the functionality when running on non-legacy XR.
Removed an unused assembly reference in the SDK .asmdef.

WorldAnchorManager could be converted to the XR{ReferencePoint|Anchor}Subsystem, but there's still some active work on the WMR implementation of that subsystem underway that prevents testing. Once that work is done, we can revisit this component.
Changes

Part of #8269

/azp run


Comparison between different stemming/generalisation approaches

Add a link to a table that shows how each generalisation method reduces tokens.

This could be shown at the bottom of the onboarding page

Resolved in f6d8abbe9c3af037f30b836257363579280b5e3a


Change icon for invalid color in ctkColorPickerButton

Button icon has no color if setColor is called with invalid QColor.

Thanks for the patch. Was it crashing before ?

No, there was no crash, but setting a QColor() applied a black icon. This is a normal behaviour (as default invalid color is 0,0,0), but I think setting an invalid color means no color at all.


Add gray-matter information to each example

Add the following information to each README.md in examples:

Title
Description
Tags
Demo URL
Deploy URL

This has been completed.


[BUG]: Cannot reliably use "programmed" name for function name, due to falling out of scope

Problem description
Sometimes it is necessary to create the name of functions and classes during module creation (e.g. for templated classes from C++).
In that case, the name attribute (const char *) can fall out of scope. The only way around it is to allocate memory, which then causes a memory leak.
In the example below, "mul" is OK, "add" fails when getting example.add.__name__ in Python, and "sub" is OK, but leaks memory.
Reproducible example code
#include <nanobind/nanobind.h>
#include <string>

int add(int a, int b) { return a + b; }
int sub(int a, int b) { return a - b; }
int mul(int a, int b) { return a * b; }

NB_MODULE(example, m) {

    m.def("mul", &mul, "Multiply");

    std::string add_name = std::string("add");
    m.def(add_name.c_str(), &add, "Add two integers");

    std::string * sub_name = new std::string("sub");
    m.def(sub_name->c_str(), &sub, "Subtract");
}

Tentative fix here (https://github.com/wjakob/nanobind/commit/3c2b64a0ccde7ff09c28dfe20af0d5de96c0f364), can you test if this fixes the issue on your end?

Yes, thanks, that looks good.

Thanks for confirming!


Add pointer and span versions

Purpose of this PR
The purpose of this PR is to address #8 by including pointer versions and automatically creating span and array wrappers which call the pointer versions.
How it works
This PR introduces a Roslyn source generator that looks at bare unsafe (i.e. taking T* arguments) P/Invoke definitions and automatically generates wrappers that take T[] and Span<T>.
Some resources for those not familiar with source generators:

Source Generator Cookbook
Video from JetBrains .NET days
Blog series by Andrew Lock

Why not incremental
Why did I go for a regular instead of an incremental source generator? Honestly that’s what I am familiar with (even though I posted resources about incremental ones as well). I guess for one-off generation they are good enough. If we expected people to use this generator in their own projects an incremental one would make a lot of sense though.
What do we generate
We generate

simple wrappers that swap T* with Span<T> and T[] (the latter should be superfluous due to automatic conversions but we don’t want to break API compatibility)
in addition versions that infer the length (except for Lapack where I was unsure about the result): works by finding a single int argument with a single character name (typically n or N) and removing it from the wrapper, inferring it from the first transformed argument’s length
in addition, only for Blas and Vml: add versions with iniX whenever we have an argument incX and X is a name of length 1 or 2 and is not A (which never had an iniA). This is also not needed anymore with Span<T> since you can just slice them, but we keep it for compatibility

Where do the pointer signatures come from
I used Regex search and replace to transform the existing ones. They are put into an Unsafe static nested class (which the generator picks up) and have their names aligned with the existing wrapped ones by using EntryPoint.
Testing
I added a test for the source generator itself and made sure the existing tests pass. Since the existing tests use the array versions which now call the pointer versions we have decent test coverage.
How does auto-generated code look like
Check e.g. MKL.NET/obj/$config/$tfm/$rid/generated/MKL.NET.WrapperGenerator/MKLNET.WrapperGenerator.WrapperGenerator/Blas.Wrappers.g.cs after building MKL.NET. Here is an excerpt

axpby wrappers
///Calls the MKL function <c>axpby</c> by pinning the given data during the computation.
[global::System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
public static void axpby(int N, float alpha, float[] X, int incX, float beta, float[] Y, int incY)
{
    unsafe
    {
        fixed(float* XPinned = X)
        fixed(float* YPinned = Y)
        {
            Unsafe.axpby( N,  alpha,  XPinned,  incX,  beta,  YPinned,  incY);
        }
    }
}
///<remarks>This version infers the length parameter <c>N</c> from <paramref name="X" />'s length.</remarks>
///Calls the MKL function <c>axpby</c> by pinning the given data during the computation.
[global::System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
public static void axpby(float alpha, float[] X, int incX, float beta, float[] Y, int incY)
{
    unsafe
    {
        fixed(float* XPinned = X)
        fixed(float* YPinned = Y)
        {
            Unsafe.axpby( X.Length,  alpha,  XPinned,  incX,  beta,  YPinned,  incY);
        }
    }
}
///<remarks>This version takes offsets in the form of <c>iniX</c> arguments. Consider using the Span version with slicing instead.</remarks>
///Calls the MKL function <c>axpby</c> by pinning the given data during the computation.
[global::System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
public static void axpby(int N, float alpha, float[] X, int iniX, int incX, float beta, float[] Y, int iniY, int incY)
{
    unsafe
    {
        fixed(float* XPinned = &X[iniX])
        fixed(float* YPinned = &Y[iniY])
        {
            Unsafe.axpby( N,  alpha,  XPinned,  incX,  beta,  YPinned,  incY);
        }
    }
}

///Calls the MKL function <c>axpby</c> by pinning the given data during the computation.
[global::System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
public static void axpby(int N, float alpha, global::System.Span<float> X, int incX, float beta, global::System.Span<float> Y, int incY)
{
    unsafe
    {
        fixed(float* XPinned = X)
        fixed(float* YPinned = Y)
        {
            Unsafe.axpby( N,  alpha,  XPinned,  incX,  beta,  YPinned,  incY);
        }
    }
}
///<remarks>This version infers the length parameter <c>N</c> from <paramref name="X" />'s length.</remarks>
///Calls the MKL function <c>axpby</c> by pinning the given data during the computation.
[global::System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
public static void axpby(float alpha, global::System.Span<float> X, int incX, float beta, global::System.Span<float> Y, int incY)
{
    unsafe
    {
        fixed(float* XPinned = X)
        fixed(float* YPinned = Y)
        {
            Unsafe.axpby( X.Length,  alpha,  XPinned,  incX,  beta,  YPinned,  incY);
        }
    }
}

Coverage of MKL functions
I covered Blas, Feast, Lapack and Vml. I did not cover

Functions not included in MKL itself
Solver: seems to have non trivial wrappers
VSL: same, and here it makes sense since the VSL API needs pinned pointers, so not sure how a good wrapper would look like
BLAS-like functions ?imatadd ?imatcopy: There seems to be problem with using the original definition with pointers. The original definition uses int for size_t, and at least on my machine using int and double* will lead to segfaults. I have since found out that (according to ClangSharpPInvokeGenerator) size_t should have been nuint (and that one works). Apparently the stub generated by the Jitter when encountering arrays to marshal is a bit more tolerant with mismatching types? I’m not sure to be honest. See possible improvements for how I would handle it in another PR.

Possible improvements

Check in generated code: See Andrew Lock’s blog post if needed. For now I added <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles> to the main project file. It will output the generated code but by default to the obj folder, so it won’t be checked in. The blog post describes how to change the path to check it in, if desired
We could add [SkipLocalsInit] on the wrappers. This would require C# 9.0 and a polyfill NuGet to add the attribute (which is only available in .NET 5+). I would suggest doing it in another PR to assess performance impact separately.
Adding tests for the Span<T> versions
Add some way to mark input-only arguments and render them ReadOnlySpan<T> instead of Span<T> for better usability (maybe we could use [In] for that). I shied away from this since it’s a lot of work to figure out which arguments are read-only. I would suggest doing this in separate PR
Include the BLAS-like functions with correct nuint (or UIntPtr) definitions in a separate BlasLike class so they can be auto-wrapped as well. We can have the existing int signatures delegate to the new wrappers then. I would suggest doing this in a separate PR

Did you consider using LibraryImportAttribute?

Did you consider using LibraryImportAttribute?

Hi @PhilPJL, I might be wrong but isn’t it .NET 7 only?

Ok fair enough, it is unfortunately.

I have had look at it some weeks ago when .NET 7 came out and for arrays it basically just pinned them and then called a pointer version. It also added [SkipLocalsInit]. So it’d be very similar but I don’t think it would allow us to do transformations like inferring length parameters.

By 'just pinning' do you mean this:
    static unsafe partial class MKLNativeMethods
    {
        [System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "<IP_ADDRESS>5")]
        [System.Runtime.CompilerServices.SkipLocalsInitAttribute]
        private static partial void vdMul(int n, global::System.ReadOnlySpan<double> a, global::System.ReadOnlySpan<double> b, global::System.Span<double> r)
        {
            // Pin - Pin data in preparation for calling the P/Invoke.
            fixed (void* __a_native = &global::System.Runtime.InteropServices.Marshalling.ReadOnlySpanMarshaller<double, double>.ManagedToUnmanagedIn.GetPinnableReference(a))
            fixed (void* __b_native = &global::System.Runtime.InteropServices.Marshalling.ReadOnlySpanMarshaller<double, double>.ManagedToUnmanagedIn.GetPinnableReference(b))
            fixed (void* __r_native = &global::System.Runtime.InteropServices.Marshalling.SpanMarshaller<double, double>.ManagedToUnmanagedIn.GetPinnableReference(r))
            {
                __PInvoke(n, (double*)__a_native, (double*)__b_native, (double*)__r_native);
            }

            // Local P/Invoke
            [System.Runtime.InteropServices.DllImportAttribute("mkl_rt.dll", EntryPoint = "vdMul", ExactSpelling = true)]
            [System.Runtime.InteropServices.UnmanagedCallConvAttribute(CallConvs = new System.Type[]{typeof(global::System.Runtime.CompilerServices.CallConvCdecl)})]
            static extern unsafe void __PInvoke(int n, double* a, double* b, double* r);
        }
    }

Which seems to be a bit more involved than fixed(float* XPinned = X) but if that works, great.
Did you handle methods with Complex values?  This is working ok for me
    [LibraryImport("mkl_rt.dll")]
    [UnmanagedCallConv(CallConvs = new [] { typeof(CallConvCdecl) })]
    private static partial void vzMul(int n, ReadOnlySpan<MKLComplex16> a, ReadOnlySpan<MKLComplex16> b, Span<MKLComplex16> r);

    public static void Multiply(ReadOnlySpan<Complex> a, ReadOnlySpan<Complex> b, Span<Complex> r)
    {
        Guard.IsGreaterThanOrEqualTo(b.Length, a.Length);
        Guard.IsGreaterThanOrEqualTo(r.Length, a.Length);

        var lhs = MemoryMarshal.Cast<Complex, MKLComplex16>(a);
        var rhs = MemoryMarshal.Cast<Complex, MKLComplex16>(b);
        var res = MemoryMarshal.Cast<Complex, MKLComplex16>(r);

        vzMul(lhs.Length, lhs, rhs, res);
    }

public struct MKLComplex16
{
    public double Real;
    public double Imag;
}

Yes, this looks like the code I remember from experimenting. You are right in that it introduces a whole new machinery of marshallers, but if you follow the code (e.g. via source.dot.net or decompiling) it ends up just fixing the span/array. However this allows you to customise marshalling if you need to (all these marshalling classes are .NET 7 only as well, mind you)
I haven’t tried complex numbers in MKL because I did not encounter them in this project. But in IPP (other Intel library) the same (struct with Re and Im) works. You can’t use the struct with stackalloc unfortunately but you can use Unsafe and MemoryMarshal:
Span<double> xs = stackalloc double[]{1,2,3,4};
var zs = MemoryMarshal.CreateSpan(ref Unsafe.As<double,MKLComplex16>(ref xs[0]), 2);
// zs is now a Span<MKLComplex16> with 1+2i, 3+4i

Regarding benchmarking, just an anecdote (I didn’t check vzMul): at work I tried benchmarking IPP and Span.Clear and Span.Fill are as performant as the corresponding IPP functions. Span doesn’t come with arithmetic operations, but using vector intrinsics (see e.g. LitMath) addition also becomes basically as fast as the IPP addition. It’s an interesting development, maybe we won’t need MKL/IPP for the simple stuff soon enough? 😄

Many thanks for this its really good. I'm going to merge this and do a bit of housekeeping on the code as it hasn't been done for a while.

Can I suggest, if you're not already doing this, make input params ReadOnlySpan.

Can I suggest, if you're not already doing this, make input params ReadOnlySpan.

This is already in the “possible improvements” list above. The reason why I haven’t done it yet is that determining whether a parameter is read-only seems to be a manual process
If you have a list of input parameters per routine or some way of inferring them, that would help me a lot

I'm no expert on MKL but it looks like there's a consistent naming pattern where 'a', 'b', 'n' are inputs and 'r' is the output.
  By default the mode is 'high accuracy' which is one explanation of the slow performance of complex multiply.  Changing that to 'performance' mode makes complex multiply run in the same time as a simple C# loop (unfortunately not better).
The mode can be globally changed if required.

I'm no expert on MKL but it looks like there's a consistent naming pattern where 'a', 'b', 'n' are inputs and 'r' is the output.

That seems to be the case for VML, but BLAS for example is not consistent. But const seems to be used correctly in the other header files as well, so I can try to use that information. Haven’t thought of looking there 👍
Re the performance: have you tried adding [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)] on the wrapper? Are you trying long vectors like 10,000 or 100,000 or rather small ones?

@PhilPJL: It could also be that your C# loop is vectorised by the JIT (RyuJIT with tiered compilation seems to be able to do that from what I’ve read). And I didn’t know that MKL supported Intel GPUs that’s cool. Of course there you'd have overhead with memory transfer/pinning.
@AnthonyLloyd: The suggestion by @PhilPJL looks promising, const is used on the header files for BLAS, LAPACK and VML. Doing some regex work to annotate all params const (double|float)* and make the generator emit ReadOnlySpan for them seems conservative to me. If you’re not in a hurry with the next release maybe I can adapt the signatures end of next week/week after that, as it would be a breaking change.

@delreluca sure sounds good.


example(toh-4,5): getHeroesSlowly() to return getHeroes()

Have getHeroesSlowly() delay and then return the value of getHeroes(). This makes it easier for user’s performing the tutorial to keep this slower method as they evolve toh-5 into toh-6.
@kwalrath @Foxandxss @filipesilva : note that I've partitioned the files.

I am good with it, but I am unsure if the new version is a bit harder to understand (the setTimeout changes).

I think that it separates concerns so that the setTimeout(...) does just that (and nothing more); otherwise it gets a bit convoluted to integrate the call to getHeroes inside the call to setTimeout (at least I couldn't find a clean way that worked).

@Foxandxss: can you tag as LGTM so that @wardbell can have a look at this too?


Write ADR for how static data will be stored

As discussed in the knowledge transfer meeting, there is a good amount of static data that will need to be transmitted to the frontend and loaded into memory. To keep this efficient, we will need to make use of ArrayBuffers/Typed Arrays, and it will most likely be a good idea to store them in a binary format that can be efficiently transferred to the frontend. This task is to identify how to best store and transfer this data, and will hopefully result in defining a shape or interface for all of this data.

Note: alternatively, if the first approach that's tried seems to work out well, it may be possible to bypass the ADR and go straight to implementation in an effort to save time.