cli: vitest '**/*.integration.js' does not behave as documented

Describe the bug
The CLI documentation for "include" states

CLI: vitest [...include], vitest **/*.test.js
A list of glob patterns that match your test files.

However, running vitest '**/*.integration.js' uses the pattern as a filter instead of an include:
> vitest --run '**/*.integration.js'

 RUN  v2.1.8 /Users/max/projects/...

filter:  **/*.integration.js
include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude:  **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*

No test files found, exiting with code 1

Setting a filter is documented as -t. There doesn't seem to be an explicit flag for overriding the default includes.
Reproduction
npm init --yes
npm i vitest -D
npx vitest --run '**/*.integration.js'

Observe that the given pattern is shown as "filter" instead of "include".
System Info
System:
    OS: macOS 15.1.1
    CPU: (8) arm64 Apple M1
    Memory: 269.09 MB / 16.00 GB
    Shell: 3.7.1 - /opt/homebrew/bin/fish
  Binaries:
    Node: 22.12.0 - /opt/homebrew/bin/node
    npm: 10.9.0 - /opt/homebrew/bin/npm
  Browsers:
    Chrome: 131.0.6778.86
    Safari: 18.1.1
  npmPackages:
    vitest: ^2.1.8 => 2.1.8

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

vitest '**/*.integration.js' is not what documentation says. Documentation shows vitest **/*.integration.js. The globbing should be done by your CLI

Sorry for not the late reaction. It's still not working as advertised if I pass specific filenames, because it treats them as filters, not as includes.
❯ npx vitest --run test.integration.js

 RUN  v2.1.8 ...

filter:  test.integration.js
include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude:  **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*

No test files found, exiting with code 1

I can't fine any other way to change include.

You cannot change include via a CLI, you can only filter included tests. Docs just mention the usage for people who are looking to filter tests


Support bring your own network resources

VPC id
Subnet ids
Security Group

Hello
This is good news that you are supporting to bring in our own VPC, subnets. We also have restrictions in creating new IAM roles and Policies, and using existing AMI ids for jump/nfs VMs when we create this cluster. Would you be able to include support to bring our own IAM roles/policies and AMI ids (for NFS and Jump)  in this script. Thanks

@gnan-s , one can bring their own IAM policy as long as it contains the items listed here. The code does currently add a policy for EBS CSI drivers on the EKS worker nodes that is required. If that's not allowed, then at this time we do not support bring your own IAM. Currently the jump/nfs vm's are controlled and we don't allow folks to change that. These are Ubuntu images. We have cloud-init scripts written for these images that add the needed software to stand up a viable eco-system for SAS Viya 4 consistently every time.  You can open separate issue and we can investigate. Create an issue for each item don't stack requirements into a single issue.


Cannot install WDK extension

It says my currently installed applications are unsupported. I have Visual Studio 2019. It should be supported by now since its the stable version. Is there a workaround or I need to remove my VS 2019 and go back to VS 2017?

Hi @Furqan136,
Yes, VS 2019 is stable and works well, but there are some braking changes in the IDE in regard to working with the WDK. VS 2019 works well with the WDK for Windows 10 version 1903 (releasing soon) but not with earlier versions. See the blog post, Windows Driver Kit (WDK) to support Visual Studio 2019 for some more information.
Eliot


Order of injection

Hi all,
Thanks for a great plugin. Makes putting things in the head a breeze.
I'm using this plugin to add in SEO items. The problem is, these appear below the inline css, so many social media providers stop reading halfway through the css and the social cards don't get rendered.
Is it possible to specify where the meta tags get rendered - ie, can you say, put the items at the top of the head?
Many thanks,
Pete.

Hi Pete, this might be useful: https://github.com/staylor/react-helmet-async#prioritizing-tags-for-seo

The problem is that this isn't done by this plugin. For this to work, it needs to implement:
${helmet.priority.toString()}

In gatsby-ssr.js.

The problem is that this isn't supported by this plugin. For this to work, it needs to implement helmet.priority.toString() in gatsby-ssr.js here. :
helmet.base.toComponent(),
helmet.title.toComponent(),
helmet.priority.toString(),
helmet.meta.toComponent(),
helmet.link.toComponent(),
helmet.style.toComponent(),
helmet.script.toComponent(),
helmet.noscript.toComponent(),

Guess it should probably be added conditionally based on a config variable.

This has been resolved in https://github.com/me4502/gatsby-plugin-react-helmet-async/commit/be98a85250191c22ee6d2b2ebb457b542abb01e5
Unless there was something else wrong here, I'll close this issue. I'll try to get the above commit into a release shortly.


Feature/private fire support

This PR adds support for downloading private short reads from FIRE using a custom script. I didn't use the fetch tool because I'm hoping the Nextflow team will upgrade/fix the nf-aws plugin (the one that adds support for S3-compatible storage).
When using the private_study flag, the code retrieves the FASTQ FTP paths and downloads the files using a boto3-based script. It then feeds those files to the other processes; the only part that was functionally modified is the beginning of the short reads assembler.
I added one relatively simple unit test, as we can't currently run EBI network-dependent tests (such as those that require FIRE). Additionally, nf-test doesn't support Nextflow secrets, which are used to obtain the FIRE credentials.
There are many changes related to formatting and linting issues that I've fixed (apologies for that, as it makes the PR messier... I just can't control myself! :).
PR checklist

[x] This comment contains a description of changes (with reason).
[X] If you've fixed a bug or added code that should be tested, add tests! (sort of)
[X] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] Make sure your code lints (nf-core lint).
[ ] Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
[ ] README.md is updated (including new tool citations and authors/contributors).

@mberacochea one general thought on integration tests for S3/FIRE dependent pipelines... could we use a minio service on GitHub actions / local docker compose setups? E.g. following https://rohanverma.net/blog/2021/02/09/minio-github-actions/

@mberacochea one general thought on integration tests for S3/FIRE dependent pipelines... could we use a minio service on GitHub actions / local docker compose setups? E.g. following https://rohanverma.net/blog/2021/02/09/minio-github-actions/

I really like that idea.


No events fired when property is changed programmatically

There are three cases in which a widget in Tabris.js could possibly fire an event:

creation of the widget while explicitly setting a property that has an associated change event
programmatically setting a property which has an associated change event
user interaction

Example for case 1:
tabris.create("RadioButton", {
  text: "foo",
  selection: true
}.on("change:selection", function() { ... }).appendTo(page);

Here I would expect the event not to be fired, because the selection property is set before the listener is attached.
Example for case 2:
var foo = tabris.create("RadioButton", {
  text: "foo"
}.on("change:selection", function() { ... }).appendTo(page);
foo.set("selection": true);

Here I would expect the event to be fired and thus the listener to be executed when the second line sets the selection property programmatically.
Currently this example does not fire an event. At least the listener is not executed. Although, the state of the button changes correctly.
What is to be expected in the following cases?
tabris.create("RadioButton", {
  text: "foo"
}.on("change:selection", function() { ... }).set("selection", true).appendTo(page);

tabris.create("RadioButton", {
  text: "bar"
}.on("change:selection", function() { ... }).appendTo(page).set("selection", true);

I would suspect both of them to be equivalent to the second example.
I would also expect that all other widgets in Tabris.js that have properties with associated change events to behave consistently. Examples for such properties are: TextInput.text, ToggleButton.selection, CheckBox.selection. Is this correct?

I agree with everything. Issues #215 and #216 also deal with this.

Fixed. See also #215.


Flatten CSS selectors

A number of components have more nested styles than necessary, and for house cleaning we want to make them more shallow.

This is not actionable, and can only be done as we consider refactoring. Closing to clean up backlog.


Order completed but no PayPal transaction

PHP Version
8.2.22
Shopware Version
<IP_ADDRESS> Paypal 9.5.0
Affected area / extension
Extension:PayPal-Integration
Expected behaviour
Money is posted to my account from the customer's Paypal account
Actual behaviour
The payment status is set to Paid, there is no error, but no money is debited from the customer and no money is posted to my account
How to reproduce
Normal ordering process

[public] Shopware automated response: This issue is linked to the internal issue: PPI-984
[created from PPI-984, comment 491838]

Hi @senme, the information provided is a bit generic, so I would ask you to have a look at the logs at <shopware-root>/var/log/prod-<date>.log and search for paypal.ERROR, paypal.WARN, paypal.INFO and paypal.NOTICE that roughly match the time of the order (note that the timestamps are UTC and so may not match your local time).
What payment method was used? When did the status change (look at the status history of an order)? Can you try to reproduce it in a staging environment, for example?
Please also make sure that no flow or plugin is causing this.

The Request was:
{
"additional_properties": {},
"header": {
"ak-client-info": "xxxxxx",
"akamai-x-forwarded-for-7": "xxxxxx",
"authorization": "xxxxx",
"client-auth": "xxxxx",
"client_info": "xxxxxx",
"correlation-id": "4a3b3c4eb4da9",
"host": "api-m.paypal.com",
"paypal-partner-attribution-id": "shopwareAG_Cart_Shopware6_PPCP",
"paypal-routing-metadata": "xxxxxx",
"pp_geo_loc": "DE",
"pp_remote_addr": "X.X.X.X",
"pp_vip": "X.X.X.X",
"traceparent": "xxxxxx",
"true-client-ip": "xxxxxx",
"user-agent": "GuzzleHttp/7",
"x-akamai-edgescape": "xxxxxx",
"x-b3-spanid": "xxxxxx",
"x-edge-security-info": "xxxxxx",
"x-forwarded-for": "X.X.X.X, X.X.X.X",
"x-forwarded-host": "xxxxxx",
"x-forwarded-server": "xxxxxx",
"x-paypal-correlation-id": "4a3b3c4eb4da9",
"x-pp-corrid": "4a3b3c4eb4da9",
"x-pp-idempotencyid": "4a3b3c4eb4da9_1725299049",
"x-pp-slingshot-targetapp": "apiplatformproxyserv",
"x-slr-orig-script_uri": "https://api-m.paypal.com/v2/checkout/orders/33601587U81448926"
},
"method": "GET"
}

Found the issue, someone created a flow-build that turns the paid-status from unconfirmed to paid


No usable version of the libssl was foun

Issue Type: Bug
After I upgrade Ubuntu to 19.04, when I launch vscode with a python file, the output panel just shows the log below:

No usable version of the libssl was found
[Info  - 8:30:07 AM] Connection to server got closed. Server will restart.
No usable version of the libssl was found
[Info  - 8:30:08 AM] Connection to server got closed. Server will restart.
No usable version of the libssl was found
[Info  - 8:30:09 AM] Connection to server got closed. Server will restart.
No usable version of the libssl was found
[Info  - 8:30:10 AM] Connection to server got closed. Server will restart.
No usable version of the libssl was found
[Error - 8:30:11 AM] Connection to server got closed. Server will not be restarted.

Extension version: 2019.3.6558
VS Code version: Code 1.33.1 (51b0b28134d51361cf996d2f0a1c698247aeabd8, 2019-04-11T08:20:22.771Z)
OS version: Linux x64 5.0.0-13-generic snap

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (4 x 2111)

GPU Status
2d_canvas: disabled_softwarechecker_imaging: disabled_offflash_3d: disabled_softwareflash_stage3d: disabled_softwareflash_stage3d_baseline: disabled_softwaregpu_compositing: disabled_softwaremultiple_raster_threads: disabled_offnative_gpu_memory_buffers: disabled_softwarerasterization: disabled_softwaresurface_synchronization: disabled_offvideo_decode: disabled_softwarewebgl: disabled_offwebgl2: disabled_off

Load (avg)
0, 0, 0

Memory (System)
10.77GB (6.51GB free)

Process Argv

Screen Reader
no

VM
100%

Same here.

v19 is not yet officially supported.
Prerequisites for using the language server in the python extension can be found here https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore2x
If you are insane to get around this by installing the dependencies manually, please use the Jedi language server.
To do so, go into your user settings.json file and add the following setting:
"python.jediEnabled": true

@DonJayamanne I'll use jedi instead for now. Thanks.

Fixes can be found here:
https://github.com/OmniSharp/omnisharp-vscode/issues/3010
https://github.com/OmniSharp/omnisharp-vscode/pull/3011

Issue moved to Microsoft/python-language-server #1007 via ZenHub

0.2.62 has been released, which has been built from an updated dotnet toolchain which supports libssl1.1.

Yes, it works now, thanks.


JoinHostPort() enclose IPv6 in square brackets even if host is already in this format

What version of Go are you using (go version)?
go version go1.8.1 darwin/amd64
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jonathan/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fb/736129595jl9p0x2nsxx2lmh0000gn/T/go-build264978375=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?
https://play.golang.org/p/nrIZqsxHIY
What did you expect to see?
[[fd41:9eff:c33::1b:ffff]]:5672
What did you see instead?
[fd41:9eff:c33::1b:ffff]:5672

Your input is invalid. JoinHostPort expects a host name, which does include square brackets. The square brackets are only valid when joined with a port.

Note that JoinHostPort doesn't do any sanitization of your host (e.g. https://play.golang.org/p/cbnk8RrQNy).
The existing docs say exactly what it does:

JoinHostPort combines host and port into a network address of the form "host:port" or, if host
contains a colon or a percent sign, "[host]:port".

I'm not sure there's anything to do here.


Add Awesome Hacking Resources

[Insert URL to the list here]
https://github.com/vitalysim/Awesome-Hacking-Resources
[Explain what this list is about and why it should be included here]
A collection of hacking / penetration testing resources to make you better!
By submitting this pull request I confirm I've read and complied with the below requirements.
Please read it multiple times. I spent a lot of time on these guidelines and most people miss a lot.

I have read and understood the contribution guidelines and the instructions for creating a list.
This pull request has a descriptive title.For example, Add Name of List, not Update readme.md or Add awesome list.
The entry in the Awesome list should:

Include a short description about the project/theme of the list. It should not describe the list itself.Example: - [Fish](…) - User-friendly shell., not - [Fish](…) - Resources for Fish..
Be added at the bottom of the appropriate category.

The list I'm submitting complies with these requirements:

Has been around for at least 30 days.That means 30 days from either the first real commit or when it was open-sourced. Whatever is most recent.
It's the result of hard work and the best I could possibly produce.
Non-generated Markdown file in a GitHub repo.
Includes a succinct description of the project/theme at the top of the readme. (Example)
The repo should have awesome-list & awesome as GitHub topics. I encourage you to add more relevant topics.
Not a duplicate.
Only has awesome items. Awesome lists are curations of the best, not everything.
Includes a project logo/illustration whenever possible.

Either fullwidth or placed at the top-right of the readme. (Example)
The image should link to the project website or any relevant website.
The image should be high-DPI. Set it to maximum half the width of the original image.

Entries have a description, unless the title is descriptive enough by itself. It rarely is though.
Includes the Awesome badge.

Should be placed on the right side of the readme heading.
Should link back to this list.

Has a Table of Contents section.

Should be named Contents, not Table of Contents.
Should be the first section in the list.
Should only have one level of sub-lists, preferably none.

Has an appropriate license.

That means something like CC0, not a code licence like MIT, BSD, Apache, etc.
If you use a license badge, it should be SVG, not PNG.

Has contribution guidelines.

The file should be named contributing.md. Casing is up to you.

Has consistent formatting and proper spelling/grammar.

The link and description are separated by a dash. Example: - [AVA](…) - JavaScript test runner.
The description starts with an uppercase character and ends with a period.
Drop all the A / An prefixes in the descriptions.
Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

Doesn't include a Travis badge.You can still use Travis for list linting, but the badge has no value in the readme.

Go to the top and read it again.

Please read the guidelines more closely. Multiple points not followed from it. Try going through the list point for point and check that you follow it. :)

Closing for lack of response.


plotting of fine grained meshes is sub-optimal because of edge drawing

When plotting a ugrid with large resolution, sometimes narrow branches with fine cells are drawn too wide because the edges are drawn. I have created a minimal reproducible example with one of the datasets included in xugrid:
import matplotlib.pyplot as plt
plt.close("all")
import xugrid as xu

uds = xu.data.adh_san_diego()
uds_sel = uds.ugrid.sel(x=slice(470000, 490000), y=slice(3605000, None))

vmin, vmax = 0, 10

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(10,7), sharex=True, sharey=True)

uds_sel.elevation.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax1) #edgecolors="face", linewidths=None
uds_sel.elevation.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax2, edgecolors="face", linewidths=None) #defaults
uds_sel.elevation.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax3, edgecolors="face", linewidths=0)
uds_sel.elevation.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax4, edgecolors="none", linewidths=0)

for ax in [ax1, ax2, ax3, ax4]:
    ax.scatter(478100, 3616500, s=300, edgecolors="r", facecolors='none')

fig.tight_layout()

This is not the best example. However, it is clear that the empty area between cells is wider in some cases than in others. The lower-left example shows the most whitespace but also has white edges all over the grid:

A better example would be with:
import matplotlib.pyplot as plt
plt.close("all")
import dfm_tools as dfmt

file_nc = r"p:\11209585-dtp\voor_Caroline\01_scripts\post\python_differences\225deg-Car004-cropped.nc"
uds = dfmt.open_partitioned_dataset(file_nc)
uds_sel = uds.isel(time=-1).ugrid.sel(x=slice(3, None), y=slice(None, 53))

vmin, vmax = 0, 10

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(10,7), sharex=True, sharey=True)

uds_sel.mesh2d_s1.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax1) #edgecolors="face", linewidths=None
uds_sel.mesh2d_s1.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax2, edgecolors="face", linewidths=None) #defaults
uds_sel.mesh2d_s1.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax3, edgecolors="face", linewidths=0)
uds_sel.mesh2d_s1.ugrid.plot(vmin=vmin, vmax=vmax, ax=ax4, edgecolors="none", linewidths=0)

fig.tight_layout()

Since you mention it's a better example, maybe you could post the pictures for the file nonetheless?
(The p:\11209585-dtp\voor_Caroline\01_scripts\post\python_differences\225deg-Car004-cropped.nc I mean)
Having the MWE is helpful, but those pictures probably clarify it. I atleast have trouble seeing what the problem exactly is in these pictures.
Also, what's the question/issue exactly? Do you suggest a different default?
Or do none of the option above look satisfactory?

Ah okay I think I understand: the rivers look really fat.

You want the lines to fill up the holes between the polygons, but not so much that it smears everything. The easiest way to achieve this seems to fiddle with the linewidths parameter.
It's a bit hard to find what linewidths means exactly, but apparently it's points:
https://discourse.matplotlib.org/t/what-are-the-units-for-linewidths-parameter/22982/3
Where 1 point is 0.353 mm or 1/72 of an inch: https://en.wikipedia.org/wiki/Point_(typography)
fig, ax = plt.subplots(figsize=(7, 7), dpi=300)
uds_sel.mesh2d_s1.ugrid.plot(vmin=vmin, vmax=vmax, linewidths=0.1)

At 300 dpi, 0.1 is not quite enough; if you zoom in:

But 0.3 seems to do the trick:

0.3 seems to work fairly well across the board here, also at the default 100 dpi:

Total extent is about 200 km in the y direction, with 7 inches times 72 is 514 points. A single point is then approximately 400 m on the map. So we'd expect the rivers to be buffered 200 m at both sides, increasing the total width by ~400 m.
This analysis doesn't help that much though, because I wouldn't know how to compute the widths of the gaps in the North sea between the polygons...
So probably best to fiddle with the linewidths: get them a small as possible, without the gaps appearing.

Thanks for these efforts. However, it is a bit cumbersome to fiddle with linewidths, since they will differ for each model output file and zoom level right? For instance in the adh_san_diego plot this still gives white lines with linewidth=0.3.
My preference would be that matplotlib does not draw lines/edges when using edgecolors="none" or linewidth=0. Furthermore, the faces are perfectly adjacent, so they should be drawn without white spaces in between I would say.

I agree, but it's a somewhat workable solution for now.
I tried switching to other backends, but the white lines are drawn consistently.
It's a matplotlib issue, so you could try raising an issue there, or ask on the matplotlib discourse forum first:
https://discourse.matplotlib.org/
It might be partly a stylistic choice. I think I recall seeing it on other patches as well, so I'm not entirely sure it's just the PolyCollection, it could be all patches. I don't have my hopes up that they'll be eager to change this though, I reckon it has been like this forever.
As you mentioned, it's partly caused by having so many small elements, which is apparently triggering it. This also means it's a little slow at drawing the plot: I'm pretty sure it's looping over every polygon while rasterizing to pixels. It's been on my list to add some datashader trimesh support: https://datashader.org/user_guide/Trimesh.html
In that case, datashader is doing the rasterization (via numba). It doesn't draw the edges, and then just class matplotlib to do an imshow (if I recall correctly).

It also gives other issues.
Matlab:

xugrid:

The dam here is curvy in the python version, not in matlab. This also has to do with edge drawing.

From matplotlib: here's a good explanation https://github.com/matplotlib/matplotlib/issues/9574#issuecomment-339532262

This is a fairly difficult to solve problem in general. See e.g. https://computergraphics.stackexchange.com/questions/1824/what-is-illustrators-vector-rasterization-process.
It's not, strictly speaking, because the edges of the polygons are not matching. It's because the renderer only sees one patch at a time. When it antialiases the pixels on the edges of the polygon, the only thing it can do is assign partial occupancy by using partial transparency. When the renderer sees the second polygon, it will do the same. So an edge pixel which was supposed to have, say, left 50% red and right 50% green gets rendered as (white background) (50% transparency red) (50% transparency green), and thus the white "leaks through". This is known as a conflation artefact (I think due to the "conflation" between coverage and alpha).
AFAIK the only way to solve this is to emit your polygons as mesh objects, so that the renderer actually "knows" that the two edges are supposed to be the same (and even then you need a renderer that actually knows how to use that information). See also #7841.

See also: https://github.com/jklymak/contourfIssues
This mentions:

If you are producing PDF files, setting the edgecolor='face' option in contourf and setting the linewidths=0.2 yields a good result on a 200-dpi monitor.

Which matches my observation of a value of ~0.3. What we could do, is simply set edgecolor="face" by default, and a linewidth of 0.2. This would not get rid of all artifacts, but would good give decent results (enough so that people don't have to fiddle with it for hours like us...).

We should also make a note in the plotting docs.


apply_func.py: from torchtext.legacy.data import Batch

🐛 Bug

In the latest torchtext (version 0.9.0a0+036df73) built from source:
The name Batch is no longer located under torchtext.data
Batch is now located under torchtext.legacy.data
--Error message--
File "/home/daniel/py38/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 28, in
from torchtext.data import Batch
ImportError: cannot import name 'Batch' from 'torchtext.data' (/home/daniel/py38/lib/python3.8/site-packages/torchtext/data/init.py)
You can fix this by changing line line 28 of pytorch_lightning/utilities/apply_func.py to:
from torchtext.legacy.data import Batch
This does not reproduce with BoringModel because BoringModel does not import torchtext.data.Batch
Expected behavior
No error message.
Environment

CUDA:
- GPU:
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- A100-SXM4-40GB
- available:         True
- version:           11.2
Packages:
- numpy:             1.19.5
- pyTorch_debug:     False
- pyTorch_version:   1.9.0a0+92a4ee1
- pytorch-lightning: 1.1.5
- torchtext: 0.9.0a0+036df73     <--------------------------
- tqdm:              4.57.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.8
- version:           #50-Ubuntu SMP Tue Feb 9 06:29:41 UTC 2021

Additional context

This should fix it:
https://github.com/PyTorchLightning/pytorch-lightning/pull/6211
I was able to successfully create the option to use torchtext.legacy.data or torchtext.data depending on the torchtext version after my 3rd commit on this pull request.

My fourth commit in the pull request (#6211 ) uses distutils.version.LooseVersion

Thanks everyone :)

It still fails when Run on Google Colab, as seen in Unable to import pytorch_lightning on google colab and yes it has been tested.

It still fails when Run on Google Colab, as seen in Unable to import pytorch_lightning on google colab and yes it has been tested.
I have the same problem as you. Have you fixed it?
I

I have followed the above instructions, and when I have installed it. I can see this error message. Is this ok to ignore this message?

You won't be able to use the torchvision package/module, if you use that solution. If you use the other one, and use the comment that has been added, you can use everything (I think).

Thank you so much

Thank you.


US601024: Bring Java Adapter SDK back into sync with .NET version

https://internal.almoctane.com/ui/entity-navigation?p=131002/6001&entityType=work_item&id=601024
Renamed references of RetrieveFilesDataRequest to RepositoryFilesRequest, to be in sync with the changes in .NET Adapter SDK code.
Also renamed RetrieveFilesDataRequestImpl to RepositoryFilesRequestImpl in order to be consistent.
NOTE: This PR must not be merged until https://github.com/FileAnalysisSuite/adaptersdk-interfaces/pull/5 is merged.

CI Build Link:
http://sou-jenkins2.swinfra.net:8080/job/FileAnalysisSuite/job/FileAnalysisSuite~adaptersdk-impl-jaxrs~US601024-Rename-RetrieveFilesDataRequest~CI


Support intel igpus

Hi, I’m submitted this PR to enable Intel iGPU through the OLLAMA_INTEL_IGPU environment variable.
Due to the limitations of Intel’s foundation software (details can be seen in this issue), it is currently impossible to directly obtain the total vram and available vram of the iGPU.
For Intel iGPU, the maximum available iGPU vram is roughly half of the total ram. Therefore, as a workaround, this PR adds the OLLAMA_INTEL_USED_SYSTEM_VRAM environment variable to set the size of the used VRAM that is shared with the host. In this PR, the size of iGPU’s freeMemory is designed to be totalram/2-IntelUsedSystemVRAM.
For the Windows platform, the used system vram can be obtained with this PowerShell command:
(((Get-Counter "\GPU Process Memory(*)\Local Usage").CounterSamples | where CookedValue).CookedValue | measure -sum).sum

For the Linux platform, it can be obtained with this command:
sudo sed 's/^.*, \([0-9]\+\) bytes$/\1/' /sys/kernel/debug/dri/0/i915_gem_objects |\
  awk '{print $1, "Bytes"; exit}'

EDIT: try this release binary to enable ollama run with Intel GPUs.
UPDATE: update experimental-oneapi ollama server to v0.0.2

At the same time, I would like to raise some discussions about the inference backend of Intel device detection.
topic1: Inference Backend
Background: With Intel launching the Core Ultra processor, it can be foreseen that the performance of newly launched iGPUs will become increasingly powerful, and they will have better performance than CPUs when doing inference.
Therefore, on systems without dGPU and equipped with Intel client CPUs, how to choose the appropriate ollama inference backend has become a topic for discussion.
Solution 1. Through the OLLAMA_INTEL_IGPU environment variable
This is the solution in this PR, the advantage is simplicity, the disadvantage is that users cannot get the best experience, they need to manually test which device has better performance.
Solution 2. Only report powerful iGPUs to ollama during device discovery
the iGPU performance can be judged by indicators such as the number of EUs or PCI ID. The advantage of this approach is that users can directly enjoy the best performance, the disadvantage is that it requires major modifications to the device detection of Intel GPU.
topic2: Intel Device Detection
Background: As mentioned in the previous section, more device-related information (such as PCI ID, number of EUs) may be needed in the future.
But through my experiments, I found that it is quite troublesome to get this information with level-zero at present.
Fortunately, we can get this information very conveniently through the api provided by SYCL (essentially, the SYCL api is also calling the level-zero api, but users do not need to understand the details of level-zero).
Therefore, I am thinking that if we may need to get more device information in the future, we might as well change the backend of Intel device detection from level-zero to SYCL.
To achieve this, one possible solution is to use the oneapi related compilation toolchain to compile a binary(e.g. intel_device_manager). When ollama needs to detect Intel devices, it will:

Preconfigure the dependency library path of the binary and device information query parameters
Execute the binary and write the query results to a temporary file or standard output,
Ollama reads the query results and parses them into objects that ollama can handle.

The reason for choosing binary instead of lib is that it is difficult to get the relevant dependency library path when starting ollama.
Any comments on the above two topics? Any suggestions are welcome.  @dhiltgen @airMeng @luoyu-intel

Fortunately, we can get this information very conveniently through the api provided by SYCL (essentially, the SYCL api is also calling the level-zero api, but users do not need to understand the details of level-zero).

how about this one? Since it is part of llama.cpp, I think it can be more smooth
https://github.com/ggerganov/llama.cpp/blob/master/examples/sycl/ls-sycl-device.cpp

how about this one? Since it is part of llama.cpp, I think it can be more smooth https://github.com/ggerganov/llama.cpp/blob/master/examples/sycl/ls-sycl-device.cpp

it could be a nice sycl reference, but I'd like to leave the device management on ollama side since in this way we will have more flexibility once we need some new management functions. besides that, ollama also need to get device related information before launch the llama.cpp related binary

For topic 1, I prefer solution 2 as default, solution 1 can override solution 2, like what OMP does.
I believe sycl::device can get everything you want for your topic 2.

It's unfortunate there isn't a clean API to expose the free VRAM for these iGPUs.  I'm concerned that having the user feed in the free memory on the iGPU via env var is going to be problematic for concurrency.  I think we should try to find a way to get it wired into the scheduler somehow so the value is current and accurate as iGPU memory usage changes over time.
For windows, if it can be done through powershell, I would think we could leverage golang.org/x/sys/windows to call the applicable APIs.  You can check out https://github.com/ollama/ollama/blob/main/gpu/amd_hip_windows.go as some example code in the repo.
For linux, it's certainly not ideal, but maybe as a stop-gap until the data is available through an unprivileged API we can attempt to read the sysfs node, and warn on failure, and document that Intel iGPUs require root for now.  Maybe the fall-back is disable concurrency by setting UnreliableFreeMemory=true, and aim for some percentage of VRAM, although I worry this could still result in OOM crashes.
On the topic of filtering out slow iGPUs and only exposing iGPUs that are more powerful than the CPU, that's a good question.  Is my assumption correct that with some of the older but still supported iGPUs, the CPU with AVX+ is actually going to be faster than using the iGPU?  Do you think we would be able to use a heuristic of checking CPU capabilities as a close-enough deciding factor?

The reason for choosing binary instead of lib is that it is difficult to get the relevant dependency library path when starting ollama.

This is one piece of the puzzle we do need to work on before we can start shipping Intel GPU support in the official builds.  I'm operating under the assumption that redistribution is OK, so we need to bundle all the libs we depend on in the official builds, so users only need to install what would typically be needed to play video games (usually just the driver) and Ollama "just works" on the system.  I haven't had a chance to test if this set is complete recently, but from what I remember I still had to fiddle with things at runtime to get it to work on my test setup.

Hi @dhiltgen, I greatly appreciate your guidance. I agree with your concerns about memory issues in concurrent situations, and your comments are very reasonable. I will attempt to support dynamic memory refresh on the Windows platform in the next few days. For the Linux platform, I guess it might be more appropriate to reserve a fixed size rather than a fixed percentage of memory for iGPU image processing. Because in the case of large host-memory, it is obviously wasteful to only use a specific percentage of memory. From my observation, on Windows, the iGPU responsible for 4k display functionality will occupy about 1G of memory. I will also try to do some tests on Linux in the future.
For weak iGPUs filtering, the cpu package in Golang can certainly provide valuable information (such as CPU supported ISA, e.g., AVX2, AVX_VNNI). However, to complete this heuristic method of comparing CPU and iGPU performance, information related to iGPU is necessary. Take Alder Lake and Lunar Lake as examples, they support the same CPU ISA. If we assume that the iGPU performance of Alder Lake is weaker than the CPU, we would conclude that the iGPU of Lunar Lake is also weaker than the CPU. However, since the iGPU of Lunar Lake will be equipped with XMX units, its actual computing power will be far stronger than the CPU.
Regarding the issue of dependencies, I confirm that in the 2024.2 version of oneapi, these copied libraries are complete dependencies. I have already verified this on a machine without oneapi installed. But the problem here is, for example, on the Linux platform, we pack all the dependent libraries into the ollama binary and unzip the dependent libraries to a temporary directory at runtime. But if we compile the intel_device_manager into a library for use, we need to specify the dependent libraries related to intel_device_manager through environment variables when running ollama (note: these dependent libraries are consistent with the oneapi version llama_cpp_server). However, this is impossible because ollama has not yet unzipped the temporary library to the temporary directory before running. Therefore, I plan to compile the intel_device_manager into a binary (similar to ollama starting ollama_llama_server) to complete ollama’s acquisition of Intel GPU information.

On the CPU vs. GPU topic, I think we should try to keep it simple for now, with an advanced setting to allow users to deviate from the basic algorithm.  We should continue to favor Discrete GPUs over iGPUs.  With our new Concurrency support, even if the iGPU is a little slower than the CPU, for users with a discrete GPU(s) plus iGPU, this will enable more models to load concurrently, so that's a nice positive.
On the dependencies front, I'm working on a refactoring on how we package the dependencies for linux which should address the ordering problem - #5631

On the CPU vs. GPU topic, I think we should try to keep it simple for now, with an advanced setting to allow users to deviate

we will favor dGPU over iGPU for sure.

On the CPU vs. GPU topic, I think we should try to keep it simple for now, with an advanced setting to allow users to deviate from the basic algorithm.

how about determine whether to use iGPU by env var for now(just like what this pr did)? It's quite simple. we can turn to more complex heuristic method in the future if we need.

I'm working on a refactoring on how we package the dependencies for linux which should address the ordering problem

glad to hear that, I will try to figure out how this PR works in a few days.

how about determine whether to use iGPU by env var for now(just like what this pr did)? It's quite simple. we can turn to more complex heuristic method in the future if we need.

Maybe, but I was hoping we might be able to identify some auto-selection algorithm to make it so most users who have fast iGPUs will see them "just work" and users with very slow iGPUs wont have them activate, and only users in some gray zone would need to adjust settings.
For example, can we assume all Raptor Lake iGPUs are fast enough to enable by default?  If so, is there a CPU feature flag we could use to identify those?

For example, can we assume all Raptor Lake iGPUs are fast enough to enable by default? If so, is there a CPU feature flag we could use to identify those?

okay, let's assume Core Ultra series processors' iGPUs are fast enough.

hi @dhiltgen , i just created an experimental-oneapi release in my forked ollama repo. I validate the uploaded binary on my local machine without oneapi toolkits and it "just work". could you please have a try when you are free?

Do you think we would be able to use a heuristic of checking CPU capabilities as a close-enough deciding factor?

Maybe check also number of available CPU cores?   Single CPU core is not faster than iGPU [1], regardless of (used) CPU capabilities.  Workload needs to be efficiently scaled over many cores for that (but CPU affinity mask may limit CPU core availability, in addition to the host core count).
[1] Running given workload just on CPU can be faster than using CPU + GPU due to different factors, but that's likely to happen then both for discrete and integrated GPUs, unless CPU cores and iGPU end up badly fighting over system memory bandwidth, constantly trashing shared LLC cache => AI cases it may be not so much about whether iGPU is faster than CPU, but dGPU memory being isolated from CPU activity.

I just tested the release from the fork on my i7-1370p and it was not able to offload
time=2024-07-17T13:42:02.118-04:00 level=INFO source=gpu.go:235 msg="looking for compatible GPUs" time=2024-07-17T13:42:02.465-04:00 level=INFO source=types.go:104 msg="inference compute" id=0 library=oneapi compute="" driver=0.0 name="\x01" total="0 B" available="0 B"
Driver in play is probably standard Intel (<IP_ADDRESS>86) but maybe lacks some additional hooks for advanced oneapi use?
The good news is its detected as without the envvar I get the "no compatible GPUs were discovered" message on init.
Unsure if it's the order @eero-t called out.  I don't have sycl-ls built on this machine yet, but working on that to see what might be tripping things up.
Happy to provide more details if it helps as getting reliable iGPU support would unlock a lot of local small model usage in the group i am in.

i7-1370p

hi, this is due to your CPU is not Core Ultra series CPUs. we only enable Core Ultra series iGPU in this release(as llama.cpp SYCL backend had done many optimizations based on Core Ultra iGPU). In other CPUs(e.g. i7-1370p), ollama will not try to get iGPU total and free ram so we get total="0 B" available="0 B"  this log. 0B available ram lead ollama offload all layers to CPU rather iGPU.
this is indeed a bit confusing, I will remove the non-Core-Ultra iGPU in discovered GPU device lists.

Got it.  So for round 1 of iGPU it's 14g with the Xe-LPG gpu that should offload correctly.
I know the initial threads on llama.cpp for sycl were thinking 11g and forward, but may take some time.
Completely get that this is a non-trivial chunk of work, but do you think that the earlier Xe has a shot or is it Arc/Xe-LPG that will be the only ones that can support the type of access needed?

Got it. So for round 1 of iGPU it's 14g with the Xe-LPG gpu that should offload correctly.
I know the initial threads on llama.cpp for sycl were thinking 11g and forward, but may take some time.
Completely get that this is a non-trivial chunk of work, but do you think that the earlier Xe has a shot or is it Arc/Xe-LPG that will be the only ones that can support the type of access needed?

Through we might not expect high performance on 11+g, we still seek opportunities to improve performance on llama.cpp side, https://github.com/ggerganov/llama.cpp/discussions/8432 e.g., please stay tuned!

Yup been watching all the SYCL work in llama.cpp and testing some of the releases as they come out.  Can't do magic, but hopefully it can beat the CPU based inference that happens today.  All the Metal crowd are flaunting their tokens/s stats and like to bring a solid number of our Intel crew back to the table.  Thanks for all the hard work.

Do you think we would be able to use a heuristic of checking CPU capabilities as a close-enough deciding factor?

Maybe check also number of available CPU cores?

as we maintain the llama.cpp SYCL backend, we can do some op-level benchmark to determine which device performs better.

hi @dhiltgen, could you please review this PR?

Hi all, 2 questions for ya:

With this push, would it still be possible to force iGPU use, even if using the CPU would be faster? Some people might prefer the LLM performance hit so that their home server's core programs are not too bogged down. Also, I did some napkin math for the Intel Skylake microarchitecture and concluded that basically anyone not running a 4 core (physical, not hyper threaded) with a 72 core iGPU (only the highest end i7 SKUs) would stand to benefit from the higher GFLOP performance of the iGPU.
Could this support be added to all capable ipex-llm GPUs (I believe it supports up to 6th gen Skylake)? I would greatly appreciate that as someone with a Skylake based Intel NUC style computer ☺.

Hi all, 2 questions for ya:

With this push, would it still be possible to force iGPU use, even if using the CPU would be faster? Some people might prefer the LLM inference performance hit (CPU to iGPU) so that their home server's core programs are not too bogged down. Also, I did some napkin math for the Intel Skylake microarchitecture and concluded that basically anyone not running a 4 core (physical, not hyper threaded) with attached iGPU should stand to benefit from the higher GFLOP performance of the iGPU.
Could this support be added to all capable ipex-llm iGPUs (I believe it supports up to 6th gen Skylake)? I would greatly appreciate that as someone with a Skylake based Intel NUC style computer ☺.

Note: On that napkin math, for Skylake, I think I came to the rough performance numbers of 1 CPU core has 64 GLOPs at 4 GHz and 1 GPU execution unit has 16 GFLOPs at 1 GHz for FP32 (all based on articles I read and wikichip). So as long as the iGPU execution unit count is greater than 4x the physical core count, the iGPU should have better performance. In my case, I have a Skylake 6500t which has an iGPU with 24 execution units, so 24/4 = ~6 CPU cores of GFLOP performance, which is great than the 4 physical CPU cores I actually have.

hi, I can add force enable intel iGPU by some env var in this PR, but can't promise it can work normal as ollama with Intel GPUs use llama.cpp-SYCL as it backend, if you run with some issue during inference, you can raise related issue in llama.cpp-SYCL community

hi, I can add force enable intel iGPU by some env var in this PR, but can't promise it can work normal as ollama with Intel GPUs use llama.cpp-SYCL as it backend, if you run with some issue during inference, you can raise related issue in llama.cpp-SYCL community

I think that env var would be this, correct? OLLAMA_INTEL_GPU=1

Thanks! I think that env var would be this, correct? OLLAMA_INTEL_GPU=1

No, in this latest commit :smile:
OLLAMA_INTEL_GPU used to enable Intel GPU in ollama, including dGPUs and Core Ultra iGPUs.
to force enable iGPUs, try OLLAMA_FORCE_ENABLE_INTEL_IGPU
note: this needs to rebuild the oneapi ollama server, I will do some test and create a new release tag here very soon, please stay tuned!

Hi all, 2 questions for ya:

With this push, would it still be possible to force iGPU use, even if using the CPU would be faster? Some people might prefer the LLM inference performance hit (CPU to iGPU) so that their home server's core programs are not too bogged down. Also, I did some napkin math for the Intel Skylake microarchitecture and concluded that basically anyone not running a 4 core (physical, not hyper threaded) with attached iGPU should stand to benefit from the higher GFLOP performance of the iGPU.
Could this support be added to all capable ipex-llm iGPUs (I believe it supports up to 6th gen Skylake)? I would greatly appreciate that as someone with a Skylake based Intel NUC style computer ☺.

Note: On that napkin math, for Skylake, I think I came to the rough performance numbers of 1 CPU core has 64 GLOPs at 4 GHz and 1 GPU execution unit has 16 GFLOPs at 1 GHz for FP32 (all based on articles I read and wikichip). So as long as the iGPU execution unit count is greater than 4x the physical core count, the iGPU should have better performance. In my case, I have a Skylake 6500t which has an iGPU with 24 execution units, so 24/4 = ~6 CPU cores of GFLOP performance, which is great than the 4 physical CPU cores I actually have.

Far from battle tested, but I have a raptor lake (13g) w a Xe-LP (96eu) and using sycl actually seems to run slower than CPU based.  Now there is the cache optimization and plenty of Intel iGPU work going on upstream in llama.cpp so that may improve.  Even if TPS is a bit lower though the CPU has a ton of other things to deal with so dumping layers to the GPU avoids causing contention with other latency sensitive activities so for me tradeoff is worth it.  Hopefully things will improve on more than the newest Ultra/Arc iGPUs similar to what we get on aarch Metal based offload for the last couple generations.  Memory architecture is very different so I don't know if that is overly optimistic, but super early days so we shall see.

Far from battle tested, but I have a raptor lake (13g) w a Xe-LP (96eu) and using sycl actually seems to run slower than CPU based using llama.cpp by itself. Now there is the cache optimization and plenty of Intel iGPU work going on upstream in llama.cpp so that may improve. Even if TPS is a bit lower though the CPU has a ton of other things to deal with so dumping layers to the GPU avoids causing contention with other latency sensitive activities so for me tradeoff is worth it. Hopefully things will improve on more than the newest Ultra/Arc iGPUs similar to what we get on aarch Metal based offload for the last couple generations. Memory architecture is very different so I don't know if that is overly optimistic, but super early days so we shall see.

Additionally, testing on an Alder Lake i7-12800H (6P8E) with 96EU Xe-LP iGPU revealed lower energy consumption per token during inference than with the CPU. This results in reduced operating temperatures for thermally constrained devices like laptops or NUCs, although the iGPU is slower in this case.

Tried to give the 0.0.2 a run on a 13g using the OLLAMA_FORCE_ENABLE_INTEL_IGPU and OLLAMA_INTEL_GPU which got oneapi loaded with 50% of addressable memory (31.8GiB noted).
Tried to use llama3:8b and got a 0xc0000135.  Switched on OLLAMA_DEBUG and noticed a "Unable to load oneAPI management library" error with a "oneapi vram init failure: 78000001".  Do I need to load any specific libraries from the oneAPI toolkit or should I assume the support is available in the Intel Iris Xe kit (<IP_ADDRESS>86)?

Unable to load oneAPI management library

what's the platform? windows? linux? Both two platforms need to install GPU Driver, Linux platform is quite complex, you can refer to this page
could you please attach the full "Unable to load oneAPI management library" log, including library path and err msg so I can check the details

Sorry my bad, Windows in this case.
And there is a DLL at that location.  I did try running serve w elevated in case it was some rights issue, but no difference.  I can see in gpu_info where it trips, but why in zesinit it was I was still poking at.
time=2024-07-29T08:05:45.687-04:00 level=DEBUG source=gpu.go:534 msg="discovered GPU libraries" paths="[c:\\Windows\\System32\\DriverStore\\FileRepository\\iigd_dch.inf_amd64_9151f7d57698ae19\\ze_intel_gpu64.dll c:\\Windows\\System32\\DriverStore\\FileRepository\\iigd_dch.inf_amd64_da775d7ecf291310\\ze_intel_gpu64.dll c:\\Windows\\System32\\DriverStore\\FileRepository\\iigd_dch.inf_amd64_f49076a71317141a\\ze_intel_gpu64.dll c:\\Windows\\System32\\DriverStore\\FileRepository\\iigd_dch_d.inf_amd64_03416bbf28994523\\ze_intel_gpu64.dll]"
wiring Level-Zero management library functions in c:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_9151f7d57698ae19\ze_intel_gpu64.dll
dlsym: zesInit
dlsym: zesDriverGet
dlsym: zesDeviceGet
dlsym: zesDeviceGetProperties
dlsym: zesDeviceEnumMemoryModules
dlsym: zesMemoryGetProperties
dlsym: zesMemoryGetState
calling zesInit
zesInit err: 78000001
releasing oneapi library
time=2024-07-29T08:05:45.713-04:00 level=DEBUG source=gpu.go:614 msg="Unable to load oneAPI management library" library=c:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_9151f7d57698ae19\ze_intel_gpu64.dll error="oneapi vram init failure: 78000001"
wiring Level-Zero management library functions in c:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_da775d7ecf291310\ze_intel_gpu64.dll
dlsym: zesInit
dlsym: zesDriverGet
dlsym: zesDeviceGet
dlsym: zesDeviceGetProperties
dlsym: zesDeviceEnumMemoryModules
dlsym: zesMemoryGetProperties
dlsym: zesMemoryGetState
calling zesInit
zesInit err: 78000001
releasing oneapi library
time=2024-07-29T08:05:45.744-04:00 level=DEBUG source=gpu.go:614 msg="Unable to load oneAPI management library" library=c:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_da775d7ecf291310\ze_intel_gpu64.dll error="oneapi vram init failure: 78000001"
wiring Level-Zero management library functions in c:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_f49076a71317141a\ze_intel_gpu64.dll
dlsym: zesInit
dlsym: zesDriverGet
dlsym: zesDeviceGet
dlsym: zesDeviceGetProperties
dlsym: zesDeviceEnumMemoryModules
dlsym: zesMemoryGetProperties
dlsym: zesMemoryGetState
calling zesInit
calling zesDriverGet
oneapi driver count: 1
calling zesDeviceGet count 0: 0000017bfd2e6500
[0:0] oneAPI device name:
[0:0] oneAPI brand: └
[0:0] oneAPI vendor: ≈
[0:0] oneAPI S/N: unknown
[0:0] oneAPI board number: unknown
discovered 0 Level-Zero memory modules

zesInit err: 78000001

looks like zesInit fail, could you please upgrade the Intel GPU driver?

zesInit err: 78000001

looks like zesInit fail, could you please upgrade the Intel GPU driver?

Installed the latest and rebooted.  No more random error code, but still getting a 0xc0000135.  Given the zesinit return contents something is not being read well from the iGPU so garbage is being injected.
debug.log

And if it helps a sycl-ls --verbose output from an install of the Toolkit.  There are a couple OpenCL devices, but this was the output for the level zero instance.
Platform [#3]:
    Version  : 1.3
    Name     : Intel(R) Level-Zero
    Vendor   : Intel(R) Corporation
    Devices  : 1
        Device [#0]:
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Iris(R) Xe Graphics
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.29803
        Aspects    : gpu fp16 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_intel_legacy_image ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_oneapi_graph
        info::device::sub_group_sizes: 8 16 32
default_selector()      : gpu, Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics 1.3 [1.3.29803]
accelerator_selector()  : No device of requested type available. Please chec...
cpu_selector()          : cpu, Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i7-1370P OpenCL 3.0 (Build 0) [2<IP_ADDRESS>.02_160000]
gpu_selector()          : gpu, Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics 1.3 [1.3.29803]
custom_selector(gpu)    : gpu, Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics 1.3 [1.3.29803]
custom_selector(cpu)    : cpu, Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i7-1370P OpenCL 3.0 (Build 0) [2<IP_ADDRESS>.02_160000]
custom_selector(acc)    : No device of requested type available. Please chec...

debug.log

this log is unnormal time=2024-07-29T10:28:44.468-04:00 level=DEBUG source=server.go:390 msg=subprocess environment="[PATH=...]", seems that the PATH env var was polluted.
i ran the same version ollama runner and PATH log was
time=2024-07-30T09:07:15.230+08:00 level=DEBUG source=server.go:390 msg=subprocess environment="[PATH=C:\\Users\\wangzhe\\Downloads\\ollama\\dist\\windows-amd64\\ollama_runners\\cpu_avx2;C:\\Users\\wangzhe\\Downloads\\ollama\\dist\\windows-amd64\\ollama_runners;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\MinGw\\bin;C:\\Program Files\\dotnet\\;C:\\Users\\wangzhe\\Downloads\\w64devkit-1.19.0\\w64devkit\\bin;C:\\Program Files\\Go\\bin;C:\\ollama\\llm\\build\\windows\\amd64\\oneapi_v2024.0.2\\bin\\ollama_llama_server.exe;C:\\Program Files\\nodejs\\;C:\\Program Files\\Git\\cmd;C:\\Users\\wangzhe\\bin;C:\\Users\\wangzhe\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\wangzhe\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\Users\\wangzhe\\AppData\\Local\\GitHubDesktop\\bin;C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\Hostx64\\x64;C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin;C:\\Users\\wangzhe\\Downloads\\ninja;C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\include;C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\lib\\C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\lib\\x64;C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\crt;C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.22000.0\\um\\x64;C:\\Users\\wangzhe\\Downloads\\python-3.11.3-embed-amd64\\;C:\\Users\\wangzhe\\Downloads\\w64devkit-1.19.0\\w64devkit\\bin;C:\\Users\\wangzhe\\go\\bin;C:\\Users\\wangzhe\\AppData\\Roaming\\npm;C:\\ollama\\llm\\build\\windows\\amd64\\oneapi_v2024.0.2\\bin\\ollama_llama_server.exe;;C:\\Users\\wangzhe\\AppData\\Local\\Programs\\Ollama]"

Yeah that was me redacting some of the internal info that is in PATH and a couple other locations in the logs.  The driver seems to be getting loaded, but still not able to pull the data from the iGPU compared to syscl.

but still not able to pull the data from the iGPU compared to sycl-ls

[0:0] oneAPI device name:
[0:0] oneAPI brand: ≈
[0:0] oneAPI vendor:
[0:0] oneAPI S/N: unknown
[0:0] oneAPI board number: unknown

we noticed this garbage output before on Windows and will raise an issue to level-zero team, but this will not affect ollama functionality.
did you install MSVC on the machine? I guess there are some libs may be missing.

Yup I was trying to build from source for some vulkan testing as well.  Running 22 version of the toolchain, but see pointers to 19 elsewhere in ollama refs.  Are you looking for the full toolchain or just the VC redist?  Any specific release?

Yup I was trying to build from source for some vulkan testing as well. Running 22 version of the toolchain, but see pointers to 19 elsewhere in ollama refs. Are you looking for the full toolchain or just the VC redist? Any specific release?

if you want to build from this branch,
the toolchain just:

Microsoft Visual Studio
oneapi-basekit 2024.2
go version 1.22 or higher

when you build the oneapi runner, please build it in Intel oneAPI command prompt for Intel 64 for Visual Studio 2022 terminal
you can comment some build func like build_cuda, build_rocm to accelerate compilation in here

Yeah was getting a cmake error with mingw not having a platform param.  Didn't spend too much time debugging or tweaking the build ps script to fix it.  It was a low priority thing for me at the time as SYCL was where I wanted to be.

I know a lot is going on with Intel for the moment so this one may be on ice for a bit, but anything I can do to try and figure out why a raptor lake gen iGPU is not being used for offload?  It seems the level zero metadata issue isn't probably causing an issue, but is there something I can look at specifically? I have the MSVC 17.8 (2022 release) both as the redistributable, but also the SDK.  I can expect the redist will be available on users, but the SDKs would be pretty uncommon.  Working on getting build to work, but the cmake is fighting me a bit.

I know a lot is going on with Intel for the moment so this one may be on ice for a bit, but anything I can do to try and figure out why a raptor lake gen iGPU is not being used for offload? It seems the level zero metadata issue isn't probably causing an issue, but is there something I can look at specifically? I have the MSVC 17.8 (2022 release) both as the redistributable, but also the SDK. I can expect the redist will be available on users, but the SDKs would be pretty uncommon. Working on getting build to work, but the cmake is fighting me a bit.

hi @byjrack sorry for the inconvenience. Have you tried the llama.cpp directly? from the community feedback https://github.com/ggerganov/llama.cpp/issues/5467, iGPU shall be supported already following SYCL docs. If the issue exists on llama.cpp side, could you open an issue in llama.cpp repo and @me to fix. Otherwise, you could open another issue in Ollama repo with the details and @zhewang1-intc will look into it.

Yup @airMeng llama-cpp is able to do an offload.  Have those perf issues compared to CPU, but i know that's an active area of work by the Intel crew as well.
As the interface ideally would be ollama though at some point I was trying to help out where in the path of this PR was going.  I can def open an Issue, but given SYCL was experimental and sort of tied back to the PR I figured keep the comments aligned.

Hi all, ive been doing more reading on improving LLM performance on Intel client CPUs via quantization from FP16/FP32 to INT8/INT16 and the capabilities of Intel integrated GPUs with their execution units having some SIMD capabilities. Does anyone know the backend implementation of LLMs enough to know if people would stand to benefit running quantized INT8 models with SIMD iGPUs instead of on CPU cores with AVX? I imagine this would only help if the amount of SIMD units in the iGPU outnumbered the AVX units found in all the CPU cores by roughly 8x (4x for clock speed difference and 2x for AVX wider registers).

hi @TheSpaceGod, First of all, we cannot simply determine the computing power of a CPU and GPU based solely on SIMD width and clock speed. In addition to these two factors:
At the hardware level, factors such as the number of CPU cores, the number of EUs in a GPU and the number of threads that each EU can launch, the IPC difference between the GPU and CPU, and the device's instruction support (such as VNNI instructions, AMX instructions on CPUs, and XMX instructions on GPUs) will all determine the inference performance of LLMs. And the combination of these factors is very complex on client CPUs.
At the software level, whether the operator library for a specific hardware is sufficiently optimized will also determine the performance of that hardware in inferring LLMs. Fortunately, Intel's software team is actively optimizing the performance of llama.cpp on Intel GPUs.
Therefore, this is a comprehensive and complex issue. I think the most appropriate way is to do an apple-to-apple comparison of the inference profiles of CPUs/iGPUs to determine which device should be used for inference. I don't think this should be a complicated task for ollama/llama.cpp.

@zhewang1-intc , thanks for the explanation!

Is there a feel yet for if we can squeeze out any more performance of the SYCL on Xe-LP (11-13g) options?  I get this is more of a llama.cpp thing, but all the context on use is via an ollama lens.
Using llama-bench prompt processing is am getting 70-80 tps which is pretty responsive in ollama compared to AVX2 around 20.  Text gen though is only around 5 tps compared to 9 on AVX2.  I know the Apple architecture is fundamentally different given UMA latency and bandwidth, but that's what i am being compared against and that 260 and 30 on an m2 so a much more responsive experience.
Just trying to see if those who understand the boundaries of the problem have a gut on what could be possible on each hardware generation.

@zhewang1-intc I know the upstream llama.cpp folks are seeing what they can squeeze out of the CUs, but will the patches from the forked experimental build make it to the ollama build anytime soon?

Intel iGPU acceleration in Ollama is working great on my Intel NUC (Core 7 processor 150U) using the following code:
https://github.com/mattcurf/ollama-intel-gpu/tree/main
Before: 16% CPU, 0% GPU
Now: 4% CPU, 98% GPU

Intel iGPU acceleration in Ollama is working great on my Intel NUC (Core 7 processor 150U) using the following code:
https://github.com/mattcurf/ollama-intel-gpu/tree/main
Before: 16% CPU, 0% GPU Now: 4% CPU, 98% GPU

it doesn't work for me:
llama-intel-gpu  | |  |                   |                                       |       |Max    |        |Max  |Global |                     |
ollama-intel-gpu  | |  |                   |                                       |       |compute|Max work|sub  |mem    |                     |
ollama-intel-gpu  | |ID|        Device Type|                                   Name|Version|units  |group   |group|size   |       Driver version|
ollama-intel-gpu  | |--|-------------------|---------------------------------------|-------|-------|--------|-----|-------|---------------------|
ollama-intel-gpu  | | 0| [level_zero:gpu:0]|             Intel Arc A310 LP Graphics|    1.3|     96|    1024|   32|  4032M|            1.3.30049|
ollama-intel-gpu  | | 1|     [opencl:gpu:0]|             Intel Arc A310 LP Graphics|    3.0|     96|    1024|   32|  4032M|        24.26.30049.6|
ollama-intel-gpu  | | 2|     [opencl:cpu:0]|AMD Ryzen 7 5800X 8-Core Processor             |    3.0|     16|    8192|   64| 67348M|2<IP_ADDRESS>.11_160000|
ollama-intel-gpu  | ggml_backend_sycl_set_mul_device_mode: true
ollama-intel-gpu  | detect 1 SYCL GPUs: [0] with top Max compute units:96
ollama-intel-gpu  | llm_load_tensors: ggml ctx size =    0.30 MiB
ollama-intel-gpu  | llm_load_tensors: offloading 32 repeating layers to GPU
ollama-intel-gpu  | llm_load_tensors: offloading non-repeating layers to GPU
ollama-intel-gpu  | llm_load_tensors: offloaded 33/33 layers to GPU
ollama-intel-gpu  | llm_load_tensors:      SYCL0 buffer size =  7605.34 MiB
ollama-intel-gpu  | llm_load_tensors:        CPU buffer size =   532.31 MiB
ollama-intel-gpu  | time=2024-08-27T12:47:18.835Z level=INFO source=server.go:566 msg="waiting for server to become available" status="llm server not responding"
ollama-intel-gpu  | time=2024-08-27T12:47:19.086Z level=INFO source=server.go:566 msg="waiting for server to become available" status="llm server loading model"
ollama-intel-gpu  | Native API failed. Native API returns: -6 (PI_ERROR_OUT_OF_HOST_MEMORY) -6 (PI_ERROR_OUT_OF_HOST_MEMORY)
ollama-intel-gpu  | Exception caught at file:/home/runner/_work/llm.cpp/llm.cpp/ollama-internal/llm/llama.cpp/ggml-sycl.cpp, line:17257, func:operator()
ollama-intel-gpu  | SYCL error: CHECK_TRY_ERROR((*stream) .memcpy((char *)tensor->data + offset, host_buf, size) .wait()): Meet error in this line code!
ollama-intel-gpu  |   in function ggml_backend_sycl_buffer_set_tensor at /home/runner/_work/llm.cpp/llm.cpp/ollama-internal/llm/llama.cpp/ggml-sycl.cpp:17257
ollama-intel-gpu  | GGML_ASSERT: /home/runner/_work/llm.cpp/llm.cpp/ollama-internal/llm/llama.cpp/ggml-sycl.cpp:3024: !"SYCL error"
ollama-intel-gpu  | time=2024-08-27T12:47:37.103Z level=INFO source=server.go:566 msg="waiting for server to become available" status="llm server not responding"
ollama-intel-gpu  | time=2024-08-27T12:47:37.487Z level=INFO source=server.go:566 msg="waiting for server to become available" status="llm server error"
ollama-intel-gpu  | time=2024-08-27T12:47:37.737Z level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: aborted (core dumped) error:CHECK_TRY_ERROR((*stream) .memcpy((char *)tensor->data + offset, host_buf, size) .wait()): Meet error in this line code!\n  in function ggml_backend_sycl_buffer_set_tensor at /home/runner/_work/llm.cpp/llm.cpp/ollama-internal/llm/llama.cpp/ggml-sycl.cpp:17257\nGGML_ASSERT: /home/runner/_work/llm.cpp/llm.cpp/ollama-internal/llm/llama.cpp/ggml-sycl.cpp:3024: !\"SYCL error\""`

@moeinio This PR is about integrated GPUs (which can potentially use most of host RAM), whereas your GPU is a discrete one, with 4GB memory.   With that, you need to use a models/types which fit into that amout of GPU RAM.

https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md

@ayttop Right, I did not notice error actually being  about host (CPU?) memory:

ollama-intel-gpu  | Native API failed. Native API returns: -6 (PI_ERROR_OUT_OF_HOST_MEMORY) -6 (PI_ERROR_OUT_OF_HOST_MEMORY)

Can that really be due to pip dependencies: https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#troubleshooting ?


Suggestion: beware of default mutable args

Hi, I'm not sure this really causes an issue on this library, but I see you do:
@planning_solution
class TimeTable:
    def __init__(self, timeslotList=[], roomList=[], lessonList=[], score=None):
        ...

The use of timeslotList=[], roomList=[], lessonList=[] may be problematic since [] is mutable. Please check https://florimond.dev/en/posts/2018/08/python-mutable-defaults-are-the-source-of-all-evil/

Should probably change the default in the example to None. It should not cause an issue if user code uses it; all fields are overwritten with clones (and the collections are not touched). The library itself does not use mutable default arguments.


Style isolation for jQuery added elements

I'm trying to locally style Tabs. This works for every element that was statically added to the html/dom so Angular had a chance to process:

        ...

becomes

        ...

Tha _ngcontent-lnl-8 is added by Angular2 for style isolation. My custom styling works great on these elements.
However, adding jquery inserts a new element - the indicator - when the element in initialized:

Angular2 is kept out of the loop here (can't add _ngcontent-lnl-8), meaning that when I have a local css file that should override the indicator style, it isn't picked up (no selector match).
I have no idea how we could fix this. Any ideas?

I'm also trying to change the indicator color without success

A workaround is adding a global style with a very specific selector for your component (eg, by id). That works very well, but is kinda dirty.


Feature: support IF conditional expression syntax

This closes #46

@git-hulk No. I will fix it. Thank you!

@git-hulk I'm too busy to have no time to review now. Please wait. Sorry.

@makenowjust Thanks for your kind reply. No hurry, can have a look after you have the free time, I'm very appreciate your great work.

@makenowjust I fix those comment issues, can help to have a look again when you have free time.

The fundamental bug is that IF is treated as the keyword instead of an identifier. You can revert all fixes, and remove "IF" from the keyword list for this issue. This fix seems canonical to me.

The fundamental bug is that IF is treated as the keyword instead of an identifier. You can revert all fixes, and remove "IF" from the keyword list for this issue. This fix seems canonical to me.

OK, I'll rollback those commits and remove the IF keyword. Thanks @makenowjust

Thanks @makenowjust review


Provide REBAR2 config for locus

Hi @g-andrade
Can you please provide a rebar.config we can use with REBAR2?
Thanks

I've restored it in 1.10.2.
Bear in mind this could stop working at any moment in the near future - it's no longer practical to ensure compatibility rebar 2.
I strongly suggest you move to rebar3 if and when it's possible for you to do so.
Closing this; please reopen if you find any issues.

I've dropped in 1.15.0 due to the hack required no longer remaining feasible.


fix unused export removal

https://github.com/ankurk91/vue-loading-overlay/pull/90
https://github.com/ankurk91/vue-loading-overlay/pull/88

module.exports = {
  //...
  optimization: {
    usedExports: false,
  },
};

this webpack config worked for me

great!


Add doc for aml

one more doc,
https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/Overview.md

one more doc,
https://github.com/microsoft/nni/blob/master/docs/en_US/TrainingService/Overview.md

yes, this pr include this overview.md file.


Updates for overview

@jwforres moved styles to _core and replaced app.js with what's in your branch.


v17.3.1

Thanks you for your contribution to the Azure-SDK-for-Go! We will triage and review it as quickly as we can.
As part of your submission, please make sure that you can make the following assertions:

[ ] I'm not making changes to Auto-Generated files which will just get erased next time there's a release.

If that's what you want to do, consider making a contribution here: https://github.com/Azure/autorest.go

[ ] I've tested my changes, adding unit tests where applicable.
[ ] I've added Apache 2.0 Headers to the top of any new source files.
[ ] I'm submitting this PR to the dev branch, or I'm fixing a bug that warrants its own release and I'm targeting the master branch.
[ ] If I'm targeting the master branch, I've also added a note to CHANGELOG.md.
[ ] I've mentioned any relevant open issues in this PR, making clear the context for the contribution.

Can one of the admins verify this patch?

@jhendrixMSFT can we pick up the latest containerservice spec too? Thanks!

@joshgav I'll do a full minor update early next week.

Okay thanks. Can you pick up the updated README from https://github.com/Azure/azure-sdk-for-go/commit/b9f76dbde4ccc31ebee29b1a2974885a613f8f87?


BootstrapOperator will load all records while bootstrap parallelism increased, ListState and SPLIT_DISTRIBUTE is not suitable for the operator?

Expected behavior
Load records was slow while I increased the parallelism of the bootstrap operator, Caused by new subtask could not get the lastInstantTime because the ListState is SPLIT_DISTRIBUTE  redistribution mode.

When I changed the mode and then the subtask failed caused by <Incompatible state assignment modes. Was [SPLIT_DISTRIBUTE], registered with [UNION].>

Is there any suggestions for resolving the large table failover easily? This load action may take long time and the flink task is not initialled for long time.
Environment Description

Hudi version :0.11
Flink version: 1.15

Additional context
Stacktrace

Can value state be used there instead?

We can use UnionListState instead, but old job will init failed caused by <Incompatible state assignment modes. Was [SPLIT_DISTRIBUTE], registered with [UNION].> , So many jobs in prod will failed when flink task failover,   it is not excepected. Another way is add a aggregateManager, notice other subtask when task init from state,But we need to verify the data quality by this way.

We can have some breaking change for release 1.0 BTW.


Fix/issue 17

This corrects the order of variables the error message from PDBufferAdaptor when the channel count is insufficient, so now the error should make sense when it happens.
closes #17 (although the substance of that issue wasn't a bug but a misnamed parameter)

This fix that fixes a problem that was not even wrong is magic. Thanks. I resisted to merge to leave you the pleasure


fixes #5234

Fixes #5234
This will also change
'''

'''

to
'''
'''

where ''' represents backticks

Hey @andrewgazelka, thanks for taking the time to work on this PR, and for your first contribution to rustfmt!!
Just want to acknowledge that I've seen this and I'll try to do a thorough review soon.

Believe this is another approach as #4902 which was to address #4793. Haven't looked closely enough at either to speak to the different approaches but sharing for reference and awareness

@calebcartwright thanks for pointing that out. When I have a chance to review this PR I'll make sure to take a look at those links!

@ytmimi is this one you were still planning on taking a look at? if so feel free to assign to yourself (will be a helpful indicator for me :smile:)

Thanks again for opening this PR, but going to close this in favor of #5601, which has a simpler implementation


Use browser.* APIs when possible

https://github.com/crimx/webpack-target-webextension/blob/cbb1185e2e8935205a1f36d862bde80e1b96a3c3/lib/background.js#L3

chrome is defined everywhere, so this piece of code has always used the chrome.* API
There's no advantage in using the browser.* API in this case, it's still dealing with a callback

For this reason, I dropped the check and the duplicate code

Hi, first off, thanks for the PR!
The webpack-target-webextension, as its name suggests, is for WebExtension target. The chrome namespace is not part of the WebExtension spec. For the time being, major browser vendors also implement the chrome namespace because Google Chrome still dominates the web browser market and most of the current WebExtension APIs are heavily inspired by Chrome Extension.
For this reason, I would prefer using the browser namespace if possible and fallback to chrome.

Thanks!


[Fix] Small UI improvements to suggested widgets section

Description
To add minor improvements to suggested widget panel, as listed in the issue.
Fixes #6189
Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

manual

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:green_circle: Total coverage has increased
// Code coverage diff between base branch:release and head branch: fix/minor-ux-changes-suggested-widgets

Status
File
% Stmts
% Branch
% Funcs
% Lines

:green_circle:
total
51.85 (0.01)
33.67 (0)
29.76 (0)
52.48 (0)

:green_circle:
app/client/src/components/editorComponents/ActionRightPane/index.tsx
34.48 (0.76)
25 (0)
0 (0)
37.5 (0.79)

Merging this PR fails only for - Smoke_TestSuite/ServerSideTests/QueryPane/AddWidgetTableAndBind_spec.js


Map on Geographic Coverage page doesn't work

The IPT uses Google Maps JavaScript API to render the map on the Geographic Coverage page. It uses a keyless request meaning it doesn't include an API key.
Unfortunately as of June 22, 2016 Google Maps JavaScript API no longer support keyless access. More information about this change can be found in this blog post.
Fortunately Google has grandfathered all previously active domains for keyless usage. This means that IPTs setup before June 22, 2016 will "continue to function both now and in the future".
This change will require the next version of the IPT to either a) use a new map vendor (e.g. Leaflet, OpenStreetMap, GBIF, etc) or b) be configured with its own API key.

Note: On localhost domain a NoApiKeys warning appears but doesn't prevent the map from rendering. Otherwise a MissingKeyMapError appears, which does prevent the map from rendering. More information about these errors can be found here. The screenshots below demonstrate them:

I'd vote for using an open source base layer, such as OpenStreetMap or the nice terrain layer used on the new GBIF portal (http://demo.gbif.org/occurrence/1258202889), which is based on OpenStreetMap.

The IPT's maps (on geographic coverage page and IPT settings page) have now been updated to use the OSM-based 'terrain' tile set from Stamen Design, rendered using Leaflet. This tile set is available for free and unlimited use under the CC BY 3.0 license and is compatible with Leaflet. It features hill shading, natural vegetation colors, advanced labeling, etc.
To draw the bounding box, the leaflet-locationFilter library is used. A screenshot showing how the geographic coverage map appears following the change can be found below. For those interested, it is possible to tryout the 'terrain' tile set here.

User manual change required in geographic coverage section to reflect recent changes.

Changes to user manual completed. Changes applied to the following sections:

https://github.com/gbif/ipt/wiki/IPT2ManualNotes.wiki#geographic-coverage
https://github.com/gbif/ipt/wiki/IPT2ManualNotes.wiki#configure-ipt-settings


Apportion the space in the paradigm layouts better

In the refactoring, there appears to be an imbalance in terms of how much space is allocated to the row-label cells in comparison to the content cells, e.g.

I'm not convinced we need to have so much space to the left of the row labels. And it appears that the content cells are designated less space than the row label cells, when it probably should be the other way around.
Generally, I'd be inclined for some form of dynamic space allocation, based on the available screen size and then maximal length of the content. Though we might need to allow for row labels to be split on multiple lines. Would this be possible in principle?

Still - why is there so much space provided for the row labels, in comparison to the associated cell content?

I had tried different amounts of space and found that less space started to look awkward and unbalanced, especially on larger screens (computer monitors). It has been reduced some since this screenshot, but I can reduce it more. Could I get some guidance on how much to reduce the space on the left? By half? More than that?


Bindless Textures causing weird artifacts in PBR rendering

When we enable bindless textures with a common binding pool as the same binding point for the 2d, 3d and cubemap textures, especially in PBR pass we can see exaggerated Fresnel effect and the textures kinda look warped around the poles of the sphere

 Bindless Textures weird artifacts in PBR rendering


Handling "See also"

For now I've handled this by just embedding the text using ![[definitions#term|term]], but it would be nice to be able to use "See also term" and have that become a clickable link in the popup.

Thanks for the suggestion. Instead of the "See also term" interaction that you mentioned, I think it would be better if definitions are also underlined in the popup, and clicking on it links the user to the definition. This would achieve the same effect that you want, albeit in a different manner. What do you think?

Sounds good!

This is implemented in 0.21.0 and can be enabled via an optional setting.


ORC-412: [C++] Fix Char(n) and Varchar(n) writers with UTF-8

This patch is the parity of ORC-406 on the C++ side. Now char and varchar writers respect maximum length of UTF-8 characters in the specified file type.

@t3rmin4t0r This is the patch to reflect your commit for ORC-406 for the C++ writers. Can you take a look at it please?
@majetideepak Please take a look when you have time as well. Thanks!

@wgtmac I will take a look at this and the other PR first thing tomorrow morning.

@majetideepak Thanks for the review. To answer your question: VarCharColumnWriter just calls Utf8Utils::truncateBytesTo to truncate to the max length and we don't need to modify its data.
I just pushed another commit as per your feedback. Please check it again. Thanks!


[OS error: Too many open files] when saving checkpoint of model with 48 encoder layers and 48 decoder layers

Hello,
I tried to train a model with num_encoder_layers = 48 and num_decoder_layers = 48. I got the following error:
I0424 03:25:54.615840<PHONE_NUMBER>75424 checkpoints.py:600] Saving checkpoint for step 5000 to /mnt/disks/persist/t5_training_models/final_exps/pretraining/LIME/t5_48_bs32/checkpoint_5000.tmp-1650770754
Traceback (most recent call last):
  File "./t5x/train.py", line 635, in <module>
    gin_utils.run(main)
  File "/mnt/disks/persist/felix-t5x/t5x/gin_utils.py", line 103, in run
    app.run(
  File "/mnt/disks/persist/envs/t5/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/disks/persist/envs/t5/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "./t5x/train.py", line 614, in main
    _main(argv)
  File "./t5x/train.py", line 632, in _main
    train_using_gin()
  File "/mnt/disks/persist/envs/t5/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/mnt/disks/persist/envs/t5/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/mnt/disks/persist/envs/t5/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "./t5x/train.py", line 519, in train
    checkpointer.save(trainer.train_state,
  File "/mnt/disks/persist/felix-t5x/t5x/checkpoints.py", line 607, in save
    written_state_dict = self._write_state_to_tensorstore(
  File "/mnt/disks/persist/felix-t5x/t5x/checkpoints.py", line 720, in _write_state_to_tensorstore
    written_state_dict = _run_future_tree(future_written_state)
  File "/mnt/disks/persist/felix-t5x/t5x/checkpoints.py", line 160, in _run_future_tree
    leaves = loop.run_until_complete(asyncio.gather(*future_leaves))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/mnt/disks/persist/felix-t5x/t5x/checkpoints.py", line 699, in _write_array
    await t[param_info.local_chunk_info.slice].write(arr)
ValueError: Error writing local file "/mnt/disks/persist/t5_training_models/final_exps/pretraining/LIME/t5_48_bs32/checkpoint_5000.tmp-1650770754/target.decoder.layers_36.encoder_decoder_attention.value.kernel/0.0": Failed to open lock file: /mnt/disks/persist/t5_training_models/final_exps/pretraining/LIME/t5_48_bs32/checkpoint_5000.tmp-1650770754/target.decoder.layers_36.encoder_decoder_attention.value.kernel/0.0.__lock [OS error: Too many open files]
  In call to configurable 'train' (<function train at 0x7fb1e5e8bca0>)
/mnt/disks/persist/t5_training_models/final_exps/pretraining/LIME/t5_48_bs32

Thanks for any help.

Can you try setting ulimit -n 1200 or an even larger number?

Thank you, this worked.


Support adopting a cluster from a set of existing machines

This item tracks supporting users providing a list of machines that will be used to bootstrap the cluster rather than requiring machines to be added via the cluster API.

This issue was moved to kubernetes-sigs/cluster-api#16


ADC device with ERROR status cannot be reclaimed.

May need to regulate the state transform diagram.

JIRA Bug 1524

https://github.com/F5Networks/f5-adcaas-openstack/pull/243


Fix gitea repo bootstrap failure

Fix issue #278
The gitea client requires the gitea host URL for the repo but the provider provides a full repo URL.
So repo bootstrap with gitea provider cannot work.

@gamerslouis Can you add documentation for this?

Hi @todaywasawesome
I am not sure what's documentation you mean?


Vanilla crafting with a fluid is not prioritized over processing an item

Issue description:
Given two recipes for an item, one that requires an item crafted from a bucket of fluid and one that requires an item processed from another item, the processing recipe is used.
What happens:
I have two recipes for making a cake: one which needs Fresh Milk (craftable from a bucket of Milk, which is stored as a fluid in the system), and one which needs Soy Milk (a byproduct of making Tofu in a Presser). When neither Fresh Milk nor Soy Milk is in the system, Soy Milk is produced for the crafting request.
What you expected to happen:
Since vanilla crafting is always faster than processing, I expected Fresh Milk to be created and used.
Steps to reproduce:

Create a vanilla crafting recipe for an item that requires an item crafted from a bucket of fluid.
Create a vanilla crafting recipe for the same item that requires an item crafted using a processing recipe and an item.
Ensure that neither of the items that are directly required already exist in the system, but the components for making them do exist.
Request the product.

Version (make sure you are on the latest version before reporting):

Minecraft: 1.12.2
Forge: <IP_ADDRESS>85
Refined Storage: 1.5.31

Seeing this same issue for destabilized redstone dust.
The system does not use the crafted destabilized redstone after creating it through a magma crucible.
Minecraft 1.12.2
Forge <IP_ADDRESS>27
Refined Storage 1.5.32
This is version 1.10 of the Direwolf20 pack.

I'm going to redo this for 1.6.x, so this can be closed.


Add support for backend to matplotlib for Travis' VMs

Make changes to code to allow for handling of multiple plots in Travis by the backend 'Agg'

Coverage increased (+0.2%) to 94.022% when pulling 029cc793c3985b3317eea832f3b7ddb7a4366132 on robust-matplotlib-tests-to-travis into 466bea10b38a8cd1a42aee64cef1f3b0aad44f38 on develop.


Creating a subset list of MONDO terms which are relevant to drug repurposing.

This is following up on a conversation at the curation call on May 11th about the EveryCure ranking project. We're trying to query translator TRAPI endpoints looking for ranks for drug-disease pairs. Presently we are working by iterating through all possible MONDO terms as a rough approximation of all possible diseases what we could target, but obviously that can be pruned down much further.
Briefly summarizing our conversation, we talked about removing susceptibility subtypes from the pool of MONDO terms, and removing terms which are above a certain number of hops from leaf nodes or have specific node properties.

@DnlRKorn to be clear, is what you want a list of Mondo disease terms with the susceptibility terms excluded? If anything else is needed, can you please provide a detailed list with the criteria you need for this subset?
Feel free to join our bi-weekly tech calls on Fridays at 9am PT (next call is on May 28)
Thank you!

@nicolevasilevsky I believe we're interested in removing susceptibility terms and "terms which are unlikely to be a useful diagnostic criteria". This second one is complicated, what I've tried to look at and discuss is how high up the tree is useful for a reasonable disease; the example I have refered before is Schizophrenia 7 (MONDO:0011295) versus Schizophrenia (MONDO:0005090). Clearly we'd want our list to include both, but MONDO:0005090 isn't a leaf node, so that isn't a very useful criteria...

@DnlRKorn - thanks for your interest! We are very keen to help you with your use case, but we are a bit snowed under right now with finishing the Mondo paper; to help you more, I think we need to talk in person. Would it be possible to postpone the discussion on this to May 28, and you joining the call? Until then, maybe you can develop a more operational notion of the "terms which are unlikely to be a useful diagnostic criteria", and we can advice on how this is best implemented.

@DnlRKorn would you be able to join one of our future calls?

Yes! Sorry it's been very busy since the relay, I'll try and make time this
week!
On Tue, Jun 20, 2023, 7:10 PM Nicole Vasilevsky @.***>
wrote:

@DnlRKorn https://github.com/DnlRKorn would you be able to join one of
our future calls?
—
Reply to this email directly, view it on GitHub
https://github.com/monarch-initiative/mondo/issues/6262#issuecomment-1599698747,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABURCRQVMZHHYBGOZAO74GLXMIUUXANCNFSM6AAAAAAX6OTU4M
.
You are receiving this because you were mentioned.Message ID:
@.***>

Use case/Exact pipeline this will be applied to:

wants a list of diseases that are drug treatable
terms like Mendelian disease are not treatable per se
would like to prune out diseases like this to avoid unnecessary calculations of disease targets
want to be able to find potential drug-disease pairs
from a user query perspective - users could be querying for specific diseases, then having extra diseases in the hierarchy won't be as problematic
nice to see the shape of the ontology, 80% of the ontology is leaf nodes. Removing grouping classes would probably only remove 10% of the classes
Daniel would like to say if he finds a hit, would like to say it could be interesting for a physician
the border b/w what is a grouping class and what isn't is fuzzy
rather than filtering certain classes, may consider information content metric for diseases. Eg Mendelian disease has a low information content, therefore it would be less relevant to report
can use ontology to report IC
run a semantic similarity analysis, the higher the term is in the hierarchy, the lower the IC and then they could be valued less in the analysis
we need a way to compare the Mondo diseases with a real source of diagnoses- ie if you had  a database that contains true diagnoses
ICD is used in clinical diagnosis, everything with a mapped ICD10 code (that is lower in their coding system - must have a decimal to indicate it is a billable code
need to improve the ICD10 code mappings in Mondo
note, any measure that you apply to diseases will be a lossy
Nico recommends trying to make the IC score a bit more meaningful, like by considering gene relationships - like if you have a lot of associated genes, this could change the weight of the IC score
Curation/technical team should think further about how to develop a product/subset that approximates diagnosability in the IC score
Sanya is indexing PubMed with Mondo terms - this could be another approach to identifying the Mondo terms that are potentially diagnosable terms
combine ontological IC score and PMIDs and matched Mondo terms - this could be a good approach


Whatever

Checklist

[x] I ran npm test locally and it passed without errors.

[x] I only edited the _data/pixels.json file.

[x] I entered the username in the pixels.json that I'm also using to create this pull request.
[x] I acknowledge that all my contributions will be made under the project's license.

i updated the file reverted the commit but also merged upstream into whatever.


Laboratory work #7, Lifshits Dana - 20FPL-2

Necessary implementation of loading your dataset.

It's time to write the loading of your dataset.

And again: it's time to write the loading of your dataset.

And again: it's time to write the loading of your dataset.

It's time to start doing your first lab.

It's time to start doing your first lab.


Is the Riak java client still active?

I'm a maintainer on the YCSB project and I'm attempting to do a pass updating the various data store bindings we have in the project.
We have a riak KV binding but it hasn't really been updated since Apr 2016. It looks like the latest riak java client artifacts in maven central are also from 2016.
Is there a different maven repository I should point folks towards for a more recent release? Is there an updated release planned?

The riak clients no longer have any active support.  The riak database continues to be supported and developed as an open source produce, just not the clients.
The new versions of Riak should remain backwards compatible with the old versions of the client, it is just new features which will be unavailable.

thanks for the answer!


Check t-test computations

One participant made a remark about the t-test used in the template spreadsheet we used.

"I just did a paired-samples t-test in SPSS, the result in SPSS is t(24) = 5.22, p < .001. Using decimals or not does not really make a difference, so that cannot be the issue, the following website also gives me the same results:  http://www.socscistatistics.com/tests/ttestdependent
Perhaps you just have a way more sophisticated  method? Or use data transformations?"

The spreadsheet returned t(24)=-4.79, p<.001 whereas the calculator supplied the result

I need to check the origins of this.

Okay, I have looked at this and I see where the difference comes from. The approach to the t-test is different. The online calculator takes just the means response latencies per participant per condition.
The template spreadsheet, on the other hand, creates the pooled standard deviation per participant across conditions, and computes the Stroop effect for that individual. Subsequently, those Stroop effect scores are supplied to a t-test with H0: mu = 0 (difference is zero).
This way of calculating the t-test was based on the script from Many Labs (around line 134), which computes the Stroop effect in this way.
@Janto91 problem solved! I don't know if we still needed to e-mail the original respondent, but if you think it apt, please do refer him/her to this page.


Enabling filtering

Hey Paul.
First off all great job on this component. It's allowing me to achieve a layout grid I couldn't do with masonry.
I need to be able to filter elements within the grid. Right now when I do, it removes them from the column they are in without reshuffling the whole grid (redistributing the elements accordingly). How would one achieve this? Is there a way to do a transition as well?
Thanks.

Hi @hugolanuit, some time has passed since you posted this issue. Did you manage to find a solution?
If not, I'm curious to the implementation. Specifically how you're removing an item from the array? Any info would be great, so we can extend the plugin to support this way of deleting items