Add support for Islamic calendars

This proposal formally defines an abstract Islamic calendar, which can be used to represent various in use forms of the Islamic calendar (as discussed in #1 ).
ISLAMIC
The Islamic calendar is a deterministic lunar Hijri calendar, based on a 30-year cycle.  There are two possible epochs for this calendar.  In the civil epoch, 1 Muḥarram 1 aligns with Julian day 16 July 622.  In the astronomical epoch, 1 Muḥarram 1 aligns with Julian day 15 July 622.
Months have fixed lengths of 29 or 30 days, except for the twelth month, Dhū al-Ḥijja, which has an extra day in a leap year.
Permitted months are:

Code
Name
Days

MUHAR
Muḥarram (المُحَرَّم)
30

SAFAR
Ṣafar (صَفَر)
29

RABIA
Rabī I (رَبيع   الاوَّل)
30

RABIT
Rabī II (رَبيع الآخِر)
29

JUMAA
Jumādā I (جُمادى الأولى)
30

JUMAT
Jumādā II (جُمادى الآخِرة)
29

RAJAB
Rajab (رَجَب)
30

SHAAB
Sha‘bān (شَعبان)
29

RAMAD
Ramaḍān (رَمَضان)
30

SHAWW
Shawwāl (شَوّال)
29

DHUAQ
Dhū al-Qa‘da (ذو القَعدة)
30

DHUAH
Dhū al-Ḥijja (ذو الحِجّة)
29 (or 30 in a leap year)

In every 30-year cycle, 11 years are leap years.  The determination of leap years depends on the variant in use.
The epoch marker BH (before Hijra) is permitted in this calendar; year y BH indicates a year y years before year 1. Thus, there is no year 0; year 1 BH was followed by year 1.
Note on variants
The determination of leap years follows one of four rules.

Variant
Leap years with 355 days in each 30-year cycle

I
2, 5, 7, 10, 13, 15, 18, 21, 24, 26, 29

II
2, 5, 7, 10, 13, 16, 18, 21, 24, 26, 29

III
2, 5, 8, 10, 13, 16, 19, 21, 24, 27, 29

IV
2, 5, 8, 11, 13, 16, 19, 21, 24, 26, 30

Note on epochs
There are two possible epoch dates for each variant.
Epoch type 'a' ("astronomical" or Thursday epoch): Julian day 15 July 622
Epoch type 'c' ("civil" or "Friday" epoch): Julian day 16 July 622
To support all of these variants, an ISLAMIC calendar escape could be suffixed by the variant and epoch.  For example, ISLAMIC_II_C could be used to indicate a variant II Islamic tabular calendar with a civil epoch of Julian day 16 July 622.  This particular form would match with the Microsoft .NET implementation of the calendar:
(https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/libraries/System.Private.CoreLib/src/System/Globalization/HijriCalendar.cs)
Thus 8 new calendars are proposed:
g7:cal-ISLAMIC_I_A
g7:cal-ISLAMIC_II_A
g7:cal-ISLAMIC_III_A
g7:cal-ISLAMIC_IV_A
g7:cal-ISLAMIC_I_C
g7:cal-ISLAMIC_II_C
g7:cal-ISLAMIC_III_C
g7:cal-ISLAMIC_IV_C
Further useful information can be found here:
http://cldr.unicode.org/development/development-process/design-proposals/islamic-calendar-types

To support all of these variants, an ISLAMIC calendar escape could be suffixed by the variant and epoch.

Are you saying that you must specify the variant, or that you can specify the variant. i.e.

if the variant and epoch are unknown, can we use ISLAMIC?
if the variant is known and epoch is unknown, can we use ISLAMIC_I, ISLAMIC_II, ISLAMIC_III, or ISLAMIC_IV?
if the variant is unknown and epoch is known, can we use ISLAMIC_A or ISLAMIC_C?

I feel this is something to be discussed.
The underlying issue is that if an Islamic date is to be used accurately in a calculation (for instance age at an event, determination of an anniversary), then the exact variant and epoch must be known, and therefore specified.  If the variant and epoch are not known, then interpretation of an Islamic date may be out by up to three days (±1 day for the epoch, ±2 days for leap year variations).
If the both variant and type are known, then one of the 8 tags in the proposal would be appropriate as it allows for exact interpretation of the date.
If this level of accuracy is not required or the type / variant is not known in the source text, then a single ISLAMIC calendar escape would seem more appropriate, with a caveat that interpretation of such a date may differ by upto 3 days.  I don't propose additional tags for epoch or variant unknown.  Such dates would still only be interpretable to the nearest 1 or 2 days and would not really be much more useful than a generic ISLAMIC tag.

The underlying issue is that if an Islamic date is to be used accurately in a calculation

If we need to convert accurately between calendars, then we also need to know whether the event occured before or after sunset - Jewish/Islamic calendars start the new day at sunset, Gregorian calendars start the new day at midnight.
An alternative approach might be to combine the variant/type into the epoch marker:

AH after Hijra (unknown variant/type - optional/default)
AH-III-A after Hijra (known variant/type)
BH before Hijra (unknown variant/type)
BH-IV-C before Hijra (known variant/type)

If we need to convert accurately between calendars, then we also need to know whether the event occured before or after sunset - Jewish/Islamic calendars start the new day at sunset, Gregorian calendars start the new day at midnight.

This is true, and another source of potential uncertainty in date interpretation.

An alternative approach might be to combine the variant/type into the epoch marker:

I feel uncomfortable with the idea of putting this information in the epoch marker. The epoch marker is really just an era name.  In my view, it has no relationship to leap year rules and is not intended to directly convey what the epoch start date is.  These things are constrained by the calendar.
My proposal is based on the principle that it is the role of the calendar tag to completely define the rules, arithmetic, and epoch(s) of the calendar.  When the calendar tag is used in this way, it allows parsers to determine evrything it needs to know about the calendar by examining a single tag (which also happens to be the first part of the date payload).  This makes parsing much easier / efficient.


Script to remove new weather app in task bar

Description
New windows 10 version add a weather app in bottom right of the task bar.
After applying your script in CLI mode I saw this weather app was not removed.
Please find below a powershell I used to remove it (after reboot)
Reason
Just remove uneeded noise of weather :)
Examples
$registryPath = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Feeds"
$Name = "EnableFeeds"
$value = "0"
if (!(Test-Path $registryPath)) {
New-Item -Path $registryPath -Force | Out-Null
New-ItemProperty -Path $registryPath -Name $name -Value $value -PropertyType DWORD -Force | Out-Null
}
else {
New-ItemProperty -Path $registryPath -Name $name -Value $value -PropertyType DWORD -Force | Out-Null
}

This tweak you said is already on the script, BUT, only on the current user, i'll modify to that, as it changes on all users (local machine)
After that all users will hide News and Interest
Old key:
Set-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\Feeds" -Name "ShellFeedsTaskbarViewMode" -Type DWord -Value 2

New key:
Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Feeds" -Name "EnableFeeds" -Type DWord -Value 0

Thanks!

This part should be explained too, after doing this tweak, restarting explorer.exe, the user cannot bring back by just right-clicking the taskbar as it hides the option to News And Interest.
But it can be undone by using the "Undo Tweaks" option through GUI


quick-start 增加镜像下载链接

quick-start 中解决镜像从哪里来的疑惑 0_0

Please use English PR title

Please use English PR title

done ^_^

We suggest to pull docker from dockerhub, download from website is also ok, you can resolve the conflict and commit again.


React attempted to reuse markup in a container but the checksum was invalid

I have this error while trying to use Next with redux and redux saga.
We use it exactly the same way as the example 'with-redux-saga', and we dispatch an action that triggers a saga in the getInitialProps function:
static async getInitialProps ({ query, store, isServer }) {
    store.dispatch(ProductsActions.getCategories())
}

When I query the page, it is rendered with the categories displayed, then they disappear and this error is printed in the console...
It all works good when I get the categories through an async await and return them in the props.
More specifically the error reads:
(client)  data-reactid="26">

<li style="flex-grow:

So if I guess correctly the client expects an empty ul meaning NO categories and the server renders the list of categories
Versions:

nextjs: 3.2.2
next-redux-saga: 1.0.1
next-redux-wrapper: 1.3.4
react: 15.6.1
react-redux: 5.0.6
redux: 3.7.2
redux-sagas: 0.15.6

Is the store in the client the same of the server ? if that's the case you should use a store per request in the server and a single store in the client instead. And if you're using a different store for the client then the dispatch doesn't work as it's going to happen only server side


Missing Azure Video Indexer Icon

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Is your feature request related to a problem? Please describe.
No , Just want to point to missing icons
Describe the solution you'd like

The Icons of Several services in Azure is missing
You can get the latest Icon sets here : https://learn.microsoft.com/en-us/azure/architecture/icons/
Please onbaord Azure Video Indexer Icon if Possible ( attached SVG)

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

any updates on this issue ?

No, otherwise there would be a comment in the issue stating what the update is.


Andorid app subscribe to a new topic with intent does not work properly

          https://github.com/binwiederhier/ntfy-android/pull/24/files implements this:

ntfy://myhost.lan/mytopic --> subscribes to https://myhost.lan/mytopic (if not already subscribed) and shows it
ntfy://myhost.lan/mytopic?secure=false --> subscribes to http://myhost.lan/mytopic (if not already subscribed) and shows it

I think this is all there is to it. All that's missing is documentation for it.

To test:
adb shell am start -W -a android.intent.action.VIEW -d "ntfy://<IP_ADDRESS>/abc?secure=false" io.heckel.ntfy

Originally posted by @binwiederhier in https://github.com/binwiederhier/ntfy/issues/20#issuecomment-1095836942
Hi!
First of all, thank you for your work! :)
I am checking ntfy-s functionality and found this issue:
In android app if I add/subscribe manualy (plus button) a new topic , I can get new messages and send test message. But if i do it with (subscibe) intent it creates the topic on the app's list and shows toast message  "subcribed to xy chanel" and also loading/downloading "old" messages (I guess it is with the default time limit) , but not receiving any new messages, and do not show subcribed tray icon, if it is the only topic in the list. (Needs to be one with manual adding from app)
Restart the app does not change this.
I only tried this with adb shell, but would like to use with intent from my app, so I thing will be the same.
Best regards
Gabor

+one (maybe just question)
how can I use in intent different port to 80.
For me does not work this way:
adb shell am start -W -a android.intent.action.VIEW -d "ntfy://<IP_ADDRESS>:99/Test?secure=false" io.heckel.ntfy
":99" part it seems is ignored
(still not works the subscribe but shows in the app list without specific port)

Also would be nice +1
use case is i want to be able to send users topics from a webui and text or email them a subscribe link that goes right to their ntfy app


Handle position with a custom class instead of string

Add datapath class to replace the 'dotted string' used to indicate the path in the data/schema

Fixes issue with nested maps in an include, closes #56
Allows top level validators other than fixed map, brought up in the now closed issue #40

Wow, thanks! This initially looks good. I'll need a little more time to go through the whole PR, I'm pretty swamped this week. Can you please fix the PEP8 formatting in the meantime?

Thanks for the comments! The code actually became a bit simpler when I fixed the mutable default arguments issue..

I only looked a little bit at this and haven't taken the time to fully think about it yet. Do you have an understanding how issue #27 would be fixed with this new structure?

Yes, I have been working on a follow-up PR that would fix #27 and #37. The idea I have been working on there is to, instead of flattening the path, keep the lists and maps encountered when processing the schema. Then when traversing the schema the list/map keys can be compared with the ones in the data to see that there are no additional keys present.

That's great, I look forward to seeing what you come up with!
As you make changes, please run a performance comparison between the original code and the new changes. We run this on tens of thousands of yaml files hundreds of times per day...so performance is important for us.
And thanks for your contributions!

@drmull thanks. really looking forward to this fix.

@mildebrandt
Do you want me to update this PR with the strict mode code or push it as a separate PR? I still got some cleanup and documentation left so not 100% yet..
I do not have access to any large enough data files to do a proper performance comparison. Do you have the possibility to help with that?

Since it's building on top of this work, it might be easier to review if it's all together. So you can use this PR.
Sorry, I don't have anything I can share. I can create something for you next week. If you get to it before I do, I think a schema with a few includes and nested lists/maps of 3-4 levels would be sufficient. The total yaml file size should be around 50kB.

Finally had some time to do the update..

Added strict mode to solve issue #27
Added possibility to include validators as requested in #37

The first commit should also fix #62

Thanks so much! I'll try to carve out some time to look at this.

There's an exception when I try to validate this yaml:
field1:
  field1_2:
    field1_2_1: value
field2: value

with this schema:
field1:
  field1_1:
    field1_1_1: str(required=False)
  field1_2:
    field1_2_1: str(required=False)
field2: str()

The error is:
Error!
Schema: m1.schema
Data file: /Users/chrism/workspace/drmull/Yamale/m1.yaml
Traceback (most recent call last):
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 75, in _validate_item
    data_item = data[key]
KeyError: 'field1_1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chrism/workspace/drmull/Yamale/yamale/command_line.py", line 29, in _validate
    yamale.validate(schema, data)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/yamale.py", line 38, in validate
    schema.validate(d, path, strict)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 56, in validate
    errors = self._validate(self._schema, data, path, strict)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 98, in _validate
    strict)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 144, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 84, in _validate_item
    return self._validate(validator, data_item, path, strict)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 98, in _validate
    strict)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 144, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/drmull/Yamale/yamale/schema/schema.py", line 78, in _validate_item
    if validator.is_optional:  # Optional? Who cares.
AttributeError: 'dict' object has no attribute 'is_optional'

Can you please also add a command line argument for strict mode? Thanks!

Good comments. Fixed the exception problem and added the strict mode flag, also added test cases for both issues.

Thanks for all the hard work @drmull! There was one unexpected change which will cause an incompatibility with previous versions, but we think that may be the right thing in the long run.
In the schema and yaml sample I provided in my comments, all the children of field1_1 were marked as optional which made field1_1 also optional in the current yamale versions. This is probably a side-affect that we didn't want. Your PR changed this behavior, and now it'll fail unless the parent is also marked as optional.
When we merge your PR, we'll be bumping the version to 2.0 since it breaks existing validations. I don't have a date when we'll merge and create the new release, but we are excited for strict mode....so it won't be too long. :)

In the schema and yaml sample I provided in my comments, all the children of field1_1 were marked as optional which made field1_1 also optional in the current yamale versions. This is probably a side-affect that we didn't want. Your PR changed this behavior, and now it'll fail unless the parent is also marked as optional.

Hm, I see. I missed that it worked that way. Spontaneously I think it will be a bit clunky to add that behavior in the code as it looks now so if you are OK with the change as it is then I am happy :+1:


Add Lintier - CLI to quickly scaffold an ESLint & Prettier setup in a…

This PR proposes adding a link to Lintier, which is a CLI to automate setting up an ESLint & Prettier setup in a TypeScript project. Lintier provides options to install the airbnb styleguide, stylelint for linting styles, and lint-staged for linting pre-commit. Lintier installs the dependencies, creates the config files, and adds linting scripts to a project's package.json file.  Here's a gif:

I also added a section for installation and setup for this bullet point.  It didn't seem to fit in with the other tools in the "tools" section.
Thanks for your consideration!  Please let me know if you have any questions.
-Josh

Hey @dustinspecker -- just wanted to check in and see if there was anything else you needed from me on this.  And thanks for all your work maintaining this list!

This looks cool! Sorry about the wait for a review/merge.

Thanks! and no worries


fix: AdLibs in playlist

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bug fix

What is the current behavior? (You can also link to an open issue here)

Only adLibs from the current rundown can be taken. AdLibs from future or previous rundowns cannot be started.

What is the new behavior (if this is a feature change)?

AdLibs from any rundown can be taken as long as the rundown they originate from has the same showstyle variant as the current rundown.

Other information:

Status

[X] Code documentation for the relevant parts in the code have been added/updated by the PR author
[X] The functionality has been tested by the PR author
[ ] The functionality has been tested by NRK

Codecov Report

Merging #555 (2949725) into release37 (0c88272) will increase coverage by 15.04%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           release37     #555       +/-   ##
==============================================
+ Coverage      67.30%   82.34%   +15.04%
==============================================
  Files            259       12      -247
  Lines          17780     1008    -16772
  Branches        4026      213     -3813
==============================================
- Hits           11966      830    -11136
+ Misses          5775      177     -5598
+ Partials          39        1       -38

Impacted Files
Coverage Δ

meteor/server/systemStatus/systemStatus.ts

meteor/server/migration/1_5_0.ts

meteor/server/api/packageManager.ts

meteor/server/api/client.ts

meteor/server/lib.ts

meteor/lib/collections/ShowStyleBases.ts

meteor/server/api/userActions.ts

meteor/server/security/peripheralDevice.ts

meteor/server/migration/0_18_0.ts

meteor/server/codeControl.ts

... and 236 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c88272...2949725. Read the comment docs.


Make default OAuth 2.0 client redirect URL be http://localhost:8080 for development

Right now it defaults to http://localhost:8081. It can be anything, but usually the client listens on http://localhost:8080.

@mcovalt, where does that default come from?

https://github.com/atlascope/atlascope/blob/dd583f2f03da2c13b98112c60e7f537e49e0c5f3/atlascope/core/management/commands/populate.py#L24


tree-sitter-grammars: add typst

Description of changes
Introducing typst to nixpkgs tree-sitter grammars packages.
Why this specific package? Because the emacs tree-sitter package for typst typst-ts-mode depends on it.
Not entirely sure weather it would be classified as a solution for #235938 due to the existence of many variants of tree-sitter-typst. But I figured that it would be worth giving it a mention.
Reference

https://github.com/uben0/tree-sitter-typst

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Fails to build and breaks tree-sitter.allGrammers.
copying path '/nix/store/0wmfpsr1m6fvxxmh1p19bpasywd091rs-tree-sitter-typst-b8f3ac3' from 'https://cache.nixos.org'...
unpacking sources
unpacking source archive /nix/store/0wmfpsr1m6fvxxmh1p19bpasywd091rs-tree-sitter-typst-b8f3ac3
source root is tree-sitter-typst-b8f3ac3
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
no configure script, doing nothing
building
src/scanner.c:1:10: fatal error: tree_sitter/parser.h: No such file or directory
    1 | #include <tree_sitter/parser.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

@mweinelt noticed that I had made a wrong assumption about the package status by mixing it with another package that I was working on at that time. Therefore I wrote #259779 to counter this issue.

A major problem here is that the tests are not running, which would have surfaced these build failures.

I've opened https://github.com/NixOS/nixpkgs/pull/260027 to fix this on darwin too.


TypeError: Cannot destructure property 'iife' of 'undefined'

Running into this error on the latest 0.3.4 version
TypeError: Cannot destructure property 'iife' of 'undefined' as it is undefined.
    at jsxInclude (/***/node_modules/vite-cep-plugin/lib/index.js:328:23)

this is an upgrade from the previous version, not a clean install if that helps

Set iife to true in your vite.es.config.ts :
      jsxInclude({
        iife: true,
        globalThis: GLOBAL_THIS,
      }),


Allow interop between Firefox and Chrome

This fixes an interop issue between Firefox and Chrome, namely, in the case where Firefox was the offerer, and Chrome the answerer.
Chrome would attempt to process the ICE candidates prior to the remote description being set, resulting in the failure to apply the candidate.
The priorities of the tasks should now reflect the expected flow.

@DamonOehlman Any thoughts on this? The change seems to restore Firefox <-> Chrome interoperability but if you can remember - was there any particular reason behind the original ordering that might cause issues?

@nathanoehlman Looks good.  But I think what is happening now is starting to herald the potential end of the rtc-taskqueue package's usefulness.  With firefox taking WebRTC seriously now, and the browsers doing a better job of ensuring they are in the appropriate ready state (or deferring operations) until they are I think you should probably look at resimplifying rtc-tools.  This may well mean deprecating and removing rtc-taskqueue.
Also, I know bandwidth is tight, but I think interop tests are probably becoming a higher priority again.  I'm still hoping that @thepatrick will manage to get permission what Tokbox have been using behind the scenes, so that it can be used by others :)

@DamonOehlman Yep - I agree, I think returning to a simpler implementation is definitely going to have to be on the roadmap for the near future - I'll see if I can find some time at some point

Microsoft edge just released ortc support - more shims necessary.


Feature request: Adding a Helm chart

Feature Request: Adding a Helm Chart
First of all, big thanks for the work done so our Talos clusters work perfectly with our existing vSphere infrastructure!
Reason for feature request
Whenever a new version of any software I deploy releases, I need to manually download the manifests and change whatever needs changing to cater to my personal needs (subnets, nodes, ...). Lately I've been deploying as much as I can using Helm charts so I can more easily make changes to a deployment without all the manual work.
Now, I know that for this application to have a Helm chart isn't really needed (as it's only 1 DaemonSet), but having the flexibility of having a Helm chart towards the future could be a nice-to-have.
Work already done
I have forked this repository and made the following changes to the master branch:

Added a helm subfolder with all files that make up the Chart (templates, Chart.yaml, README.md, ...)
Added a new workflow (helm.yaml) that builds a new release every time a tag is created (usually on every release).

The workflow only triggers on tag releases, not on commits. the ghcr.yaml doesn't trigger on changes to any file in the helm subfolder (as that wouldn't change the application, therefore not needing a rebuild).
After building the Chart, the archive is published to a gh-pages branch and deployed using GitHub pages (this branch is created the first time an archive is published) using the https://mologie.github.io/talos-vmtoolsd URL.

Follow-up
If you accept the request, I will create a pull request for review to integrate these changes into the master branch.
Please note:

An archive can't be made for the 0.2 release. So if you wanted to retro-actively make one:

A tag needs to be made to trigger the workflow to create one (eg. 0.2.1), no new release is needed. A container rebuild wouldn't be needed either, so the ghcr.yaml workflow could be manually cancelled.
The Helm templates files need to be adjusted to reflect the tag version (changing version to 0.2.1, but leaving appVersion intact).

For when the 0.3 version is ready for release, the same files (the helm subfolder and the helm.yaml workflow) would need to be added. I've added those already to my forked develop-0.3 branch, so a pull request could be made to easily make those changes as well.

For any subsequent version, no changes would be needed and everything would be done automagically.

Hi, thank you for the request, I really appreciate your effort to improve the project. However, as you have already noted, there is little benefit in adding a Helm chart to talos-vmtoolsd.
The project is designed to be free of any configuration, and even provisioning of Talos' config will eventually no longer be required. This is tracked by #3. As such, Helm's primary benefit of making application configuration more accessible does not really work for this project in current of future state.
At Authentic Vision we deploy talos-vmtoolsd to our clusters using Kustomize, which applies a strategic merge patch that adds tolerations to the DaemonSet. Helm does not provide such flexibility without explicitly programming it, so I would expect a majority of the Helm file to include options not related to talos-vmtoolsd's primary functionality if one is added.
If automatic deployment to a large set of clusters is your aim then tools such as ArgoCD can perform the modification and deployment task for you without involving Helm.
Note regarding your Helm chart itself: It appears to drop the image hash and adds an imagePullPolicy=always line; you likely do not want such a configuration as it could result in unexpected cluster changes if the image behind the 0.2 tag changes (not that it ever will, this is more of a general note).


[Hero]Black Hand

Mele, strengh, long cast times ~0.75-1.5 . (suggested base model Wraith King)
Q [Long shot] Throws projecktail. Deals magic damage and x% rightclick damage as phisical. Projecktail moves slowly, around 290 speed, long range. Around 0.75 cast animation on max lvl.

visualy something like rianheart abbility from overwatch or raven secondary ultimate usage from league of legends

W [Earth Shock] Small AoE in front of caster, slows, deal damage.
E [Tount] [Pierce spell immunity] Force target to attack caster, target attack damage is reduced but have higher attack speed. Attack efeckts can't triger when taunted.  Target is slowed and disarmed afterwards.

after effeckts are not dispelable, they apire even if it was dispelled. Also turns off  evry skills that grants attack modificators like frost arrow, bash, medusa's q, drow aghanim etc.

R [Power Strike] [Pierce spell immunity] Deals damage. If target dies from that spell evry nerby enemy takes full damage dealt to target.

secondary targets get full damage dealt to target not damage taken by target, for eg, abbility can deal 1000 dmg but target have 100 health, target dies by takeing 100 dmg but secondary targets take 1000 dmg not 100.
Damage_Dealt = Base_Damage + Stack_Damage * DarkPresence_Stack_Value.

F [Dark Presence] [Passive] Spells and evry x attack apply debuff on target/hited enemy hero. Debuff reduce reducktions, stacks. Evry stack on enemy gives "caster" cooldown reducktion.
Aghanim Scepter upgrade: [Dark Presence] Enemy heals also heal you for x% value. Have chance to get enemy AoE buff when casted. After acktivation gain all enemy aura's effeckts, need to be in range on them.

work also with mek, arcane boots, pipe, drums etc. lvl 10 talent enchance also effeckt's gained that way.

Tree
lvl 10 incrise item possitive effeckts on you. / Decrese all abbility cast times.

auras, acktives and passives excluding stats. /...
lvl 15 +x% bonus health from heals taken / +x mana regen for evry Dominant Presence stack
lvl 20 x% stat reducktion of Tounted unit / Tounted hero is forced to get in to mele range
lvl 25 Killing enemy with ultimate transfers ~1/3th his Dark Presence stacks to enemies around / +4 Long Shot Projecktails
.../ projecktails are unlished at the same time in front of caster, so can be possible hit one target with all 5 of them when very close to enemy. Projecktails are traveling symetricaly to base one at around 20* and 35*.

(i can write Lore and flavored spell discription)

#393


something is wrong with codespace: process 1918 stderr: Could not execute because the specified command or file was not found.

Extension started for VS Code Stable.
create-tool-manifest: Executing [dotnet new tool-manifest] in '/home/vscode/.vscode-remote/data/User/globalStorage/ms-dotnettools.dotnet-interactive-vscode'.
create-tool-manifest: Finished with code 0.
create-tool-manifest: STDOUT:
An issue was encountered verifying workloads. For more information, run "dotnet workload update".
The template "Dotnet local tool manifest file" was created successfully.
tool-uninstall: Executing [dotnet tool uninstall Microsoft.dotnet-interactive] in '/home/vscode/.vscode-remote/data/User/globalStorage/ms-dotnettools.dotnet-interactive-vscode'.
tool-uninstall: Finished with code 1.
tool-uninstall: STDERR:
Cannot find a manifest file that contains package id 'microsoft.dotnet-interactive'.
tool-install: Executing [dotnet tool install --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json --ignore-failed-sources Microsoft.dotnet-interactive --version 1.0.522904] in '/home/vscode/.vscode-remote/data/User/globalStorage/ms-dotnettools.dotnet-interactive-vscode'.
tool-install: Finished with code 0.
tool-install: STDOUT:
You can invoke the tool from this directory using the following commands: 'dotnet tool run dotnet-interactive' or 'dotnet dotnet-interactive'.
Tool 'microsoft.dotnet-interactive' (version '1.0.522904') was successfully installed. Entry is added to the manifest file /home/vscode/.vscode-remote/data/User/globalStorage/ms-dotnettools.dotnet-interactive-vscode/.config/dotnet-tools.json.
Started process 1918: dotnet /workspaces/interactive/artifacts/bin/dotnet-interactive/Debug/net8.0/Microsoft.DotNet.Interactive.App.dll notebook-parser
Starting kernel for 'file:///workspaces/scripting-notes/notebooks/en/traffic-through-AzureFW-plotly.ipynb' using: dotnet /workspaces/interactive/artifacts/bin/dotnet-interactive/Debug/net8.0/Microsoft.DotNet.Interactive.App.dll [vscode] stdio --working-dir /workspaces/scripting-notes/notebooks/en
Kernel for 'file:///workspaces/scripting-notes/notebooks/en/traffic-through-AzureFW-plotly.ipynb' started (1920).
process 1918 stderr: Could not execute because the specified command or file was not found.
process 1918 stderr:

kernel (1920) stderr: Could not execute because the specified command or file was not found.
kernel (1920) stderr:

Process 'dotnet' with PID 1918 exited with code 1 and signal null
Started process 1956: dotnet /workspaces/interactive/artifacts/bin/dotnet-interactive/Debug/net8.0/Microsoft.DotNet.Interactive.App.dll notebook-parser
Kernel for 'file:///workspaces/scripting-notes/notebooks/en/traffic-through-AzureFW-plotly.ipynb' ended (1920) with code 1.
process 1956 stderr: Could not execute because the specified command or file was not found.
process 1956 stderr:

@eosfor ➜ /workspaces/scripting-notes (feature/codespaces) $ dotnet --info
.NET SDK:
 Version:           8.0.204
 Commit:            c338c7548c
 Workload version:  8.0.200-manifests.9f663350

Runtime Environment:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.204/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.4
  Architecture: x64
  Commit:       2d7eea2529

.NET SDKs installed:
  8.0.204 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

This comes from the repo here https://github.com/eosfor/scripting-notes/tree/feature/codespaces. Basically I took devcontainer.json and Dockerfile from the main branch of this repo and it, somehow, does not work...

Fixed it by updating this way

Fixed it by updating this way


Clarify zero-pixel intersection behavior

I'm confused by this test:
    it('handles targets on the boundary of root', function(done) {

      var spy = sinon.spy();
      io = new IntersectionObserver(spy, {root: rootEl});

      runSequence([
        function(done) {
          targetEl1.style.top = '0px';
          targetEl1.style.left = '-21px';
          targetEl2.style.top = '-20px';
          targetEl2.style.left = '0px';
          io.observe(targetEl1);
          io.observe(targetEl2);
          setTimeout(function() {
            expect(spy.callCount).to.be(1);
            var records = sortRecords(spy.lastCall.args[0]);
            expect(records.length).to.be(1);
            expect(records[0].intersectionRatio).to.be(0);
            expect(records[0].target).to.be(targetEl2);
            done();
          }, ASYNC_TIMEOUT);
},

Neither targetEl1 nor targetEl2 contain any pixels that intersect with rootEl. Why does the test expect an intersection observer callback for targetEl2?
cc @tobytailor

In particular, the spec says:

callback will be invoked when intersectionRect’s area changes from greater than or equal to any threshold to less than that threshold, and vice versa.

and I don't understand how this justifies treating the two elements differently.

FYI, the discussion around this behavior is captured in #69.

Well, somewhat... as far as I can tell, the discussion does not mention requiring browsers to fire the callback when the element becomes adjacent to an edge. Even an isIntersecting flag would be false for both targetEl1 and targetEl2, wouldn't it?

This comment in that thread describes the updated behavior:

We think the spec should change to just invoke an inclusive intersection (includes the boundary points in the intersection).

In the test case you referenced above, targetEl2 is on the boundary (though not overlapping) and targetEl1 is not on the boundary (and also not overlapping). The use of inclusive intersections explains why an intersection is expected for targetEl2.

Ah, thank you. It wasn't clear to me whether that was just one idea of many, or an actual decision.
I'd appreciate the spec being updated, so that the tests can actually test specced behavior.


Enterprise Features

In the readme:

Consider purchasing a license from Ag-Grid if you are going to use enterprise features!

What are these enterprise features that are in the streamlit-aggrid?

The main additional enterprise features are: row grouping, tree data and aggregation.
One very useful feature that I miss in the community version is the ability to copy the cell content (by using an option in the context menu, witch is only available with an enterprise license).
More information (and a feature comparison between Ag-Grid community and enterprise licenses) can be found in:
https://www.ag-grid.com/angular-data-grid/licensing/


gsutil cat intermittently fails

I keep getting the following error message randomly when using webdataset with gsutil cat.  I have num_workers=4 in the dataloader, so it seems unlikely to be too many requests.  Any suggestions?  It would be nice to at least get a more useful error message.
Exception: ("((' gsutil -o GSUtil:parallel_process_count=1 -o GSUtil:parallel_thread_count=1 cat gs://dc-erik-data/210228_213525/data/v3/train/20.tar',), {'shell':
 True, 'bufsize': 8192}): exit 1 (read) {'nodeinfo': ('erik-dc', 7595), 'rank': -1, 'size': -1, 'worker_id': 0, 'num_workers': 4} @ <Pipe ((' gsutil -o GSUtil:para
llel_process_count=1 -o GSUtil:parallel_thread_count=1 cat gs://dc-erik-data/210228_213525/data/v3/train/20.tar',), {'shell': True, 'bufsize': 8192})>", <webdatase
t.gopen.Pipe object at 0x7fce1cc86e80>, {'__url__': 'pipe: gsutil -o GSUtil:parallel_process_count=1 -o GSUtil:parallel_thread_count=1 cat gs://dc-erik-data/210228
_213525/data/v3/train/20.tar', '__worker__': '(0, 4)', '__rank__': 'None', '__nodeinfo__': "('erik-dc', 7595)"})

I'm not sure why gsutil would fail. Does it fail if you just run gsutil cat ... > /dev/null? Does it fail if you use zero workers?
You can use arbitrary scripts as commands. I recommend writing a small wrapper script that logs gsutil output and stderr to some log file and then looking at it. You can also use your wrapper script to ignore/handle errors.
If you don't need gsutil authentication, you can also access Google Cloud Storage through curl -s -L http://storage.googleapis.com/bucket/object.
PyTorch's use of multiprocessing in DataLoader makes it generally quite difficult to debug errors in Dataset. We have a better solution based on ZMQ; I'm looking at integrating that.

I also have the same error with S3 every now and then. I tried using s3cmd, awscli and s5cmd
Exception: ("(('s5cmd --numworkers 1 cat s3://datasets/test/shard-0005087.tar',), {'shell': True, 'bufsize': 8192}): exit 1 (read) {'nodeinfo': ('model-training-qxx5t-68847550', 10), 'rank': -1, 'size': -1, 'worker_id': 31, 'num_workers': 64} @ <Pipe (('s5cmd --numworkers 1 cat s3://datasets/test/shard-0005087.tar',), {'shell': True, 'bufsize': 8192})>", <webdataset.gopen.Pipe object at 0x7fa8962f9d90>, {'__url__': 'pipe:s5cmd --numworkers 1 cat s3://datasets/test/shard-0005087.tar', '__worker__': '(31, 64)', '__rank__': 'None', '__nodeinfo__': "('model-training-qxx5t-68847550', 10)"})
and the biggest issue is this errors are somehow not retriable, training just errors out.

If you are getting errors in a subprogram and want to ignore them and simultaneously log the errors, just use something like this:
url = "pipe:s3cmd cat ... 2> /tmp/iolog.$$ || true"
dataset = wds.WebDataset(url, handler=wds.ignore_and_continue). ...

If you have even more complicated logic for error detection, logging, and/or recovery, just put them into a script:
url = "pipe:/usr/local/myscript.sh s3://bucket/object-%06d.tar"
dataset = wds.WebDataset(url, handler=wds.ignore_and_continue). ...

That log information should help you see what's happening.
It looks like you are running single node, single GPU, with 64 workers. That's probably too high: I suspect you are getting timeouts because of it, either hitting throttling from Amazon or overwhelming the local machine The num_workers should only be as high as you need to keep the GPU busy. Try starting with something like num_workers=4 and then increasing only if necessary to keep the GPU busy.
Where are you running your training job? On AWS or locally?

let me try with script. I eseentially want to retry when this happens.
and I am using a pretty beefy machine. I was running on a p3.16xlarge instance.

I'm also having issues reading datasets from gs buckets with wds via gopen/pipe (currently on main branch 0.2.3).
I'm training on TPU VM instances, 8 train processes, 4 workers per process on ImageNet-1k/22k and similar large(ish) datasets.
I was hoping to switch from TFDS to WDS for cloud training due to simplicity of WDS and much fewer dependencies (ie not needing all of tensorflow + tfds) when I'm using PyTorch. With TFDS, on same machine types, with same # worker processes, I have working TFDS wrapper that has hundres of training days with maybe one or two instances of gs read failures that aborted train (and very few retries logged). So, while the failure clearly isn't in WDS code, there seems to be reliability issues (or failure handling) using gsutil cat that aren't happening in TFDS.
Has anyone experiencing this issue come up with a robust solution?
The error I'm typically seeing is
File "/snap/google-cloud-sdk/229/platform/gsutil_py2/gslib/vendored/oauth2client/oauth2client/transport.py", line 282, in request
  connection_type=connection_type)
File "/snap/google-cloud-sdk/229/platform/gsutil_py2/third_party/httplib2/python2/httplib2/__init__.py", line 2192, in request
  cachekey,
File "/snap/google-cloud-sdk/229/platform/gsutil_py2/gslib/gcs_json_media.py", line 453, in OverrideRequest
  headers)
File "/snap/google-cloud-sdk/229/platform/gsutil_py2/gslib/gcs_json_media.py", line 668, in _conn_request
  response = conn.getresponse()
File "/snap/google-cloud-sdk/229/platform/gsutil_py2/gslib/gcs_json_media.py", line 369, in getresponse
  orig_response = http_client.HTTPConnection.getresponse(self)
File "/usr/lib/python2.7/httplib.py", line 1178, in getresponse
  response.begin()
File "/usr/lib/python2.7/httplib.py", line 452, in begin
  version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 416, in _read_status
  raise BadStatusLine("No status line received - the server has closed the connection")
httplib.BadStatusLine: No status line received - the server has closed the connection

and then
File "/home/xxxx/.local/lib/python3.8/site-packages/webdataset/gopen.py", line 86, in read
   self.check_status()
 File "/home/xxxx/.local/lib/python3.8/site-packages/webdataset/gopen.py", line 66, in check_status
   self.wait_for_child()
 File "/home/xxxx/.local/lib/python3.8/site-packages/webdataset/gopen.py", line 81, in wait_for_child
   raise Exception(f"{self.args}: exit {self.status} (read) {info}")
Exception: ('(("gsutil cat \'gs://xxxx/imagenet22k-train-0082.tar\'",), {\'shell\': True, \'bufsize\': 8192}): exit 1 (read) {} @ <Pipe (("gsutil cat \'gs://xxxx/imagenet22k-train-0082.ta
r\'",), {\'shell\': True, \'bufsize\': 8192})>', <webdataset.gopen.Pipe object at 0x7f15155425e0>, 'gs://xxxx/imagenet22k-train-0082.tar')

handler=wds.ignore_and_continue works well when reading from gcs. Did you use that @rwightman ?
We've been training while reading from gcs without issues for days (from TPUs)

@rom1504 that's a good work-around for now, thx. I've enabled warn_and_continue so I can track the occurences and will see how that goes. If it's happening too frequently I'll likely need to invest some time in a retry mechanism, as it drops the whole shard. Not too big of a concern with the larger datasets, but not ideal.

@rom1504 I've set that up and had it running and it keeps chugging along now, but it's concerning how many failures I see and the variation ... failing at pretty much any point across the http connection for the gsutil cat (from connection refused, initial header / SSL exchange, to part way through the transfer).
I find this odd because I don't see equivalent failures in TFDS use cases, I wonder if the spacing / nature of the http requests done this way is somehow more bursty. I don't feel I have an abnormally large number of worker processes for keeping 8 TPU cores fed (only 4 per core here, I actually need more for smaller models to avoid bottlenecking) ....

How often do you see failures?
Sounds like an issue in the gsutil command.
Maybe there are some retry/robustness options in that command?

I suggest trying gsutil cp your_file - instead of gsutil cat and then playing with the retry option mentioned at https://cloud.google.com/storage/docs/gsutil/commands/cp

https://cloud.google.com/storage/docs/retry-strategy#gsutil looks very relevant as well

@rom1504 yes, I believe TFDS is using curl (c-lib) directly in C++ code. Some transport errors are handled as retries (not sure if all retries are logged or just some). I do know some resumable failures are logged and some can abort training, Logged failures I see maybe once every few weeks, and I've only had failures that abort training a few times (and that was during a service level issue for gs storage).
In contrast to gsutil cat from the worker processes they're getting transport failures every few hours...

@rom1504 I'm going to try the streaming cp, retries are enabled by default (6 with exponential backoff), but I don't think that applies to cat...

I guess one more option would be to use curl cli like this https://stackoverflow.com/a/64827955/1658314 and use curl retry options. But it's significantly less nice

@rom1504 FYI, gsutil cp doesn't behave any differently. Using curl CLI doesn't look particularly fun. There is a streaming blob open in the Pythong google-storage API now, so I might take a look and see if that is less problematic

Also looking at this more, not sure if this is a main branch issue, but I have a rather insane number of gsutil proceseses lying around (don't seem to be cleaned up as they are growing with time, 200+ on one instance, 500+ on another that's been going longer) well out of sync with 8 * 4 workers. And also have > 1000 sockets related to gs downloads that are in TIMED_WAIT .. hmmmm

Update:

There definitely seems to be a process leak for the gsutil laucnhes, overnight I accumulated 2000+ gsutil related processes on one cloud machine
I ran a quick test with the GOPEN_VERBOSE enabled and within an epoch it looks like the pipes are being destroyed, so perhaps the leak is in epoch transitions? or specific cases
Sockets are steady, at around 1300 (but majority) timed wait as mentioned), each file is likely one to a few connections and timed wait state on Linux is fairly long default. So this is expected. Reusing connections would be ideal, but not possible through current mechanism
I hacked in the google-cloud-storage API (as per below) and it seems to work better re failures, none overnight but does have an occasional pause that pauses train for a few seoncds. It would likely be better with an async or thread based mechanism but don't think that's easy with an out of the box lib...

    pr = urlparse(url)
    if pr.scheme == "":
        bufsize = int(os.environ.get("GOPEN_BUFFER", -1))
        return open(url, mode, buffering=bufsize)
    if pr.scheme == "file":
        bufsize = int(os.environ.get("GOPEN_BUFFER", -1))
        return open(pr.path, mode, buffering=bufsize)
    # hack start
    if pr.scheme == "gs":
        client = storage.Client()
        return storage.Blob(pr.path[1:], client.get_bucket(pr.netloc)).open(mode)
    # hack end
    handler = gopen_schemes["__default__"]
    handler = gopen_schemes.get(pr.scheme, handler)
``

@rom1504 close to 36 hours now with no errors / failures using the storage API so all failures do appear to have been caused by piping gsutil via popen. Unfortunately, there does appear to be higher throughput variation and some loss of throughput (~5-8%) doing blocking network reads like this in the dataloader workers. Not clear if I can recitfy that with extra workers or different chunk sizes...

@rom1504 last update on this until I have some time to try some performance improvements... the instances that I have the Python client running on have been running since last comment without a single error.
For this sort of multi-week training the extra drag isn't acceptable, so I built a TFDS version of the dataset and continuing with that for now despite the mess and high memory utilization that TFDS incurs over WDS. WDS is working well from my local NAS though. My local machines have less RAM than cloud VM so nice having 1/3 the memory consumption.

@rwightman interesting! Btw I don't know if you noticed but img2dataset now supports the tfrecord output format in addition to webdataset, thanks to Boris.
Should be possible to have good webdataset support in Google cloud though. I'm guessing the correct solution would be either a PR in gsutil either building another cli tool that can read from Google cloud storage, maybe using fsspec for example

I'm not sure where the problems with gsutil are coming from; we haven't seen those problems with subprocesses and I can't reproduce this. I have several possible solutions:

use http://storage.googleapis.com/bucket/shard.tar to access the data; this works authenticated or unauthenticated
turn on caching in WebDataset; with caching the data is downloaded first in one go and then just read from a local file descriptor
wrap gsutil inside a script/pipe: that copies the data first, as in gsutil cp $1 /tmp/$$.tar && cat /tmp/$$.tar

Please let me know whether this addresses your problem.
 I'll think about adding native support for storage.Client, though you lose some parallelism in the process.

@tmbdev yeah this behaviour is puzzling
The googleapis (http) could work, but I can't do unauthenticated so I'd need to hack in a token to the curl args (or use a shell script that includes it) ... I'll give that a try next. Current runs are still going so I'll wait for them to finish next week before I try more options.
caching / local filesystem isn't practical here, TPU-VM have 3x more RAM (~300GB vs 100GB) than local storage, and datasets I'm interested in for this are past 1 TB with fairly large shard sizes, already running into disk space issues with checkpoints, etc

Yes, my suggestion for caching was just to enable pre-copying; that is, if the cache size is smaller than the dataset, shards will just get downloaded, used, and then quickly discarded.
The simplest and most informative test, however, would be to explicitly pre-copy (something like "gsutil cp $1 /tmp/$$.tar && cat /tmp/$$.tar; rm -f /tmp/$$.tar").

@rom1504 @tmbdev
I'm running into a similar issue with gsutil cat or gsutil cp.
During mid training for large shards 10GB+ per shard, some network errors occur which the data loader cannot recover for WDS
Epoch 0: : 371it [2:20:27, 22.72s/it, loss=5.85, v_num=19r8]OSError: Connection reset by peer.
Epoch 0: : 380it [2:23:57, 22.73s/it, loss=5.82, v_num=19r8]/*stdin*\ : Read error (39) : premature end
Epoch 0: : 382it [2:24:43, 22.73s/it, loss=5.81, v_num=19r8]OSError: Connection reset by peer.

I suspect this is related to streaming transfers (https://cloud.google.com/storage/docs/gsutil/commands/cp#streaming-transfers) not being able to be resumed according to the docs. e.g., gsutil cp [path] - would be a streaming transfer, and if the network cuts off it has to restart from the beginning?
(Note: I'll be trying out the handler wds.ignore_and_continue, though I'm not sure if this would create incorrect training behavior where a whole shard is skipped if a network occurs, thus making the samples at the tail of the tar file less accessed and undersampled)

You don't have to use a streaming transfer; you can also use something like:
gsutil cp $url /tmp/$$ && cat /tmp$$; rm -f /tmp/$$
You can either put that directly after "pipe:" or put it into a small script, say gsdlcat:
#!/bin/bash
gsutil cp $1 /tmp/$$ 1>&2 &&
cat /tmp/dldata.$$
rm -f /tmp/dldata.$$

@tmbdev For shards of size 10GB, this would require waiting for the entire file to download prior to loading the data? I guess that's a tradeoff but it will avoid the streaming-related issues.

@tmbdev Thanks for the info - I'm running this on a server outside of GCloud (but the region is nearby). Downloading the entire object works well - the issue happens only during streaming (seems like holding that connection open for a long period of time is an issue...).

I would strongly advise against training against Google Cloud directly from outside Google's compute clusters: Google's egress costs are high; you are either costing yourself or someone else a lot of money, and any dataset you train against like that may simply go away. And, in addition, WIAN connection to Google Cloud Storage just don't seem entirely reliable.
Unless you are training on Google Compute, just enable caching in WebDataset: you'll save a lot of money and you won't have the reliabilty problems.

@rom1504 last update on this until I have some time to try some performance improvements... the instances that I have the Python client running on have been running since last comment without a single error.
For this sort of multi-week training the extra drag isn't acceptable, so I built a TFDS version of the dataset and continuing with that for now despite the mess and high memory utilization that TFDS incurs over WDS. WDS is working well from my local NAS though. My local machines have less RAM than cloud VM so nice having 1/3 the memory consumption.

@rwightman curious, were you able to improve the performance of the storage API solution? I may try it out as I am also experiencing this exact error.
One thing I also notice is that when enabling wds.ignore_and_continue, shards getting dropped results in training delays as the downloads don't immediately resume ... I do wonder if there isn't some rate-limiting going on on the Google side too.
Also curious @tmbdev what you meant by losing parallelism via the storage api. Is the google storage api just not safe for multiprocessing?

Right now, the DataLoader spawns multiple workers, and each worker spawns an I/O subprocess. The I/O subprocess communicates via UNIX pipes with the worker. UNIX pipes are a highly efficient means of communicating between the I/O subprocess and the worker.
I don't believe that the Google Cloud storage API  implementation is multithreaded or uses multiprocessing. So, if you directly run it inside of a PyTorch worker, you lose the parallelism between I/O and the worker computation. The worker could spawn a subprocess via Python multiprocessing and call the Google Cloud storage API in that subprocess; I assume that's safe, but it wouldn't be any faster than what WebDataset is doing right now.

Hello,
Is there any news on the subject ? I have the same problem.
@rwightman Did you find a clean implementation to use your hacky fix and enable retry ?
I want to avoid downloading the shard on disk since i'm just using webdataset to process samples.
Thanks !

Rereading the thread, I'm not sure what I can do about it.
As far as I can tell, WebDataset handles everything correctly: it opens the pipe to the subprocess, streams the data, and closes it. If the subprocess hangs indefinitely or gets errors, that's kind of outside of WebDataset's control.
I'm not sure how tfds addresses this better; maybe they have some special case code somewhere to deal with GCS failures or retries; but I would assume and hope that such code is also in gsutil.
Here are the options that I can see:

I may add a timeout to WebDataset so that subprocesses are killed after a certain amount of time no matter what.
You could switch to curl https://storage.googleapis.com/... to access the data.
You could use the aistore data server as a reliable proxy.

Is there any specific ask you have?
(Eventually, I may also add a multithreaded native WebDataset client written in Rust with a Python binding. But if GCS is not reliable, that by itself wouldn't address this issue.)

I just noticed the activity here @adrienpl I haven't been doing much gcp training lately but I never did find a great solution so typically stuck with TFDS when I was using TPUs. And yeah @tmbdev, it's certainly no fault of this lib, working around the intermittent failures and orphaned processes with gsutil would be a big headache and not worth the maintenance burden.
With s3 any of the popular CLI tools works great with the pipe setup up to somewhere in the ~4000+ DL worker range (before S3 itself starts bonking out and rate limiting, dropping requests) that I babysat for some OpenCLIP training at 1000+ GPUs (rare).


Testing more installation options

Some installation issues that have been noted by users:

pip is only installed if minifyjs is enabled
ispc and embree links are sometimes not working
check to see if we have to clone with --recursive or not

Fixed


Change kani-compiler to use -C llvm-args for Kani specific flags

Description of changes:
Changed how kani-driver passess arguments to kani-compiler. We now have one option, --kani-compiler, that changes how we configure rustc, and all the other Kani specific arguments are added to -C llvm-args. We no longer need to keep the arguments in a certain order or need to use KANI_FLAGS environment variable.
In the kani-compiler we no longer parse the arguments before invoking rustc, we initially detect whether users want to use the Kani flavour of the compiler by looking for a --kani-compiler flag.
If --kani-compiler flag is found, we configure rustc's driver to use our custom backend and our custom callback. We then invoke the driver with the command line arguments excluding the --kani-compiler flag. During the callback configuration we initialize the Kani specific flags that are passed via -C llvm-args.
If there isn't any --kani-compiler flag, we just invoke rustc with all the given arguments.
Resolved issues:
Resolves #2137
Related RFC:

Optional #ISSUE-NUMBER.
Call-outs:
Since we initialize the QueryDb after sharing it, I put the entire structure behind a mutex, and converted its members to simple non-synchronized values.
Testing:

How is this change tested? Existing tests

Is this a refactor change? Yes

Checklist

[ ] Each commit message has a non-empty body, explaining why the change was made
[ ] Methods or procedures are documented
[ ] Regression or unit tests are included, or existing tests cover the modified code
[ ] My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

I updated the comments. Let me know if that makes sense.


chore: update build workflow

Do we have any way to check “make_flags” is correct ？

It would be built fail if incorrect;

merged due to cooperate commands


Just asking

i tried to install a mod for my aternos server and it asked me to install other versions of the mods and i installed them but the enchdesc mod doesn't exist on aternos

For mod Enchencments

Hello.
My module is unofficial and is not related to Aternos GmbH.
You should suggest the "enchdesc" mod to the Aternos developers:

Also, check if you didn't misspell mod's name.


🙅 [PROBLEM]: install_container_service should check if install_xnat succeeded

Problem Description
install_xnat had failed for me, but I hadn't realised, this then meant that install_container_service was failing. There should be some kind of check at the start to see if XNAT is up.
Acceptance Criteria

### Related Work Items

A check would be good, or combine the two playbooks (https://github.com/UCL-MIRSG/ansible-collection-infra/issues/72#issuecomment-2016732440)


Make beluga_amcl a drop-in replacement of nav2_amcl

Description
This ticket is meant to track the progress in making a drop-in replacement of nav2_amcl in terms of interface and parameters.
Specific features and requirements will be best tracked in their own issues.
Definition of done
Complete the implementation of the following list of parameters, services, publishers and subscribers:

Parameters

ROS topics and frames

[x] base_frame_id
[x] global_frame_id
[x] odom_frame_id
[x] scan_topic
[x] map_topic
[x] initial_pose_topic (extra)

Initial pose (see #41)

[x] set_initial_pose
[x] initial_pose
save_pose_rate
[x] always_reset_initial_pose
[x] first_map_only

Publish map to odom transform (see #40)

[x] tf_broadcast
[x] transform_tolerance

Particle filter

[x] max_particles
[x] min_particles
[x] pf_err
[x] pf_z
[x] recovery_alpha_fast
[x] recovery_alpha_slow
[x] spatial_resolution (extra)
[x] resample_interval (see #57)
[x] selective_resampling (see #57)

Motion model

[x] robot_model_type
[x] alpha1
[x] alpha2
[x] alpha3
[x] alpha4
[x] alpha5 (see #56)
[x] update_min_a
[x] update_min_d

Sensor model

[x] laser_model_type
[x] laser_max_range
[x] laser_min_range
[x] max_beams (see #84)
[x] sigma_hit
[x] z_hit
[x] z_rand

Likelihood field model exclusive

[x] laser_likelihood_max_dist
[ ] beam_skip_distance
[ ] beam_skip_error_threshold
[ ] beam_skip_threshold
[ ] do_beamskip (see #187)

Beam model exclusive (see #54)

[x] lambda_short
[x] z_max
[x] z_short

Services

[x] reinitialize_global_localization
[ ] request_nomotion_update (see #184)

Publishers

[x] pose
[x] particle_cloud
[x] likelihood_field (extra)

Subscribers

[x] map
[x] initial_pose
[x] scan

With #196 documenting the current compatibility state between beluga_amcl and nav2_amcl, this ticket is no longer necessary.
#172, #184 and #187 describe the remaining work to be done to achieve full feature parity.


chore: move to node 20, make backtrace always compile release as we don't care about their internals

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Incorrect Daily challenges for roles

Found a new location? Use this form here

What day cycle it is?
Coordinates or a clear image of the location
Include the item name

I use the map almost every day and the General daily challenges are always correct. But the roles challenges are always different from the challenges in the game.
Is it a bug or am I doing something wrong?

It is not a bug. All role challenges displayed on the map are for role rank 15+
You need at least 15 role rank to get this dailies. Note that role rank is not your game level.
This site, can show you all dailies for all ranks.

Thank you


Multithreaded refinement

Medium-grained parallelism may be possible in the refinement stages. The idea would be to do the refinement in an interlaced pattern. Locking would only be needed near the end of a group of rows or columns.

Why don't you use OpenMP? It is supported by gcc as well. Moreover, should the decoder allocate all available processors in a machine or should it be configurable? Some people, specially when they are running the decoder on a notebook, might not like to allocate all their processors to a process.


Custom message for custom validator

Custom messages not working as expected with custom validators when containing hyphens.  Not sure if this is intended or a bug.
<input data-parsley-trigger="submit"
           data-parsley-length="[8, 16]"
           data-parsley-password-complexity="3"
           data-parsley-password-complexity-message="Custom error."
           type="password"
           value="">

window.ParsleyValidator.addValidator('passwordComplexity', function (value, requirement) {...}, 32);

Inside validator, how access data attributes @jalandis? thanks.

Any idea?

@brunowego please realize that it would be best to

read the Readme and the Contributing guidelines, and thus ask questions on StackOverflow (as requested)
not post in issues that have been closed for months
post a question that is understandable, ideally with a working example
This is valid for any open source project, btw...

I also ignore why you ask the question specifically to @jalandis who was simply kind enough to signal a problem and provide a solution.

Ok, thanks @marcandre


Add GridBuffer

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

41 of 61 (67.21%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-2.2%) to 77.855%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/src/grid/grid_view.dart
5
12
41.67%

lib/src/grid/grid_buffer.dart
36
49
73.47%

Totals

Change from base Build<PHONE_NUMBER>:
-2.2%

Covered Lines:
334

Relevant Lines:
429

💛  - Coveralls


[Bug]: Dark mode is  working on about us

Bug Description
Dark mode is  working on about us
Provide a detailed description of the bug
Dark mode is  working on about us
Expected Behavior
when clicking on dark mode toogle button , bg-color and text color should change
What browsers are you seeing the problem on?
No response
Put here any screenshots or videos (optional)
No response
By submitting this issue, you agree to

[X] I agree to follow this project's Code of Conduct
[X] I’ve read all the issues in the issue section, and this has not been reported yet.

Other Information

[X] I'm a GSSoC '24 Extended contributor
[X] I'm a Hacktoberfest '24 contributor
[X] I am willing to work on this issue (optional)

@maniyaom please assign this issue to me


PWN-3096 - make error handling from FeeRelayer

https://p2pvalidator.atlassian.net/browse/PWN-3096
[ Made error handling from FeeRelayer and pass this error to OrcaSwapPresenter to show error on UI ]

Screenshot 1
Screenshot 2

will move to QA after #381 as it is related to FeeRelayer and should be checked only after this PR will be merged


Feature/GUUI 3064 document the business rules about how emails are being sent

This PR has changes for email notification test fixtures which contains notifyRoles to notify specific roles for document state change, also updated test snapshot for email-notification to receive accurate notification.
For reference here is the document of email notification rules: Business rules about email notification from Meditor-notifier
To Test:
run test inside email-notifications. should pass all test

This is great—I love that we now have the business logic documented! All the tests passed fine for me, too. Two questions for you:

Do you think that the Google Doc should live in the notifier package so that it lives with the code to which it applies?
What does changing the fixtures and test do?

Thank you Ben...!

I think it is a good idea to keep a copy of google doc in notifier package.
by changing fixtures and tests, I have tested locally if we apply notifyRoles to fixture then only user with that specific roles should get email notification on document state change and not all users with any role. so once this PR is approved I can set up notify roles for workflow in meditor Prod website.


Adding a nice hover effect on submit button on contact.html page

I am working on it.

Work done!!, merged successfully.


Properly throw in case of numeric overflow in Calendar mathematics

We have applied multiple different strategies with regard to overflow during calendrical calculation, including capping input Date and clipping Julian day calculation with artificial bounds. Those are still insufficient as there are too many paths where numeric overflow may happen.
This patch patches those places by throwing whenever applicable. Also adds fuzzing tests to cover many more Calendar API.
Resolves rdar://133558250

@swift-ci please test

@swift-ci please test


add -a/--all option to annotatevariants intervals

Broke VSM.annotateIntervals into two functions (presence vs value).

One comment, back to you.

Done, back to you.


Wiki: client concurrency model improvement

On the following wiki page:
https://github.com/real-logic/aeron/wiki/Client-Concurrency-Model
I'm suggesting a minor improvement for the following text:
Publications with a different channel and stream id are completely independent from each other and operate in parallel.

My suggestion:
Publications with a different channel and stream id are completely independent from each other and operate concurrently.

Parallel should be used when they are happening at the same time, which isn't required. Concurrent means that they are not ordered and may or may not be processed at the same time.

The existing sentence is a little clumsy as the Publications don't really operate, it is the client application that operates them.  I'll reword it.


[WIP] Free memory after running normalize

This PR relates to issue #155
Running scIB.pp.normalize() starts an R kernel which requires memory that is not freed after running the function.
Several things to try:

[x] Deleting the objects moved to R
[x] Unloading modules in R
[ ] Closing down the R kernel after the computation is done

So removing the object and calling the garbage collector in R doesn't change anything:

@scottgigante could you check if it works any better with modules unloaded? My memory use is not making a lot of sense...

Without unloading modules:

With unloading modules:

Very confusing...

Note that memory use is system-wide so if there's anything else going on that will affect results. I'll try again with the new code when I get a chance to close everything else down.

Yeah, of course... i have a lot of tabs open... but haven't really been doing anything in between. Maybe some system job.

this seems to have fixed things. Tested and ready to merge.


Improve naming consistency

A while back there was a big push to cleanup naming consistency in ublue RPMs.
Seems like this could benefit from a similar effort.
I'd probably pick one name/phrase and stick to that.
It seems that ublue-update is the most commonly used so I'll use that as an example:
Changes I'd suggest:

rename update-ublue script to ublue-update
rename /etc/update.d to /etc/ublue-update.d (and also ublue-update.conf in that dir)
rename ublue-updater.spec to ublue-update.spec (and the related resulting RPM, the name, etc in spec file)
basically all the ublue-updater files and references drop the trailing r

This is not exhaustive but it hopefully communicates the idea.
It will feel cleaner and easier to discover relevant conf if the binary(script), systemd units, config files, paths, are all named consistently.

Fixed


Flaky test: Stack not rebuilding updated packages in a snapshot

Context
Stack builds the snapshot once and caches it in $STACK_ROOT/snapshots/SYSTEM/HASH/COMPILER_VER.
The hash seems to be made from the snapshot's name attribute only, not from its contents.
Therefore, if we update a snapshot YAML, but don't update its name, Stack may try to use outdated snapshot builds, leading to obscure compile errors.
This happens both locally and on Buildkite CI.
Test Case
Stack build.
Failure / Counter-example
byron-spec-ledger          > configure
byron-spec-ledger          > Configuring byron-spec-ledger-<IP_ADDRESS>...
byron-spec-ledger          > build
byron-spec-ledger          > Preprocessing library for byron-spec-ledger-<IP_ADDRESS>..
byron-spec-ledger          > Building library for byron-spec-ledger-<IP_ADDRESS>..
byron-spec-ledger          > [ 1 of 16] Compiling Byron.Spec.Ledger.Core
byron-spec-ledger          > Failed to load interface for  Cardano.Crypto.Hash.Keccak256
byron-spec-ledger          > There are files missing in the  cardano-crypto-class-2.0.0  package,
byron-spec-ledger          > try running 'ghc-pkg check'.
byron-spec-ledger          > Use -v (or `:set -v` in ghci) to see a list of the files searched for.

--  While building package byron-spec-ledger-<IP_ADDRESS> (scroll up to its section to see the error) using:
      /build/cardano-wallet.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_<IP_ADDRESS>_ghc-8.10.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-<IP_ADDRESS> build --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

Workarounds

Remove all stack caches.
Any time that something needs to be changed in the snapshot, change its name.

Therefore, if we update a snapshot YAML, but don't update its name, Stack may try to use outdated snapshot builds, leading to obscure compile errors.

This happens both locally and on Buildkite CI.

If @jonathanknowles ran into this locally from current master, it seems to suggest this happens irregardless of whether the snapshot name is updated along side changes, as the name for the new snapshot in #2801 was a new and unique one.

Should be fixed with #2831


Update endpoint

Description
I'm seeing this error when adding an additional model

It might be related to the time it takes to download the model.

@Zenthon @vukosim
Ishe has finished the Sentence Alignment on the gov-za multilingual so he'll jump ahead of me and work on Masakhane.
However, I do have an idea for masakhane on this bug so I just want to document it so Ishe might try it out.
We have the suspicion that the program runs out of memory when trying to add multiple models so perhaps we can try only dealing with one model at a time.
You can use python manage.py add_language to add language refs to the database and download them. Perhaps the /update endpoint should just be for downloading the models that have references in the database.
Note: when you python manage.py remove_language you should delete the directory storing the model for data coherence.
Once the models are downloaded, the client should list what models are available for translation.
When the user picks a model - say eng-swa only that model should be loaded into memory.
If the user picks another model - say eng-tiv, the old model should be overwritten and the new one placed into memory.
My only concern with this approach is that it takes a long time to load a model and that might affect UX.

@Kabongosalomon It seems that some of the language models config files are incorrect(their checkpoint files parameters are wrong or missing)  hence why you can't add some language models such as the en-ln- model
I have tested however that you can add multiple languages at once.
Example:

en-sw-JW300           (Swahili)
en-tiv-                       (Tiv)
en-iso-                      (Isoko)


Transfer dorks list to .db file and extend the variety of dorks

Dorks list already exists as .txt file, but it'll be more professional to keep dorks in simple database. Moreover, user should have some options about dorking with prepared lists:

Basic: search with basic dorks (~10-15 dorks)
Extended: search with extended dorks (~35-50 dorks)
Maximal: search with huge amount of dorks (~100 or even more dorks)
Custom: search with custom dorks database (user can fill the database with his own dorks)

Better idea will be to make DBs with categories separation. Some categories variants:

Basic
Web (configs/index of/logs/)
Files (docx, xlsx, and so on)
IoT
Strings/words finding
Leaks


Falcon request factory not working

With the latest openapi-core version (i.e. 0.14.2) Falcon request factory (i.e. FalconOpenAPIRequestFactory) not working at all. It's the same behavior with embedded middleware and custom one.
Code for reproducing:
import logging
import json
import falcon
from wsgiref import simple_server

from openapi_core import create_spec
from openapi_spec_validator.schemas import read_yaml_file
from openapi_core.validation.request.validators import RequestValidator
from openapi_core.contrib.falcon import FalconOpenAPIRequestFactory
# from openapi_core.contrib.falcon.middlewares import FalconOpenAPIMiddleware

logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

class Resource:
    def on_get(self, req, resp):
        logger.info('Calling GET...')
        resp.body = json.dumps({'foo': 'bar'})
        resp.status = falcon.HTTP_200

class OpenAPI3Middleware:
    def __init__(self, spec_dict):
        try:
            self.spec = create_spec(spec_dict)
            self.spec.security = []
        except Exception:
            logger.exception('Failed to create OpenAPI 3 spec from input dict.')
            raise

    def process_resource(self, req, resp, resource, params):
        logger.info('Processing request...')
        openapi_request = FalconOpenAPIRequestFactory().create(req)
        logger.info('Validating request...')
        validator = RequestValidator(self.spec)
        result = validator.validate(openapi_request)

        result.raise_for_errors()

if __name__ == '__main__':
    spec_dict = read_yaml_file('swagger.yml')
    openapi_middleware = OpenAPI3Middleware(spec_dict)
    # openapi_middleware = FalconOpenAPIMiddleware.from_spec(spec_dict)

    falcon_app = falcon.App(middleware=[openapi_middleware])

    test_resource = Resource()
    falcon_app.add_route('/v1/test', test_resource)

    httpd = simple_server.make_server('<IP_ADDRESS>', 8000, falcon_app)
    logger.info('Now serving on port 8000')
    httpd.serve_forever()

Swagger spec for above example:
openapi: 3.0.1

info:
  version: 1.0.0
  title: Test
  description: Test description.

servers:
  - url: http://localhost:8000
    description: Local server instance

tags:
  - name: Test
    description: Test description.

paths:
  /v1/test:
    get:
      tags:
        - Test
      summary: Test resource
      description: Test resource
      responses:
        200:
          description: successful operation
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/job_response'

components:
  securitySchemes:
    jwt_token:
      type: http
      scheme: bearer

  schemas:
    job_response:
      type: object

security:
  - jwt_token: []

GET to http://localhost:8000/v1/test gives error:
{
    "title": "415 Unsupported Media Type",
    "description": "text/plain is an unsupported media type."
}

Maybe I'm missing something here!? This is pretty critical issue for me, I'm not able to upgrade openapi-core to the latest version, which brings pretty much other bugfixes. :(
Falcon version is the latest one, i.e. 3.0.1.
Btw, from which openapi-core version is dropped off support for Falcon <3.0.0? Not sure what is reason for this?!

This behavior is reproducing from version 0.13.4, i.e. last working version is 0.13.3.

Does it work for master branch?
Falcon <3.0 was dropped on master.

@p1c2u Will check as soon as I get chance and respond here. Thanks!

@p1c2u @stojan-jovic
This is still happening on 0.14.2 with Falcon 3.0.1. Furthermore, if I try with:
curl --request GET \
  --url http://localhost:8000/v1/test \
  --header 'content-type: application/json'

I get the following:
2022-01-18 14:07:23 [FALCON] [ERROR] GET /things => Traceback (most recent call last):
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/falcon/app.py", line 323, in __call__
    process_request(req, resp)
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/contrib/falcon/middlewares.py", line 29, in process_request
    req_result = super(FalconOpenAPIMiddleware, self).process_request(
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/validation/processors.py", line 11, in process_request
    return self.request_validator.validate(request)
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/validation/request/validators.py", line 30, in validate
    path, operation, _, path_result, _ = self._find_path(request)
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/validation/validators.py", line 24, in _find_path
    return finder.find(request)
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/templating/paths/finders.py", line 25, in find
    if not paths_iter_peek:
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/more_itertools/more.py", line 316, in __bool__
    self.peek()
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/more_itertools/more.py", line 330, in peek
    self._cache.append(next(self._it))
  File "/Users/alex/dev/recipe-book/backend/api/venv/lib/python3.10/site-packages/openapi_core/templating/paths/finders.py", line 45, in _get_paths_iter
    paths = self.spec / 'paths'
TypeError: unsupported operand type(s) for /: 'dict' and 'str'

Interestingly, I have a large app I am working on where it's working fine (I was trying to repo a different issue), but there's too much code to provide a snippet as it is. I'll try to figure out what's different between them.


Add shortcut for focusing on the URL bar

Is your feature request related to a problem? Please describe.
Add another shortcut that does the equivalent of cmd-L but one handed.
Describe the solution you'd like
"l" to focus on the URL bar?

Just passing by, yet I think this request is irrelevent since vimari is a safari port of vimium, which does not have this shortcut. Moreover l is already used to scroll to the right. Imho it is better to say in sync with you are porting.
On a better note there is already a built in short cut for this which works cross browser ⌘l


feat(ssr): ssrPrefetch option

What kind of change does this PR introduce? (check at least one)

[ ] Bugfix
[x] Feature
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes
[x] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[x] It's submitted to the dev branch for v2.x (or to a previous version branch), not the master branch
[ ] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[x] All tests are passing: https://github.com/vuejs/vue/blob/dev/.github/CONTRIBUTING.md#development-setup
[x] New/updated tests are included

If adding a new feature, the PR's description includes:

[x] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

New lifecycle hook: ssrPrefetch
New context hook: context.rendered = context => {}

I would do terrible things for this to be merged 😓

What's the status of merging this PR? I'm in npm version hell because of it and and my project is screaming at me with errors. The bottleneck is getting caused by version mismatches of vue-apollo nuxt and nuxt/apollo-module.  SSR is very important. Hoping to see this resolved so I can deliver my product.

@dylanmcgowan As Evan said it's not a bug fix around Vue itself, but a feature (as declared in PR).
The thing is that this new feature will allow @Akryum to fix issues within vue-apollo (https://github.com/Akryum/vue-apollo/issues/402 https://github.com/Akryum/vue-apollo/issues/392)

@trainiac ssrPrefetch purpose is to allow this kind of usages which where impossible to achieve previously. It will mainly be used internally by both Nuxt and vue-apollo to simplify SSR data fetching and allow fetching in any component (not just route-level ones). You are more to welcomed to create a library that does what you describe on top of it. 😉

@Akryum Yes, I see that. It seems especially helpful for a GraphQL UI. Nice work! Hopefully my little blurb might help give others context to what this is trying to solve and some of the decision making that goes into where it's best to make API requests in an SSR app.
Might make sense to add the example of using beforeRouteUpdate in the SSR docs to fully cover all the hooks that need to be added in order make sure your data is always requested when you want.

Or rather, since this PR is focused on enabling data fetching capabilities for all components, using a watcher on $route would be better for the docs vs beforeRouteUpdate which is only available to route components.

Looks very good!!!! if have this, I can do more things like executing unsubscribe in SSR mode when using vue-rx, which can prevent for the memory leak~~


Recipe for each HAPI endpoint + refactor of DB schema

For this issue:
https://app.asana.com/0/1206255505490544/1207458813726923/f
We need a recipe for each API endpoint in the new HADX API. This task will also result ina refactored DB schema to separate recipes and memories into their own tables, currently they reside in the cmetadata JSON field on the langchain embedding table, which is a bit cumbersome and doesn't allow referential integrity enforcement.

Added cksum so process only checks in changed assets
Merged create recipe into recipes_sync, reduced number of steps needed
Migrated to use jinja templating for import and new recipe assets
Tidied up folder structure and intermediate files
Created a command line interface (CLI) for managing recipes, to make it simpler for users

Changed recipes separator to be if __name__ == '__main__': so that recipe.py can be imported easily

Added LLM prompt to create recipe starter boilerplate
Added run recipe to CLI which updates meta output

Migrated all LLM prompts to jinja, implemented intent standardization, generated using LLM CLI assistant (create and edit) set of recipes.


Command Wall do not save all files

For unknown reason command Wall/wall do not save anything. But W and w works.

The issue was in vim-eunuch that defines the same command.


Check model reinitialization when used in SPOTPY

There might be some elements that are not correctly reinitialized and are responsible for differences between a SPOTPY run and a single run.

SPOTPY_SingleRun_Comparison.xlsx

Fixed with #728b1d69a75fc457ccc17ad7c4aead99e15a4703


.Net: Fix bug to allow activation of JSON mode through execution_settings

Motivation and Context
Fixes #4719
Issue

When setting the execution_settings from YAML or config.txt, the response_format field is not recognized, and thus the JSON mode cannot be activated.

Root Cause

The issue arises because the ResponseFormat property, which is of type object?, stores values as JsonElement types when read from the aforementioned files.
The CreateChatCompletionOptions method does not account for this and performs type checking directly against types such as string, leading to improper conditional branching.
As a result, the ChatCompletionsOptions's ResponseFormat value remains unassigned, preventing the activation of JSON mode.

Description
This PR addresses the problem by adjusting the type checks to properly recognize the JsonElement objects.
This ensures that when ResponseFormat is set, the request sent to OpenAI is correct, and JSON mode can be utilized.
Please let me know if any further changes or discussions are needed regarding this pull request.

Contribution Checklist

[x] The code builds clean without any errors or warnings
[x] The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
[x] All unit tests pass, and I have added new tests where possible
[x] I didn't break anyone :smile:

@microsoft-github-policy-service agree company="FIXER Inc."

@markwallace-microsoft
Hi there, I've made the necessary code adjustments to pass CI. It would be great if you could take a look. Thanks!


Custom Toast Button rather than just text

I have gone through these following points

[x]   Check latest documentation: https://docs.nativebase.io/
[x]   Check for existing open/closed issues for a possible duplicate before creating a new issue: https://github.com/GeekyAnts/NativeBase/issues
[x]   Use the latest NativeBase release: https://github.com/GeekyAnts/NativeBase/releases
[x]   Check examples from NativeBase KitchenSink https://github.com/GeekyAnts/NativeBase-KitchenSink
[]   For discussion purpose make use of NativeBase Slack: http://slack.nativebase.io/
[x]   For queries related to the theme, check Theme Variables from Docs and live NativeBase Theme Editor http://nativebase.io/customizer/

Issue Description
I'm using Toasts, and there I want to give a custom Button. What I really want to give an Icon in the button,
node, npm, react-native, react and native-base version, expo version if used, Xcode version
node: v8.9.1
npm 6.4.1
react-native-cli: 2.0.1
react-native: 0.57.3
Expected behavior
Took the  in the place f buttonText prop of Toast component
Actual behavior
It is throwing error Can't find variable: React.
Steps to reproduce
import { Toast, Icon } from 'native-base';
 export const editProfile = (userId, data) => (
    dispatch => {
        dispatch({ type: 'EDITING_PROFILE' });
        requestEditProfile(userId, data).then(
            res => {
                         dispatch(userProfileEdited(res.data)),
                         Toast.show({position: 'bottom',
                              duration: 7000,
                              textStyle: { fontSize: 12, fontFamily: 'Raleway-SemiBold', color: 'white'},
                              type: 'success'
                              buttonText: <Icon name='ios-checkmark-circle-outline' size={16} />,
                         })
             }
            err => dispatch(userProfileEditFailed(err))
        )
    }
)

#2194


Creating CS1751

Summary
Describe your changes here.
Fixes #Issue_Number (if available)

Hi @BillWagner ,
How can I fix this OPS status checker? In VS, I can't see any errors/warnings. I can't see anything relevant on this page too.
Thanks,
Balazs


Fixed some typos, added information.

Fixed some typos, added information that was lost in the merge, and added answer from #1083.
Also cleaned up links, and added some new ones.

Nice!


Add missing commas to jdk16 config file

An errant c+p removed commas between multiple elements in a list,
causing pipeline failure. This change restores those commas.
Signed-off-by: Adam Farley<EMAIL_ADDRESS>

Fixes: #2254

Thanks for the reviews. :)
Please merge when ready.

Pipeline running: https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk16-pipeline/196/


[Bug]: ValueError: Model architectures ['Qwen2ForCausalLM'] are not supported for now

Your current environment
PyTorch version: 2.1.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: glibc-2.31

Python version: 3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-99-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100S-PCIE-32GB
Nvidia driver version: 525.105.17
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          80
On-line CPU(s) list:             0-79
Thread(s) per core:              2
Core(s) per socket:              20
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel Xeon Processor (Skylake, IBRS)
Stepping:                        4
CPU MHz:                         3000.032
BogoMIPS:                        6000.06
L1d cache:                       1.3 MiB
L1i cache:                       1.3 MiB
L2 cache:                        160 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0-39
NUMA node1 CPU(s):               40-79
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ibrs ibpb fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.1.1+cu118
[pip3] triton==2.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.1.1+cu118              pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypiROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.2.4
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity
GPU0     X      0-79    0-1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
when run " python -m vllm.entrypoints.openai.api_server --model /root/autodl-tmp/models/Qwen1.5-14B-Chat-AWQ --trust-remote-code"
i get :
MegaBlocks not found. Please install it by pip install megablocks.
STK not found: please see https://github.com/stanford-futuredata/stk
Traceback (most recent call last):
File "/root/miniconda3/envs/python310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/envs/python310/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 729, in
engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 495, in from_engine_args
engine = cls(parallel_config.worker_use_ray,
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 269, in init
self.engine = self._init_engine(*args, **kwargs)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 314, in _init_engine
return engine_class(*args, **kwargs)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 109, in init
self._init_workers(distributed_init_method)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 145, in _init_workers
self._run_workers(
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 750, in _run_workers
self._run_workers_in_batch(workers, method, *args, **kwargs))
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 724, in _run_workers_in_batch
output = executor(*args, **kwargs)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/worker/worker.py", line 72, in load_model
self.model_runner.load_model()
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 36, in load_model
self.model = get_model(self.model_config)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/model_executor/model_loader.py", line 88, in get_model
model_class = _get_model_architecture(model_config.hf_config)
File "/root/miniconda3/envs/python310/lib/python3.10/site-packages/vllm/model_executor/model_loader.py", line 82, in _get_model_architecture
raise ValueError(
ValueError: Model architectures ['Qwen2ForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'FalconForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'OPTForCausalLM', 'PhiForCausalLM', 'QWenLMHeadModel', 'RWForCausalLM', 'YiForCausalLM']

vLLM Version: 0.2.4

Your vLLM version is too old. Qwen2 is support in v0.3.0 and higher

Is your problem solved？

Is your problem solved？

Name: vllm
Version: 0.6.3.post1
has same error
model_path = snapshot_download("Qwen/Qwen2.5-7B-Instruct")
@simon-mo