Using ESMF unstructured grids

I'm wondering what it would take to support using ESMF's unstructured grids with xESMF. I had a conversation with @bekozi today where we discussed this briefly but we weren't sure where xESMF stood on the issue.
Ultimately, it would be great if xESM could support bi-directional remapping between structured/unstructured grids/meshes.

My major concern is how can xarray represent unstructured grids (as discussed in docs)?
I have little experience with unstructured grids like in MPAS and don't know how MPAS folks analyze their data. I believe that it is possible to design a data model that works fairly well with one specific type of unstructured grid (say the one used by MPAS), but an intuitive, easy-to-use frontend for arbitrary unstructured grid sounds too fancy for me.
For my personal research I use GFDL-FV3 which fortunately uses a (multi-tile) quadrilateral grid and can be handled gracefully by current xESMF. So I don't have personal needs to support unstructured grids. That said, if there's any good idea about unstructured meshes I am willing to try. It will be pretty cool.

My major concern is how can xarray represent unstructured grids (as discussed in docs)?

My thought here is that we can follow existing conventions for this. There are existing specs/conventions for how these grids can be stored in the common data model (e.g. UGRID).
While these formats are not as intuitive as a logical grid may be, xarray should not have a problem storing a UGRID dataset. ESMF includes UGRID as a source/target grid spec so this should not be a problem.

if there's any good idea about unstructured meshes I am willing to try. It will be pretty cool.

To be clear, I'm not asking you to do any work here but just evaluating what it would take to make this work. I do have a use case that I would like to see an ESMF unstructured grid work for.
Perhaps we can get @bekozi to comment on what the workflow might look like here.

xarray should not have a problem storing a UGRID dataset.

xarray can indeed store all UGRID information in a Dataset but I wonder if that gives any benefit than simply using dictionaries or something simpler to store the same data. A whole extension is needed to utilize the connectivity information, such as allowing dr.sel(), dr.plot() on unstructured mesh.
If xarray doesn't give additional advantage for analyzing unstructured mesh, it doesn't seem to worth the effort to use it as the frontend for ESMPy regridding. Directly writing ESMPy code might be easier.
A similar issue is the current treatment of cell boundaries (pydata/xarray#1475, referred so many times...). N+1 sized boundary variables can indeed live in a Dataset but they cannot be utilized by other xarray functionalities such as plotting. And this is a much simpler problem than unstructured mesh...

If xarray doesn't give additional advantage for analyzing unstructured mesh, it doesn't seem to worth the effort to use it as the frontend for ESMPy regridding.

I'm guessing my use case is sufficiently different from your typical use case that what doesn't seem useful to you is indeed useful to me.
As background, my use case is mapping gridded datasets to hydrologic response units (HRUs, e.g. river basin polygons). This can be done with ESMF using unstructured mesh regridding. Currently, I'm looking to build an xarray oriented tool kit for working with HRU meshes.
So my basic question for this issue is whether or not remapping between xarray datasets that use unstructured grids is within the scope of this package?

whether or not remapping between xarray datasets that use unstructured grids is within the scope of this package?

This is out of the current scope of my package. Right now I don't have a clear plan to deal with unstructured meshes, and wonder if directly interfacing xarray with ESMPy for unstructured mesh is the right way to go. The gridded API looks like a more suitable frontend for unstructured mesh, but it is not mature yet. There are some discussions on interfacing gridded with xarray (NOAA-ORR-ERD/gridded#20). Maybe xarray <-> gridded <-> ESMPy is a better way to use ESMPy's unstructured mesh regridding from xarray?

Perhaps we can get @bekozi to comment on what the workflow might look like here.

I'm awful (terrible really) with catching these tagged issues. I'll do better. So, sorry again for the embarrassingly slow response. :bash
Creating a generic unstructured grid interface can be a headache since there is no defacto standard in the climate/netCDF world. There are tools in ocgis that could help with interpreting unstructured formats and getting them into xarray if you are not interested in coding from scratch. Depending on the workflow, you can convert from shapely geometries or go directly from coordinates in a netCDF. The latter is still being ironed out. In terms of unstructured grid metadata conventions, SCRIP Unstructured, ESMF Unstructured, and UGRID are supported in ocgis.
I'd be happy to help you get set up with these interfaces as they can be a bit tricky to use. If the workflow is xarray-->ocgis-->xarray, we'd need to add a to_ocgis somewhere. Given my experience writing one going the other way, I don't expect that to be too difficult...and it's good to have a use case!
Unfortunately I don't have any experience with gridded, so I cannot comment on how interoperable it is with ESMPy. My sense is that it is probably sufficient? :confused: The way ESMF represents meshes internally is similar to UGRID, so if you decided to go with supporting a single convention and wanted to roll your own code, the UGRID to ESMPy code is not a deal breaker.
Anyway, let me know if you have any questions!

@bekozi - thanks for revising this issue.

This is out of the current scope of my package.

I'm curious what it would take to scope this issue such that it fits in xESMF. If we just stick to the UGRID spec and ESMpy, is there a development path for adding unstructured mesh regridding? Does ocgis need to get involved or can ESMF handle the regridding procedure by itself? Does ESMF need to get involved (maybe we just use ocgis but maintain the existing regridding API)?

I'm curious what it would take to scope this issue such that it fits in xESMF. If we just stick to the UGRID spec and ESMpy, is there a development path for adding unstructured mesh regridding?

There are multiple things need to be done:

The "front-end design": How to encode UGRID spec in xarray Dataset? As @bekozi said, there is no no de-facto standard for general unstructured grids. The specification we choose should be at least compatible with popular unstructured-grid models (e.g. MPAS), to minimize conversion effort.
The "bridge": Convert xarray Dataset to ESMPy Mesh object. OCGIS might help (see the next reply)?
The computation: Retrieve the regridding weights as explicit numpy arrays, and apply weights using the Dask/Scipy(Numba/Cython) stack, just like xESMF does. This is necessary for scaling-up the regridding computation to the Pangeo platform. ESMPy's built-in, MPI-based parallelization doesn't integrate well with dask/Pangeo.

For (1) and (2), I have little idea since I seldom work with unstructured meshes. I am only looking into (3), which is a general problem for both structured and unstructured grids.

Does ocgis need to get involved or can ESMF handle the regridding procedure by itself?

According to @bekozi 's reply in this comment, ocgis is used to:

Read a shapefile representing an unstructured grid
Interface with ESMPy for regridding

Seems that it can save some coding compared to pure ESMPy. Also, ocgis has an to_xarray() method. How would it handle unstructured meshes?

Does ESMF need to get involved

Do you mean the Fortran library? No, ESMPy supports unstructured mesh, so everything can be written in Python.

The "front-end design": How to encode UGRID spec in xarray Dataset? As @bekozi said, there is no no de-facto standard for general unstructured grids. The specification we choose should be at least compatible with popular unstructured-grid models (e.g. MPAS), to minimize conversion effort.

Looking through the MPAS Mesh Specification Document, I don't see any mention of a larger convention. I suspect a converter exists somewhere out there to UGRID - maybe you even know of one? In sum, we may need to ask some specialists to clarify this.

The "bridge": Convert xarray Dataset to ESMPy Mesh object. OCGIS might help (see the next reply)?

Yes, OCGIS can help here. I can also reasonably say that supporting the MPAS convention directly in OCGIS would not be out of the question.

ESMPy's built-in, MPI-based parallelization doesn't integrate well with dask/Pangeo.

Tangent incoming...
I have some technical inquiries regarding this, and I'm not sure where it would be best to discuss them. @jhamman, maybe you could direct me.
There are these OcgVM and OcgDist objects that handle MPI communication, decompositions, etc. Colleagues at @bird-house https://github.com/bird-house/bird-house.github.io/issues/17 and myself are interested in dask for memory scaling, but would like to leverage MPI as well for asynchronous IO, some algorithms like spatial subsetting, and interfacing with things like ESMPy. I think dask and MPI would be somewhat interoperable provided the decomposed/chunked dimensions don't change size during processing. I really hope I can reuse these vm objects to enable this (In part, I'm also looking around for shortcuts...err...elegance).
Furthermore, I think it may be possible to take another step and enable spatial decompositions in dask allowing a greater chunking variety. For example, the memory requirements for high spatial resolution regridding could be amenable to dask, and potentially xESMF, as opposed to the intermediate file approach we are using for chunked weight generation. In sum, we need a spatial halo for some operations and this is one potential approach for getting it. Maybe someone has already done this?
We're probably going to move forward with this pretty soon, and it would be great to get advice / buy in from dask experts as opposed to charging blindly into the night!

Read a shapefile representing an unstructured grid

Just to clarify, it can work with a shapefile, shapely, or directly from coordinates (like stored in a netcdf).

Interface with ESMPy for regridding

Depending on the workflow you can get a grid, mesh, field, or the whole kit and kaboodle (never actually typed that before).

Seems that it can save some coding compared to pure ESMPy. Also, ocgis has an to_xarray() method. How would it handle unstructured meshes?

I'm not sure how much to_xarray would come into this unless you offloaded all regridding to OCGIS/ESMPy. You would probably just want the ESMPy field or mesh object. We'd need a to_ocgis function that I could definitely help with. So, in the ideal case:

Read data with xESMF.
Convert to ocgis with metadata and tell it the convention.
Have ocgis convert to appropriate ESMPy object.
Continue with xESMF.

and it would be great to get advice / buy in from dask experts as opposed to charging blindly into the night!

Probably best to open an issue on the pangeo issue tracker and ping mrocklin.

Probably best to open an issue on the pangeo issue tracker and ping mrocklin.

Thanks. I'll try and make something more cohesive. I know the mpi-dask thing is kind of fleshed out on the dask issue tracker.


Unify FAB attack

framework-independent components in FAB attack are collected in class FABATTACK.

Pytorch specific functions and TF2 specific functions are implemented in class FABAttack_PT and FABAttack_TF respectively.

The major benefit of this PR is source code is unified. We only need to maintain a copy.

Additionally, I would suggest that autoattack.fab_projections should be class FABATTACK's member functions.

This makes a lot of sense, thanks! I'll do something similar for APGD, since anyway I wanted to update its implementation: this indeed would simplify maintaining the code. I'll have a more detailed look and merge.
About moving the projections to FABATTACK class, is there a particular benefit in that, or is it about clarity of the code?

About moving the projections to FABATTACK class, is there a particular benefit in that, or is it about clarity of the code?

I guess that the original authors did not want to maintain two copies. So they created a new file fab_projections.py which is imported by fab_tf and fab_pt. However, I have solved this issue in this PR. In my opinion, projections are the private functions which are only used by fab attack. Therefore, those functions should be member functions.

I see. Then it's fine for me to move them.

I am planning to simplify non-targeted and targeted version attacks.
I fond that function attack_single_run_targeted is slightly different from others implementation(function attack_single_run in PT and TF and function attack_single_run_targeted in PT).
--- auto-attack/autoattack/fab_pt.py: function attack_single_run
+++ auto-attack/autoattack/fab_tf.py: function attack_single_run_targeted
            counter_iter = 0
            while counter_iter < self.n_iter:
                with torch.no_grad():
                    df, dg = self.get_diff_logits_grads_batch_targeted(x1, la2, la_target2)
+                    if len(df.shape) == 1:
+                        df.unsqueeze_(1)
+                        dg.unsqueeze_(1)

Which implementation should I use ?

I'd have to check how it behaves after the new modifications, but originally the unsqueeze was used because for the untargeted version the dimension of df was n_classes x batch_size, but for the targeted one it got squeezed since only one target class is used (and similarly for dg). Then it depends on the shape the df and dg, which is expected to have number of classes or 1 for the targeted version as first entry.

OK, I think that I can do reshape in tf side first. So the output tensor's shape in TF and PT are consistent.

The new implementation looks indeed much cleaner, thanks!


add domain for puthisastra university

add puthisastra univerisity domain to jetbrain
here is university website: https://www.puthisastra.edu.kh/

@RamekhCHHOENG I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host up.edu.kh not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!


Adding Several Prototypes to Mink Interface

The following prototypes are added to the Mink psuedo-interface.

fillFields()
selectOption()

These are to resolve Scrutinizer issues showing up in projects which use Flex Mink.

Paging @Chekote to review the back-end work in this pull request!


Do not populate bookmarks cache with favicons for all visited sites

Task/Issue URL: https://app.asana.com/0/414709148257752/1202223699061973/f
CC: @miasma13
Description:
When we import bookmarks from HTML file we want to download favicons for them not before by messing with caches.
Steps to test this PR:

Export bookmarks to HTML file.
Remove all bookmarks.
Import bookmark from HTML file.
Make sure that imported websites get the favicon.

Hi 👋
The fix you made is exactly what the Privacy team has ruled out here.
Also, I've made a quick video and it doesn't resolve this issue - the favicon is still not visible in the bookmarks screen at the end of video.
The test scenario I was using is importing a large number bookmarks into the app that were not previously bookmarked, as importing bookmarks is primarily for users who have changed device, re-installed the app or are new users and are bringing in bookmarks for the first time. So it's great that you have  another use case that you are testing however you also need to test the use case I mentioned, thanks 🙂
https://user-images.githubusercontent.com/91189922/168596245-a1ffbea8-285e-4d4b-9c51-7c4b462afc98.mov


76664 create sap mock api endpoint

PBI#76664 - Created SAP mock service
PBI#75446 - Consumed the mock service in ERP Facade.
PBI#76752 - Deploy SAP mock service

This files are related to CI/CD. @amogh14893 @kiran1501497 please look into this.


list.pod6 example is broken

https://docs.perl6.org/language/list.html#Sequences - the second example. The comment says about an error, but this code actually works and it loops forever. Seems like a pre-GLR example, so it needs to be investigated and resolved: do we need to remove this example or do we need to rewrite it?
Don't forget to remove :skip-test(if it will be there) if the result will be finite.

This example was added by @skids in 4a47be526cf4631fee8a8bb862d84bee38ee6821 - just after Christmas.
I'm just going to remove that paragraph and example.

m: my @s := Seq.new();
OUTPUT«Type check failed in binding to $iter; expected Iterator but got List ($("a", "b", "c"))␤  in block  at  line 1␤␤»

fixed with 1619a0683c


BadParcelableException fix

In response to issue: https://github.com/BranchMetrics/android-branch-deep-linking/issues/464
Added a try/catch when parsing the Chrome intent's parcel.

LGTM 👍

Hey @SAPNA-BHOR -- we released SDK version 2.12.0 which includes a fix for your reported BadParcelableException issue. Let us know how it goes.


"ERROR: Signature extraction failed"

[x] Using latest version as provided on the master branch
[x] Searched for similar issues including closed ones

What is the purpose of your issue?

[x] Bug
[ ] Feature Request
[ ] Question
[ ] Other

Description

Downloading my playlist from another system worked (different version from github). When downloading from my fresh windows 10, it downloaded only ~80/180. Every failed download gives the same error message.
Updating youtube-dl via pip install -U youtube_dl didn't help (latest was already installed prior).
Here are some tracks, which failed to download:

https://open.spotify.com/track/3hn2s09PyNrYSqCnhYjm7L
https://open.spotify.com/track/0cPsHxvuICaPKmWRFCRe1i
https://open.spotify.com/track/5ERWSokrBWx8sUwHV6QN93
https://open.spotify.com/track/1mPPo1eG1joz3HX0Ecw9LL
https://open.spotify.com/track/4DErCvO9irCDFmNZWnNw3E
https://open.spotify.com/track/66WcaWVGRFmpx0euxmw9yM
https://open.spotify.com/track/6bV83bOk40Vdtdgicmqv3S
https://open.spotify.com/track/0ndt8ke0ApnXkNxEhamOnW
https://open.spotify.com/track/2WaA5bVXcIqqpz9j0gd7qK
https://open.spotify.com/track/0UFtsK5aSrq5zYai4aikAu

Log

F:\Development\spotify-downloader>python spotdl.py --list "(8.txt" --log-level=DEBUG
DEBUG: Python version: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Intel)]
DEBUG: Platform: Windows-10-10.0.17134-SP0
DEBUG: {'album': None,
'avconv': False,
'config': None,
'download_only_metadata': False,
'dry_run': False,
'file_format': '{artist} - {track_name}',
'folder': 'C:\Users\removedName\Music',
'input_ext': '.m4a',
'list': '(8.txt',
'log_level': 10,
'manual': False,
'music_videos_only': False,
'no_metadata': False,
'no_spaces': False,
'output_ext': '.mp3',
'overwrite': 'prompt',
'playlist': None,
'search_format': '{artist} - {track_name} lyrics',
'song': None,
'trim_silence': False,
'username': None,
'version': False,
'youtube_api_key': None}
INFO: Preparing to download 98 songs
DEBUG: Fetching metadata for given track URL
DEBUG: Fetching lyrics
DEBUG: {'album': {'album_type': 'album',
'artists': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/043Bey7ngSGRFpJftAlKpi'},
'href': 'https://api.spotify.com/v1/artists/043Bey7ngSGRFpJftAlKpi',
'id': '043Bey7ngSGRFpJftAlKpi',
'name': 'Spag Heddy',
'type': 'artist',
'uri': 'spotify:artist:043Bey7ngSGRFpJftAlKpi'}],
'external_urls': {'spotify': 'https://open.spotify.com/album/1FmvCvnTpGAnhEPTVrUUiz'},
'href': 'https://api.spotify.com/v1/albums/1FmvCvnTpGAnhEPTVrUUiz',
'id': '1FmvCvnTpGAnhEPTVrUUiz',
'images': [{'height': 640,
'url': 'https://i.scdn.co/image/b5992ca4e972b13fdd4b846b171e4399a8007a8e',
'width': 640},
{'height': 300,
'url': 'https://i.scdn.co/image/9a5994ff2d8a1e24e32d88a37163826be72673e5',
'width': 300},
{'height': 64,
'url': 'https://i.scdn.co/image/9c7e05c88fa8adbcca28eebd30e731f7f73a5ea2',
'width': 64}],
'name': 'The Classics',
'release_date': '2017-05-05',
'release_date_precision': 'day',
'total_tracks': 20,
'type': 'album',
'uri': 'spotify:album:1FmvCvnTpGAnhEPTVrUUiz'},
'artists': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/043Bey7ngSGRFpJftAlKpi'},
'href': 'https://api.spotify.com/v1/artists/043Bey7ngSGRFpJftAlKpi',
'id': '043Bey7ngSGRFpJftAlKpi',
'name': 'Spag Heddy',
'type': 'artist',
'uri': 'spotify:artist:043Bey7ngSGRFpJftAlKpi'}],
'copyright': '2017 Create Classics',
'disc_number': 1,
'duration': 233.8,
'explicit': False,
'external_ids': {'isrc': 'CA5KR1554926'},
'external_urls': {'spotify': 'https://open.spotify.com/track/0fUF5bxAbaEkZeRatGvnG0'},
'genre': 'Brostep',
'href': 'https://api.spotify.com/v1/tracks/0fUF5bxAbaEkZeRatGvnG0',
'id': '0fUF5bxAbaEkZeRatGvnG0',
'is_local': False,
'lyrics': None,
'name': 'Permanent',
'popularity': 47,
'preview_url': 'https://p.scdn.co/mp3-preview/cbfd07113bfdeb71d0484fa80cb176acef754164?cid=4fe3fecfe5334023a1472516cc99d805',
'publisher': 'Create Classics',
'release_date': '2017-05-05',
'total_tracks': 20,
'track_number': 1,
'type': 'track',
'uri': 'spotify:track:0fUF5bxAbaEkZeRatGvnG0',
'year': '2017'}
DEBUG: Opening URL: https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=Spag Heddy - Permanent lyrics
ERROR: Signature extraction failed: Traceback (most recent call last):
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1210, in _decrypt_signature
video_id, player_url, s
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1121, in _extract_signature_function
res = self._parse_sig_js(code)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1182, in _parse_sig_js
jscode, 'Initial JS player signature function name', group='sig')
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\common.py", line 972, in _search_regex
raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by RegexNotFoundError('Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: Failed to download song. Will retry after other songs
DEBUG: Fetching metadata for given track URL
DEBUG: Fetching lyrics
DEBUG: {'album': {'album_type': 'single',
'artists': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/0daugAjUgbJSqdlyYNwIbT'},
'href': 'https://api.spotify.com/v1/artists/0daugAjUgbJSqdlyYNwIbT',
'id': '0daugAjUgbJSqdlyYNwIbT',
'name': 'Vicetone',
'type': 'artist',
'uri': 'spotify:artist:0daugAjUgbJSqdlyYNwIbT'}],
'external_urls': {'spotify': 'https://open.spotify.com/album/6Xv7zgkIswLSsnqmYeo8OL'},
'href': 'https://api.spotify.com/v1/albums/6Xv7zgkIswLSsnqmYeo8OL',
'id': '6Xv7zgkIswLSsnqmYeo8OL',
'images': [{'height': 640,
'url': 'https://i.scdn.co/image/06439f999bd4fe50eb8a8b79585f518dc16ce041',
'width': 640},
{'height': 300,
'url': 'https://i.scdn.co/image/d4af804fa5e0668de91bd0d31b22809b39f2814a',
'width': 300},
{'height': 64,
'url': 'https://i.scdn.co/image/1a6d716342c553fe8ccecfe9519a79d1695ddc57',
'width': 64}],
'name': 'Nevada (feat.  Cozi Zuehlsdorff)',
'popularity': 65,
'preview_url': 'https://p.scdn.co/mp3-preview/3fdbd9870e39d5da03933f7e084aca251a8bcbbb?cid=4fe3fecfe5334023a1472516cc99d805',
'publisher': 'Monstercat',
'release_date': '2016-06-24',
'total_tracks': 1,
'track_number': 1,
'type': 'track',
'uri': 'spotify:track:5BIjJP1zlOa3iY65RSkXkr',
'year': '2016'}
DEBUG: Opening URL: https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=Vicetone - Nevada (feat. Cozi Zuehlsdorff) lyrics
ERROR: Signature extraction failed: Traceback (most recent call last):
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1210, in _decrypt_signature
video_id, player_url, s
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1121, in _extract_signature_function
res = self._parse_sig_js(code)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\youtube.py", line 1182, in _parse_sig_js
jscode, 'Initial JS player signature function name', group='sig')
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\youtube_dl\extractor\common.py", line 972, in _search_regex
raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by RegexNotFoundError('Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: Failed to download song. Will retry after other songs
DEBUG: Fetching metadata for given track URL
DEBUG: Fetching lyrics
ERROR:
Traceback (most recent call last):
File "spotdl.py", line 198, in main
download_list(text_file=const.args.list)
File "spotdl.py", line 83, in download_list
download_single(raw_song, number=number)
File "spotdl.py", line 119, in download_single
meta_tags = spotify_tools.generate_metadata(raw_song)
File "F:\Development\spotify-downloader\core\spotify_tools.py", line 66, in generate_metadata
meta_tags['name'])
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lyricwikia\lyricwikia.py", line 30, in get_lyrics
return get_all_lyrics(artist, song, linesep, timeout)[0]
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lyricwikia\lyricwikia.py", line 36, in get_all_lyrics
response = _requests.get(url, timeout=timeout)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 644, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 644, in
history = [resp for resp in gen] if allow_redirects else []
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 222, in resolve_redirects
**adapter_kwargs
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 662, in send
r.content
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\models.py", line 827, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\models.py", line 749, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\response.py", line 465, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\response.py", line 413, in read
data = self._fp.read(amt)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 447, in read
n = self.readinto(b)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 491, in readinto
n = self.fp.readinto(b)
File "C:\Users\removedName\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 589, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt

Seems like YouTube changed something (yet again). Let's hope if those awesome youtube-dl guys can release a fix soon.

Here is the error that happens for me when downloading a single song.

$ python3 spotdl.py -s "ncs fade"
ERROR: Signature extraction failed: Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1210, in _decrypt_signature
    video_id, player_url, s
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1121, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1182, in _parse_sig_js
    jscode, 'Initial JS player signature function name', group='sig')
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player signature function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1210, in _decrypt_signature
    video_id, player_url, s
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1121, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1182, in _parse_sig_js
    jscode, 'Initial JS player signature function name', group='sig')
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 502, in extract
    ie_result = self._real_extract(url)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1803, in _real_extract
    encrypted_sig, video_id, player_url, age_gate)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1220, in _decrypt_signature
    'Signature extraction failed: ' + tb, cause=e)
youtube_dl.utils.ExtractorError: Signature extraction failed: Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1210, in _decrypt_signature
    video_id, player_url, s
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1121, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1182, in _parse_sig_js
    jscode, 'Initial JS player signature function name', group='sig')
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player signature function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ritiek/Downloads/pafy/pafy/backend_youtube_dl.py", line 40, in _fetch_basic
    self._ydl_info = ydl.extract_info(self.videoid, download=False)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 815, in extract_info
    self.report_error(compat_str(e), e.format_traceback())
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 620, in report_error
    self.trouble(error_message, tb)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 590, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: Signature extraction failed: Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1210, in _decrypt_signature
    video_id, player_url, s
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1121, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1182, in _parse_sig_js
    jscode, 'Initial JS player signature function name', group='sig')
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player signature function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "spotdl.py", line 216, in <module>
    main()
  File "spotdl.py", line 196, in main
    download_single(raw_song=const.args.song)
  File "spotdl.py", line 120, in download_single
    content = youtube_tools.go_pafy(raw_song, meta_tags)
  File "/home/ritiek/Downloads/spotify-downloader/core/youtube_tools.py", line 35, in go_pafy
    track_info = pafy.new(track_url)
  File "/home/ritiek/Downloads/pafy/pafy/pafy.py", line 124, in new
    return Pafy(url, basic, gdata, size, callback, ydl_opts=ydl_opts)
  File "/home/ritiek/Downloads/pafy/pafy/backend_youtube_dl.py", line 31, in __init__
    super(YtdlPafy, self).__init__(*args, **kwargs)
  File "/home/ritiek/Downloads/pafy/pafy/backend_shared.py", line 97, in __init__
    self._fetch_basic()
  File "/home/ritiek/Downloads/pafy/pafy/backend_youtube_dl.py", line 43, in _fetch_basic
    raise IOError(str(e).replace('YouTube said', 'Youtube says'))
OSError: ERROR: Signature extraction failed: Traceback (most recent call last):
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1210, in _decrypt_signature
    video_id, player_url, s
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1121, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/youtube.py", line 1182, in _parse_sig_js
    jscode, 'Initial JS player signature function name', group='sig')
  File "/home/ritiek/.local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player signature function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player signature function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

youtube-dl <IP_ADDRESS> just came out and fix the issue :)

Yep, it did! You can update your youtube-dl to latest version with
$ pip3 install youtube-dl -U

Gotta close this.


As an LA I want to see more details about a scheme in the scheme list

Some LAs have expressed a desire to see more details about a scheme in the scheme list. For example: current milestone, planned construction completed milestone date.
We currently use a table to display the schemes, which at 4 columns is at capacity. To show further details we may want to consider using a card list instead. This would allow us to surface more details without being constrained by a column layout.

Authority feedback:

This seems like a longer process than the excel sheet as we have to click into lots more boxes and cannot compare schemes on the same sheet to check information.
One future improvement we have looked at was surfacing more information on the initial schemes list page, so rather than just scheme reference, scheme name, funding programme we might show more data like spend to date, funding allocation, current milestone. Would that allow you to do the comparisons you mentioned? Yes that would be better – just to ensure the correct information is in the correct scheme at a glance rather than having to go into multiple pages. Outputs would also be useful if there was space.


Second iteration of DDP solver

In GitLab by @cmastalli on Dec 14, 2018, 17:06
This PR includes an vast change of the code structure for the increase of efficiency. Additionally, there are few bug fixes most of them regarding to the solver.

In GitLab by @cmastalli on Dec 14, 2018, 17:06
merged

In GitLab by @cmastalli on Dec 14, 2018, 17:06
mentioned in commit 58174d6e99949e04384dd5aeadd122ccbd85c323


Create ringAlarmWithSensors.groovy

A Ring Alarm device with sensors as child devices to be used as "Things".

Great! I will check this on weekend.


config's default output format doesn't render very nicely

Currently, the output of a config command is something like this:
>>> subprocess.getoutput(['ruff', 'config', 'lint.preview'])
'Whether to enable preview mode. When preview mode is enabled, Ruff will\nuse unstable rules and fixes.\n\nDefault value: false\nType: bool\nExample usage:\n```toml\n# Enable preview features.\npreview = true\n```'

In the terminal, this looks nice:
Whether to enable preview mode. When preview mode is enabled, Ruff will
use unstable rules and fixes.

Default value: false
Type: bool
Example usage:
```toml
# Enable preview features.
preview = true
```

However, when rendered, the "Default value", "Type" and "Example usage" lines are collapsed into a single
 (Commonmark dingus):

Whether to enable preview mode. When preview mode is enabled, Ruff will
use unstable rules and fixes.

Default value: false
Type: bool
Example usage:

# Enable preview features.
preview = true

A possible fix is to append two spaces to the end of the first two lines in question (Commonmark dingus):
'...Default value: false  \nType: bool  \nExample usage:...'

Backslashes also work as markdown newline indicators (and are more obvious than double spaces) demo

Thanks for opening this issue. I agree, the output in PyCharm isn't ideal.
I assume that the hover text in PyCharm comes from the JSON schema. If that's the case, I would prefer a local change in the schema generation to use \ over the somewhat obscure use of two spaces at the end of the line.

@MichaReiser -- Is there still something to do here given the PR was closed?

ruff config only supports the output format text and json. The ask in this issue is to add a markdown output format.

This looks great!


Vue2.insertAdjacentElement is not a function

Describe the bug
I imported a Vue 2 component library in vitest. The test should pass but it throws the following TypeError:
Vue2.insertAdjacentElement is not a function

I don't know whether it is a bug of vite, vitest or jsdom. Since there is no such issue when Vite <= 3.1.4, I post this bug here.
Reproduction
https://stackblitz.com/edit/vitest-dev-vitest-lhw5je
Steps to reproduce
Open the link and wait for the test to fail
System Info
System:
    OS: Linux 5.0 undefined
    CPU: (6) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 16.14.2 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 7.17.0 - /usr/local/bin/npm
  npmPackages:
    vite: ^3.1.6 => 3.1.6

Used Package Manager
pnpm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

Changing the vue-echarts alias from vue-echarts/dist/index.esm.min.js to vue-echarts/dist/index.esm.js worked.
Also reverting #10300 (effb7c3fd5ecc2cc93928928adfc6f69bb1cd32e) from v3.1 branch worked.
/cc @bluwy

Thanks for finding the cause sapphi. #10388 should fix this issue as var declarations was accidentally treated as block-scoped instead of function-scoped in #10300


Missing device configuration: Minoston MP31ZP

Checklist

[X] It is not in the configuration DB

[X] It was not merged recently or has a pending PR

Which device is missing?
Minoston MP31ZP
Manufacturer ID
786
Product Type
65280
Product ID
65294
Firmware Version
FW 1.0; SDK 7.13.10
Is the device listed on the Z-Wave Alliance website?
https://products.z-wavealliance.org/products/4626?selectedFrequencyId=-1
Do you have a manual?
No response

Seems like this device is showing up as the MP21ZP not the MP31ZP

See https://github.com/zwave-js/node-zwave-js/pull/4882 and https://github.com/zwave-js/node-zwave-js/pull/4894
It seems like Minoston reused the MP21ZP firmware, making the MP31ZP internally the same.


Unhandled Exception: NoSuchMethodError: The getter 'closeCode' was called on null.

I have been trying for many days still don't understand what problem is this.
My URL is working on the web but not on the app from peer to peer connection dialog.
https://115.166.***.178:8086/
I don't get how a mobile device will connect with web video calling.

There are two points to note about mobile connection:

If your web server uses a valid domain name and SSL certificate, the mobile terminal can connect directly.
If not, you need to modify the code to connect your self-signed WSS server using the _connectForSelfSignedCert function.

@cloudwebrtc  yes manually  I have added certificate over the server and it works. and change my IP URL WSS to WS and port 4442 it works from mobile to web.
url= ws//115.166.***.178:4442   it works.

@cloudwebrtc yes manually I have added certificate over the server and it works. and change my IP URL WSS to WS and port 4442 it works from mobile to web.
url= ws//115.166.***.178:4442  it works.

This solved my issue. You mean  'url= ws://115.166.***.178:4442 '


Raspberry Pi Doc

Hi,
One thing I noticed when running npm install after following the instructions some of the build fails because the kerberos dev stuff isn't installed.
The docs just need to say to run
sudo apt-get install libkrb5-dev
before running npm install
Fantastic project by the way, you've done a great job, love it.
cheers,

Ah gotcha - thanks for the report! Would you mind updating the Wiki page with that?

Thanks, I just did, never realised I could.
thanks again,

On 29 Oct 2015, at 16:42, Nick Farina<EMAIL_ADDRESS>wrote:
Ah gotcha - thanks for the report! Would you mind updating the Wiki page with that?
—
Reply to this email directly or view it on GitHub https://github.com/nfarina/homebridge/issues/344#issuecomment-152242730.


Provide a stable html-escape API

See notes:

there is no stable API with Qt binding
there is no stable API with Python
It is maybe not a good idea to use cgi mudule, or xml module dependencies

Looks good to me


Switching view tabs with hotkeys

Hello!
It would be a pleasure to switch view tabs via ctrl+TAB and ctrl+shift+TAB

Those key bindings are not possible as they are reserved in the system. There are some navigation shortcuts you can see in the Window->Navigation menu.

There are some navigation shortcuts you can see in the Window->Navigation menu.

@smirnoww : The ones you should look at are CTRL + F6 and CTRL + SHIFT + F6.
@Phillipus : I find the menu entries name misleading as they use the term "View" for Eclipse Views, not ArchiMate view, and "Editor" instead of (ArchiMate) "View". I'd suggest to:

use "Pane" instead of "View"
use "View" instead of "Editor"

@jbsarrodie I can't change them, they come with Eclipse.

@jbsarrodie I can't change them, they come with Eclipse.

Scrub that...there might be a way...

The menu item text can be changed but the text on the popup that appears cannot be changed.


Marking animation as loop in Editor does not work. We have to set loop = true in code.

Godot version:

Beta 3.2.2-beta4 for Windows.
OS/device including version:

Windows 10
Issue description:

I marked animations as loop in Editor. They play as expected in the Editor. However, they stop in the game. I have to set them to loop in code to work.
Steps to reproduce:
In the included code, check Cube scene check_loop_bug boolean to true and play the game. We can see animations not looping. When this boolean is false game works as expected because loop is set in code.
Minimal reproduction project:
Here it is:
Bug-Report-Loop.zip

I was able to reproduce on 3.2 branch 7f67674 but doesn't seem to happen anymore on master 84e7552 Windows 10

This is the same issue as #20467
The animation Player in the given scene is imported from the .glb file.

Can confirm this problem is still present in 3.2.3 with both GLB and FBX.

Can confirm this problem is still present in 3.2.3 with both GLB and FBX.

Temporary workaround for anyone who encounters this issue.
Export your animations with a -loop suffix on the respective animation clip for your model and Godot will automatically import it as a looping clip.

Temporary workaround for anyone who encounters this issue.
Export your animations with a -loop suffix on the respective animation clip for your model and Godot will automatically import it as a looping clip.

Duplicate of #34394

Godot version:
Beta 3.2.2-beta4 for Windows.
OS/device including version:
Windows 10
Issue description:
I marked animations as loop in Editor. They play as expected in the Editor. However, they stop in the game. I have to set them to loop in code to work.
Steps to reproduce: In the included code, check Cube scene check_loop_bug boolean to true and play the game. We can see animations not looping. When this boolean is false game works as expected because loop is set in code.
Minimal reproduction project: Here it is: Bug-Report-Loop.zip

Same issue. How to loop it in code?


5-6 seconds - or more, before model load and client (user) can use the page - how to improve?

I'm currently testing this interesting nsfwjs module on a live server.
I activate this code:
pdModel = await nsfwjs.load('model/');
via a call to an async function triggered by the onLoad event.
The 6 model files are fetched from the disk and is using approximately 960ms to load (I see this using Chrome inspector - Network tab).
After this I have to wait approx. 5-6 seconds before I get any response from the page. From what happens in the Memory tab in Inspector, it looks like the nsfwjs load is using this time to do something.
After the 5-6 seconds. I can select an image. When that is done, this code (in another async function):
var img = new Image();
img.src = document.getElementById('img_url').src;
var predictions = await pdModel.classify(img);
console.log('Predictions', predictions);
is using about 1,5 - 2 seconds to finish.
This is acceptable time for me.
This time usage of about 1.5 -2 seconds - is consistent.
But the usage of 5-6 seconds nsfwjs use to do something - before anything else can happen in the page (e.g. select an image to analyze), is quite a long time - for an end user which want to do something as soon as possible.
So... is there a way to speed up the time nsfwjs uses to finish this part:
pdModel = await nsfwjs.load('model/');
and whatever it does before the page is ready for handling user triggered events in the page?
Could the preparation work done by nsfwjs (these first 5-6 seconds) maybe be split up in 2 parts, and then do half the job - onLoad, and then the second part - which is triggered to happen when an image actually is selected by the user?

Try updating your version of Tensorflow maybe?

I'm using this:
https://cdn.jsdelivr.net/npm/<EMAIL_ADDRESS>Is there a newer version available?

You should definitely keep track of what CDN packages are available
https://cdnjs.com/libraries/tensorflow

Thanks for that information. Now tested with latest tfjs  1.1.0.
Same results.
Any idea on how to not make the client (user) wait "so long" (5-6 seconds) before he can interact with the page (see suggestion in the end of thread start)?

Hrmmm. I really don't know. Is your code open source? I can take a look if so.

Thanks for that, but the code is currently beeing tested on a web hotel behind "closed doors".
But here is the information that is relevant:
`

const nsfwjs = require('nsfwjs');
var pdModel;
async function loadAwait() {
pdModel = await nsfwjs.load('model/');
}
async function analyzeImageAwait(){
if (pdModel){
	var img = new Image();
	img.src = document.getElementById('img_url').src;
	var predictions = await pdModel.classify(img);
	console.log('Predictions', predictions);
    }

}
`
in the body element I use this:
onload="loadAwait()"
on the server I have these sub directories with the following files:
model
files:
group1-shard1of6
group1-shard2of6
group1-shard3of6
group1-shard4of6
group1-shard5of6
group1-shard6of6
model.json
nsfwjs/
files:
bundle.js
tf.min.js.map
<EMAIL_ADDRESS>Except for this the main file only contain an input file field where I can select an image on the computer, and some javascript code which copy the relevant file information to the img.src with id img_url - like this:
function img_pathUrl(input){      $('#img_url')[0].src = (window.URL ? URL : webkitURL).createObjectURL(input.files[0]);      setTimeout(analyzeImageAwait, 1000); }
But as allready mentioned the delay of 5-6 seconds happens before I can click the file button which activate the img_pathUrl function above.
What I can do except for this, is to send you the main html file that I use. Then you can eventually test the code in your own localhost or webhotel.
If you want this file, please mention your email address.
I know that requesting the main file the first time will cause a delay of at least 7-9 seconds when the nsfwjs.load('model/') is activated (using live web hotel), due to the download of the large model files. But after the first request, those files will be fetched from the memory or from disc. It is the delay of 5-6 seconds in this "final mode" I find is a bit too much to wait for the end user, and would like to see a way to make more responsive for the client (end user) - if possible.

So this is going to sound strange.  Neither NSFWJS or your page have this delay for me.  When I load your page, the first time, it downloads the model and then I was immediately able to access the UI.
The second time I access the page, it reconfirms the cache for a few seconds, but the very second your loader goes away, I'm able to click.
Perhaps it's a browser/computer difference?  Would you be willing to google/run some other TensorflowJS examples and see if this happens across the board for you?
Perhaps it's when the model is loaded into memory?   Instant for some, slower on other computers?

What is the load time in seconds (which shows in blue text in page after loader hides) when the model files loads from disk, when you use this page?
nude-test-await

Cold start

Second page load

For me, the spinner shows me that the UI would be inaccessible.  But the second it goes away I can click.
Side note
As far as I can tell, this is verifying local cache and loading the model into memory.
I might train a smaller model soon and see if I can get it nearly as accurate.  If so, that will be helpful with these kinds of times.  As you can guess, training models can take a long time.

My guess is that both cold start and second page load, fetched the model files from disk (and not from the server).
The thing is, there is a huge difference between time to fetch the model files from disk, which on my computer takes only 400 - 600 milliseconds, and the total time the load stuff actually works.
Mye question is related to if there is a way to make the load time (except for the 400 - 600 milliseconds for the files to be read from disk cache) shorter, or if possible to divide this work into e.g. two parts. If the latter (two parts), then one part could be activated when user selects the image, and the other part when he submit the form. This way - the wait, would seem shorter for end user - in my opinion.
But press F12 to show the inspector (in Chrome), and click the Network "tab", before refreshing the page - then you see what I mean.

GantMan, there is something called Web Workers in Javascript. Is it possible to load the model in a web worker, and then make the model load behind the scenes, so to speak, so the user does not have to wait until the model has loaded before i starts to interact with the web page.
When I have this in the worker.js
importScripts('tfjs@1.1.0.js');  */ importScripts('bundle.js');
Then I get these errors in the console:
Uncaught Error: Could not find a global object     at engine.ts:829     at engine.ts:818     at engine.ts:836     at tfjs@1.1.0.js:2     at tfjs@1.1.0.js:2     at worker.js:13
If I click the "at engine.ts:829, it reports error at this line:
throw new Error('Could not find a global object');
So, do you know if it is possible to run the nsfwjs module from within a web worker?
More about web workers here: Multithreading Java script

Personally, I haven't worked with Web Workers just yet, but some searching found this discussion which I think should be noted:  https://github.com/tensorflow/tfjs/issues/102

Personally, I haven't worked with Web Workers just yet, but some searching found this discussion which I think should be noted: tensorflow/tfjs#102

Thanks for that information, just what I needed.
But could you please tell me if it is possible in some way to not use the model for Drawing and Neutral. That is, just download (load) the model files for analyzing image for Porn, Sexy and Hentai? Thereby save time by not having to download the two mentioned model files (Drawing and Neutral) to the client, and hopefully awoid that the model.classify() use time on testing for Drawing and Neutral.

Unfortunately the model is all in one and would not be made smaller by removing classifications.
However!  I'm nearing the end of training a smaller and nearly as accurate model. Expect that soon. It should speed everything up

Look forward to test that.
But, if you trained a model to recognize only Porn, Sexy and Hentai images, or just Porn, would that model (the files) not be smaller than a model trained to recognize (predict) Porn, Sexy, Hentai, Drawing and Neutral?

So the model is architected first and only the final layer is affected in size by number of possibilities.
Now, fewer possibilities could be supported by a smaller model architecture, but that would require building a smaller model and retraining.  Aka lots of time. It's hard to remove those parts without throwing the whole model out of balance. As far as I know it's not even a practice.  But it should be! That would be cool.
Anyhow, best case is to get this new smaller model I'm making and do a conditional on the predictions.
I fully believe model optimization will become a very real thing in the next 5 to 10 years. Just a simple solution.
So

Is there noe way to run it non-blocking? Which means it should run async in background with a callback when it finishes...

danieldaeschle:
You can load the model and analyze the images in a non blocking mode. Just create the NSFWJS and load the model in a web worker - then it run in a separate thread on the client. I have tested this, and it works in latest versions of Chrome and Opera. This way the web page UI is not blocked by the loading procedure of NSFWJS.
@GantMan , could you please confirm if the suggestion below is technically possible:
After figuring out that I can load the NSFWJS module in a web worker, my next thought was if it was possible to load the data from the different model files using separate Web workers, and just send the loaded array or whatever is loaded, to the main web worker (the one you want to use for analyzing the images). This way the data from the model files could be loaded in parallell - and could probably minimize the total load time - which would be especially beneficial if the client is on a mobile device, but should be faster on all devices. Note: It is possible to send e.g. arrays and objects from a worker back to main script (page) and from that script to another worker.

@bongobongo like i already said, i want to create a chrome extension. which does not support web workers. i just tried it :(

@bongobongo like i already said, i want to create a chrome extension which does not support web workers. i just tried it :(

@danieldaeschle , have you looked into this:
https://bugs.chromium.org/p/chromium/issues/detail?id=357664#c3

I checked both. Nothing works. They are 5 years old...
bongobongo<EMAIL_ADDRESS>schrieb am Di., 30. Apr. 2019, 17:36:

@bongobongo https://github.com/bongobongo like i already said, i want
to create a chrome extension which does not support web workers. i just
tried it :(
@danieldaeschle https://github.com/danieldaeschle , have you looked
into this:
https://bugs.chromium.org/p/chromium/issues/detail?id=357664#c3
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/infinitered/nsfwjs/issues/94#issuecomment-488001585,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHKTWHAKXDBRIT67JTJT5JLPTBRQNANCNFSM4HHYUUNA
.

@danieldaeschle , do you know that you have to use OffscreenCanvas to use the NSFWJS module in a Web Worker? Like this - place it first in the Web Worker JavaScript file:
`if (typeof OffscreenCanvas !== 'undefined') {
// currently tested and works in my Chrome and Opera
self.document = {
    createElement: () => {
        return new OffscreenCanvas(640, 480);
    }
};
self.window = self;
self.screen = {
    width: 640,
    height: 480
};
self.HTMLVideoElement = function() {};
self.HTMLImageElement = function() {};
self.HTMLCanvasElement = OffscreenCanvas;

}`

I tried it with cpu instead of gpu which does not require a canvas. That
also didn't work.
Daniel Däschle<EMAIL_ADDRESS>schrieb am Di., 30. Apr. 2019,
17:48:

Thank you, i'll try this.
bongobongo<EMAIL_ADDRESS>schrieb am Di., 30. Apr. 2019,
17:46:

@danieldaeschle https://github.com/danieldaeschle , do you know that
you have to use OffscreenCanvas to use the NSFWJS module in a Web Worker?
Like this - place it first in the Web Worker JavaScript file:
`if (typeof OffscreenCanvas !== 'undefined') {
// currently tested and works in my Chrome and Opera
self.document = {
createElement: () => {
return new OffscreenCanvas(640, 480);
}
};
self.window = self;
self.screen = {
width: 640,
height: 480
};
self.HTMLVideoElement = function() {};
self.HTMLImageElement = function() {};
self.HTMLCanvasElement = OffscreenCanvas;
}`
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/infinitered/nsfwjs/issues/94#issuecomment-488005415,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHKTWHCTQVYJE3R2BGOUHPLPTBSWDANCNFSM4HHYUUNA
.

@danieldaeschle - did u have any progress on making the chrome extension? I have just started looking building one myself, and then saw ur comment above saying u were trying it.

I'd be interested in seeing how this is performing on the latest TensorFlow.js
I know they've been working on the blocking UI problem.

Is there any progress in solving the blocking UI issue?

I've found using the small model doesn't even appear to cause any complication.   I haven't tried the larger model in a while.

Which small model? How long does it take to initialize?

By default https://nsfwjs.com/  uses the small model.  You can change it to the larger model in the dropdown.  I have a new small-medium sized model that is more accurate, as well.

@GantMan https://link.springer.com/chapter/10.1007/978-3-030-39575-9_31

As a point of reference, I make a Firefox plugin that loads a MobileNetV2-based model and see similar performance for load times. I've done a good bit of poking on it but so far I haven't been able to significantly reduce the load time as it is just TF.js. There's a progress handler one can look at, but the output wasn't terribly helpful. One thing that generally helped performance was getting a graph model vs. a layers model when doing tensorflowjs_converter as it can run through the graph optimizer, but much of that was on inference time rather than load time. I'm guessing this is already being run through the graph optimizer, though.

Hmmm, interesting question. I haven't dug into the internals of how it loads enough to know if that would help or not. In Python it wouldn't be hard to stitch them back together, but in TF.js I'm not so sure...


Recovery from network failures

I was browsing the crate to understand if there is any form of automatic recovery when a network failure occurs (similar to what is provided by the .NET client here).
If not, is this something you'd be interested to integrate into lapin or do you want to keep lapin as thin of a client as possible?

From my experiments it seems I can detect when the connection with the broker is lost because my consumers will receive a
ProtocolError(AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'") })

but I would then have to figure out how to re-establish that connection afterwards (in particular if using set_delegate instead of the iterator interface).

There is currently no auto-reconnection feature in lapin, indeed.
Splitting the AMQP stuff in its own async fn and using the on_error hook to re-spawn it is the best way to handle this atm, I think. As it's usually quite easy to do just that, and adding automatic recovery can add a lot of complexity, there hasn't been any plan to implement such a thing yet, but we could explore this idea at some point

Thanks a lot for the advice!
Somewhat related: what is the story when it comes to resource management/clean up of resources?
I noticed that dropping a connection does not affect the behavior of channels - they keep working fine, including their consumers.
Dropping channels instead cancels the consumers (they receive None).
When is the connection closed? If all channels on top of it are dropped? Does it always need to be closed explicitly?
Trying to understand how to implement fault-tolerance without leaking resources.

Since 1.0, channels are closed once their last reference is dropped, unless they have been manually closed before.
Connection is closed once the connection and all channels have been dropped, unless it has been manually closed before.

Amazing, thank you :+1:

There is currently no auto-reconnection feature in lapin, indeed.
Splitting the AMQP stuff in its own async fn and using the on_error hook to re-spawn it is the best way to handle this atm, I think. As it's usually quite easy to do just that, and adding automatic recovery can add a lot of complexity, there hasn't been any plan to implement such a thing yet, but we could explore this idea at some point

Is there any demo ?
e.g.  A publish message to rabbitmq broker method, how to recover by on_error hook ?

Well, it's not trivial, and the problem is inherent to the AMQ protocol.
WRT basic_publish:

In "standard" mode, there is no way to know if a message was successfully published, you can only know when you finished writing all the bytes to the socket. In case of a network error, you can just intercept the error, reconnect and retry sending.
In "publisher-confirms" mode (enabled with confirm_select), you get an ack once the message has finished processing on the rabbitmq side. You can then track in your application a list of "on-flight" messages and drop them once you get the ack, or retry them on network failure + reconnection. But if the network fails just when the rabbitmq server is done dealing with the message, right when it tries to send you the ack... it won't try to send it again once the reconnection is done, so you'll publish it twice.

There is no proper way of "resuming" an AMQP session.
Other caveat, basic_consume:
You got a message, you finished dealing with it, and just when you want to ack it: network failure. The server requeues the message and unless you have a proper way to rollback what you've done with it, you'll apply your actions twice.
Even trickier: you get a message a start a rather long computation. Meanwhile, network is lost and you reconnect, you do other stuff. Once the computation is done, you ack => unrecoverable error. The "delivery_tag" used for ack is incremental, per channel, per connection and just gets back to 0 on reconnect. It's not some kind of uuid or such.
What most clients do is basically keep some kind of topology descriptor around, and just redeclare the queues and the consumers, leaving all the rest to you, but that can lead to very awkward situations like this and it has bitten me several times in the past, that why as of now I didn't want to implement something that people will expect to "just work" when things are really not that simple and some things just aren't possible to handle automatically.
All of these and more examples should definitely be somewhere in the documentation

I agree that examples showing how to register an on_error hook would greatly help. I'm trying to figure out how to implement some kind of limited client recovery as described here. An opinionated way of "resuming" a session might benefit most casual users, but I agree that allowing you to do it yourself is probably the best bet.
That's what I'm trying, but I'm stumped right at the first step - on_error is supposed to call my async function to reconnect which Tokio is supposed to execute - am I supposed to start a Tokio task inside it? How do I know it's ready? A minimal working example that shows how to achieve this would help, as I'll probably end up overcomplicating it somehow.

I know that people use on_error for this but haven't actually used it this way myself (I use it to just crash the program and have systemd restart it).
Here is one way to handle reconnections: https://github.com/CleverCloud/lapin/blob/lapin-1.x/examples/reconnect.rs
Sample output with example started without rabbitmq running, then starting rabbitmq, then restarting rabbitmq:
[2020-07-22T08:05:30Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:05:32Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:05:34Z INFO  reconnect] CONNECTED
[2020-07-22T08:05:34Z INFO  reconnect] Declared queue Queue { name: ShortString("hello"), message_count: 0, consumer_count: 0 }
[2020-07-22T08:05:34Z INFO  reconnect] will consume
[2020-07-22T08:06:00Z ERROR lapin::channel] Connection closed on channel 0 by 0:0 => AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'") } => CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
[2020-07-22T08:06:00Z ERROR lapin::channels] Connection error
thread '<unnamed>' panicked at 'error in consumer: ProtocolError(AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure with reason \'shutdown\'") })', examples/reconnect.rs:57:39
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2020-07-22T08:06:00Z ERROR reconnect] Error: protocol error: AMQP hard error: CONNECTION-FORCED: CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
[2020-07-22T08:06:02Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:04Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:06Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:08Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:10Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:12Z ERROR reconnect] Error: IO error: Connection refused (os error 111)
[2020-07-22T08:06:14Z INFO  reconnect] CONNECTED
[2020-07-22T08:06:15Z INFO  reconnect] Declared queue Queue { name: ShortString("hello"), message_count: 0, consumer_count: 0 }
[2020-07-22T08:06:15Z INFO  reconnect] will consume

FYI, I just landed #311 which basically allows you to save your current topology (exchanges, queues, consumers) and load it back from a json or such (documentation incoming).
With this in place, we just need a way for a failed connection to "absorb" a new one (so that streams etc stay the same as before) to get proper automatic reconnection working.

Now tracked in #312


2: Improve error handling

Complements the already closed #2

Codecov Report

Merging #20 (900ae1f) into master (31e5a40) will increase coverage by 6.79%.
The diff coverage is 74.19%.

@@            Coverage Diff             @@
##           master      #20      +/-   ##
==========================================
+ Coverage   79.61%   86.40%   +6.79%
==========================================
  Files           3        3
  Lines         103      103
==========================================
+ Hits           82       89       +7
+ Misses         12        6       -6
+ Partials        9        8       -1

Impacted Files
Coverage Δ

pkg/task/repository.go
63.63% <60.00%> (+2.34%)
:arrow_up:

pkg/config/config.go
95.74% <87.50%> (+14.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31e5a40...900ae1f. Read the comment docs.


ScrollView: when always_overscroll is enabled and scroll_type is set to 'bars', touches in bar area will be always blocked even when scroll is disabled

Software Versions

Python: 3.8
OS: Windows 10
Kivy: 2.0.0
Kivy installation method: Installer

Describe the bug
When a scrollview has always_overscroll enabled, and scroll_type is set to 'bars', touches in the bar area will always be intercepted (not passed to children), regardless of if do_scroll is disabled.
Proposed solution
This bug can be fixed by changing lines 711 and 712 (currently) of scrollview.py to:
        width_scrollable = (self.always_overscroll and self.do_scroll_x) or vp.width > self.width
        height_scrollable = (self.always_overscroll and self.do_scroll_y) or vp.height > self.height

Expected behavior
When do_scroll_x or do_scroll_y is disabled, touches in the horizontal or vertical scrollbar area should not be intercepted.
To Reproduce
Run the following code:
from kivy.app import App
from kivy.lang.builder import Builder
KV = """
BoxLayout:
    ScrollView:
        do_scroll_x: True
        do_scroll_y: False
        scroll_type: ['bars']
        bar_width: 40
        Button:
            size_hint: None, None
            height: self.parent.height - 40
            width: 1000
"""
class Test(App):
    def build(self):
        return Builder.load_string(KV)
Test().run()

Note that the button will not respond to clicks on the right-side.

Thanks for the report! can you do a PR with that change? it's simple enough you could do it through github, go to the file and edit it, it'll create a pull request when you save.
This way you get credit for the fix :).

Presumably the PR fixed it!


Updates to index, animate.css, and animate.min.css

I updated the animate.css and animate.min.css files within this branch to reflect the current version. Further, I optimized the zoom animations, which originally were in a optgroup called "zooms", while everything else has exits and entrances. Zoom now also has zoom exits, and zoom entrances.
Further, I added SlideOutDown, and SlideInUp options to their respective optgroups, as they are possible animations.
Happy to support a project that has helped me so much. Thanks!

Added the updated version of animate.min.css to the pull request


chore(allow-scripts): handle spaces in execPath in Windows tests

Fixes breaking test on Windows when nodejs installed under C:\Program Files

great work

Thanks! Just replaced p with inPath, otherwise identical to before.


[CUDA] GPU build on Ubuntu 20.04 crashing on training

Description
CUDA GPU version on Ubuntu 20.04 Linux crashes during training.
Reproducible example
import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate a dummy classification dataset
X, y = make_classification(n_samples=100000, n_features=20, n_classes=2, random_state=42)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train the LightGBM classifier with GPU support
clf = lgb.LGBMClassifier(
    objective='binary',
    device='cuda',
    verbose=1,
)

clf.fit(X_train, y_train)

# Predict and evaluate
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy}")

Environment info
LightGBM version or commit hash: 4.5.0
Command(s) you used to install LightGBM
conda install -c conda-forge 'lightgbm>=4.4.0'

I've tried installing several different ways and in Python 3.11 and 3.10. It always seems to end up with this error.
Additional Comments
It works fine on CPU or with the other GPU build, just not with CUDA.
The GPU is NVIDIA Tesla M60
The NVIDIA driver version is 535 and CUDA 12.2 (I think, may have also been 12.7). Also tried with 565.57.01 and 12.7.
The traceback I'm getting is this:
{
	"name": "LightGBMError",
	"message": "Check failed: (split_indices_block_size_data_partition) > (0) at /home/conda/feedstock_root/build_artifacts/liblightgbm_1728547676427/work/src/treelearner/cuda/cuda_data_partition.cpp, line 280 .
",
	"stack": "---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
Cell In[1], line 19
     12 # Create and train the LightGBM classifier with GPU support
     13 clf = lgb.LGBMClassifier(
     14     objective='binary',
     15     device='cuda',
     16     verbose=1,
     17 )
---> 19 clf.fit(X_train, y_train)
     21 # Predict and evaluate
     22 y_pred = clf.predict(X_test)

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/sklearn.py:1284, in LGBMClassifier.fit(self, X, y, sample_weight, init_score, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_metric, feature_name, categorical_feature, callbacks, init_model)
   1281         else:
   1282             valid_sets.append((valid_x, self._le.transform(valid_y)))
-> 1284 super().fit(
   1285     X,
   1286     _y,
   1287     sample_weight=sample_weight,
   1288     init_score=init_score,
   1289     eval_set=valid_sets,
   1290     eval_names=eval_names,
   1291     eval_sample_weight=eval_sample_weight,
   1292     eval_class_weight=eval_class_weight,
   1293     eval_init_score=eval_init_score,
   1294     eval_metric=eval_metric,
   1295     feature_name=feature_name,
   1296     categorical_feature=categorical_feature,
   1297     callbacks=callbacks,
   1298     init_model=init_model,
   1299 )
   1300 return self

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/sklearn.py:955, in LGBMModel.fit(self, X, y, sample_weight, init_score, group, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_group, eval_metric, feature_name, categorical_feature, callbacks, init_model)
    952 evals_result: _EvalResultDict = {}
    953 callbacks.append(record_evaluation(evals_result))
--> 955 self._Booster = train(
    956     params=params,
    957     train_set=train_set,
    958     num_boost_round=self.n_estimators,
    959     valid_sets=valid_sets,
    960     valid_names=eval_names,
    961     feval=eval_metrics_callable,  # type: ignore[arg-type]
    962     init_model=init_model,
    963     callbacks=callbacks,
    964 )
    966 self._evals_result = evals_result
    967 self._best_iteration = self._Booster.best_iteration

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/engine.py:307, in train(params, train_set, num_boost_round, valid_sets, valid_names, feval, init_model, feature_name, categorical_feature, keep_training_booster, callbacks)
    295 for cb in callbacks_before_iter:
    296     cb(
    297         callback.CallbackEnv(
    298             model=booster,
   (...)
    304         )
    305     )
--> 307 booster.update(fobj=fobj)
    309 evaluation_result_list: List[_LGBM_BoosterEvalMethodResultType] = []
    310 # check evaluation result.

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/basic.py:4135, in Booster.update(self, train_set, fobj)
   4133 if self.__set_objective_to_none:
   4134     raise LightGBMError(\"Cannot update due to null objective function.\")
-> 4135 _safe_call(
   4136     _LIB.LGBM_BoosterUpdateOneIter(
   4137         self._handle,
   4138         ctypes.byref(is_finished),
   4139     )
   4140 )
   4141 self.__is_predicted_cur_iter = [False for _ in range(self.__num_dataset)]
   4142 return is_finished.value == 1

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/basic.py:296, in _safe_call(ret)
    288 \"\"\"Check the return value from C API call.
    289
    290 Parameters
   (...)
    293     The return value from C API calls.
    294 \"\"\"
    295 if ret != 0:
--> 296     raise LightGBMError(_LIB.LGBM_GetLastError().decode(\"utf-8\"))

LightGBMError: Check failed: (split_indices_block_size_data_partition) > (0) at /home/conda/feedstock_root/build_artifacts/liblightgbm_1728547676427/work/src/treelearner/cuda/cuda_data_partition.cpp, line 280 .
"
}

Thanks very much for the excellent report.
I think that maybe LightGBM's CUDA build doesn't currently support Tesla M60 (CUDA compute capability 5.2). The oldest compute capability we support is 6.0 (Pascal).
https://github.com/microsoft/LightGBM/blob/dc0ed538aa09b755ba4a293dc6f344da51674260/CMakeLists.txt#L226-L239
But I'm not sure. @shiyu1994 can you help investigate this report?

Oh, I see, thanks. But only up to 7.5 looks like? That seems it would include other options on azure like V100 and T4.

Ah yeah, that was it - on M60 it doesn't work, on V100 it does. D'oh.

I assume find_package(CUDAToolkit 11.0 REQUIRED) means anything >= 11.0?

Yes. This line:
https://github.com/microsoft/LightGBM/blob/dc0ed538aa09b755ba4a293dc6f344da51674260/CMakeLists.txt#L220
There are some details on this here:

https://cmake.org/cmake/help/latest/command/find_package.html#find-package-version-format
https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html
https://docs.nvidia.com/deploy/cuda-compatibility/

You can also see some hints about this in the logs of a recent CI job here using CUDA 12.6.
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.146.02             Driver Version: 535.146.02   CUDA Version: 12.6     |
...
-- The CUDA compiler identification is NVIDIA 12.6.68
...
-- Found CUDAToolkit: /usr/local/cuda/targets/x86_64-linux/include (found suitable version "12.6.68", minimum required is "11.0")

(build link)
If you want to see how the conda-forge packages are built, see https://github.com/conda-forge/lightgbm-feedstock.

BTW, is the CUDA build substantially better/faster than the OpenCL version?

Yes. The OpenCL version here is basically unmaintained at this point: https://github.com/microsoft/LightGBM/issues/4946#issuecomment-2126170630
The CUDA version does more work on the GPU, with less copying between host and device. It's more actively developed and more thoroughly tested. If you have a compatible NVIDIA GPU, prefer "device": "cuda" to "device": "gpu" for LightGBM.

Ah yeah, that was it - on M60 it doesn't work, on V100 it does.

Ah great! Sorry there is not a more informative error there. I'm glad it's working well for you on using V100s.
I'm going to close this at it seems that that resolves the issue, but please post if you have additional questions. At this point, we won't add support for older GPUs. Even RAPIDS dropped support for Pascal earlier this year: https://docs.rapids.ai/notices/rsn0034/

I just noticed that you double-posted this here and on Stack Overflow (link). Please do not do that.
Maintainers here also monitor the [lightgbm] tag on Stack Overflow. I could have been spending time preparing an answer here while another maintainer was spending time answering your Stack Overflow post, which would have been a waste of maintainers' limited attention that could otherwise have been spent improving this project. Double-posting also makes it less likely that others with a similar question will find the relevant discussion and answer.
Since we've answered this here and your Stack Overflow post hasn't received any votes or comments, I think you should delete it.

Got it, thanks for all the helpful info. Should we just close the SO question (needs one more vote)?

Thanks for that. I don't have sufficient reputation there to vote to close it, maybe someone else will come along. I appreciate that you linked back to this issue in your answer, that helps! We just want to be sure we're making good use of everyone's time.
Thanks again for the excellent report, the reproducible example and details you shared made it easy to get to a resolution quickly, and I know that takes some effort to put together.


Add an explicit wait for all HW to be ready on startup

Seen a few cases on release where encoder or DSP aren't fully up.
dmonitoringmodeld + encoderd
870e88b8c4490f87|2024-01-03--08-03-46--0
6e6fdf358e8f77a2|2024-01-03--10-05-27--0
5aa0e49a65c57480|2024-01-04--16-09-52--0

Still happens to dmonitoringmodeld sometimes


Give extrinsicRPY a clearer definition

correct the description of extrinsicPRY in README and modify some codes to ensure that the algorithm works correctly with new extrinsicRPY definition.

Modify the description of extrinsicRPY in the IMU alignment section of the readme file.
Modify the extrinsicRPY in the "params.yaml". The old extrinsicRPY represents q_bl, the new one represents q_lb.
Modify ParamServer's construction funtion in "utility.h", This change is to make the input q_lb become q_bl.
Add note about imu2Lidar and lidar2imu in "imuPreintegration.cpp", making the meaning of these two parameters more clear.

After testing, the modified algorithm with new extrinsicRPY definition can run normally on the Walking dataset.

Thank you.


LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20230106164035861 to main

LEGO: Pull request from lego/hb_04604851-bac4-4681-9f74-73de611d6e48_20230106164035861 to main with localized lcls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.02%) to 42.094%

Files with Coverage Reduction
New Missed Lines
%

extensions/notebook/src/dialog/configurePython/configurePathPage.ts
1
57.02%

Totals

Change from base Build<PHONE_NUMBER>:
0.02%

Covered Lines:
28597

Relevant Lines:
63115

💛  - Coveralls


Add announcement for aruba 0.10.0

@aslakhellesoy @mattwynne Ping. Also announced on http://www.rubyflow.com/p/dmeya1-aruba-0100-released and on https://groups.google.com/forum/#!forum/cukes.
@aslakhellesoy Is this forum moderated? It takes a long time until posts appear there.

@aslakhellesoy Can you put me on the liste of allowed writers on cukes google channel?

Done. I deleted your first pending message and let the 2nd one through (they were very similar). You can post forever now.

@aslakhellesoy Thanks.

Can anyone of you two change the description of aruba on Github? It would be great if it would be "Extension for popular TDD and BDD frameworks like "Cucumber", "RSpec" and "Minitest" to make testing of commandline applications meaningful, easy and fun." instead of just "CLI Steps for Cucumber". Nowadays it's more that just "cli steps". Thanks.

Done! Changed it to "Test command-line applications with Cucumber-Ruby, RSpec or Minitest"

:-) Thanks.


Release v0.10.0

CLA: The additional contributors confirmed on their respective PRs that they give the permission to push their commits to the master branch.

Pull Request Test Coverage Report for Build 1214

471 of 499 (94.39%)  changed or added relevant lines in 22 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 92.728%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

container/grid/grid.go
84
85
98.82%

internal/segdisp/segdisp.go
31
33
93.94%

internal/segdisp/dotseg/dotseg.go
84
90
93.33%

widgets/segmentdisplay/segmentdisplay.go
17
26
65.38%

internal/segdisp/dotseg/attributes.go
46
56
82.14%

Files with Coverage Reduction
New Missed Lines
%

widgets/segmentdisplay/segmentdisplay.go
1
85.71%

Totals

Change from base Build 1176:
0.2%

Covered Lines:
7498

Relevant Lines:
8086

💛  - Coveralls

Pull Request Test Coverage Report for Build 1214

471 of 499 (94.39%)  changed or added relevant lines in 22 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 92.728%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

container/grid/grid.go
84
85
98.82%

internal/segdisp/segdisp.go
31
33
93.94%

internal/segdisp/dotseg/dotseg.go
84
90
93.33%

widgets/segmentdisplay/segmentdisplay.go
17
26
65.38%

internal/segdisp/dotseg/attributes.go
46
56
82.14%

Files with Coverage Reduction
New Missed Lines
%

widgets/segmentdisplay/segmentdisplay.go
1
85.71%

Totals

Change from base Build 1176:
0.2%

Covered Lines:
7498

Relevant Lines:
8086

💛  - Coveralls


Move Any Corp (generic and semantic) into an /examples folder

Let's get them out of Codepen and into version control:

Generic AnyCorp
Semantic AnyCorp

Done and done.


rustup, and adjust timing tests

I saw a timing failure on macOS, so leave some more slack space.

@bors r+

:pushpin: Commit dcb2b30982352362a0f10cbd45be298b31306eff has been approved by RalfJung

:hourglass: Testing commit dcb2b30982352362a0f10cbd45be298b31306eff with merge 592d621b785bf9036bce4deeeaa3db13b19d589b...

:sunny: Test successful - checks-travis, status-appveyor
Approved by: RalfJung
Pushing 592d621b785bf9036bce4deeeaa3db13b19d589b to master...


Make params/routes unicode strings

A tentative PR addressing issue https://github.com/perwendel/spark/issues/266

There's something incomplete/not quite right with this still, if you change BooksIntegrationTest line 30 to
private static String AUTHOR = "Лев Толстой";

the test fails. This should be passing (and, in fact, should be part of the test). I'll take a closer look at it in the next couple of days and send you another PR.

I was looking at the wrong test, added a test in the right place, PR #276


Minder CLI improvements - table interface, refactor use of flags, context, etc.

The following PR:

Leverages the use of PersistentFlags and FlagGroups. This resulted in a lot of housekeeping on helper functions we don't really had to have since they are already provided by cobra/viper. Essentially there a no major code logic changes but because of this cleanup the PR looks way bigger than I expected.
Refactors the use of tables within the cli. Adds a Table interface with two supported tables (Simple and Glossy). Currently everything is migrated to the Simple one for consistency, but the glossy one is left for eventual PoC since it looks much better, but lacks the appropriate features (or we don't know how to implement them yet).
Adds provider and project values (I have started this earlier so I haven't rebased the latest context changes, will do though that's why it's WIP)
Went through the CLI stack of commands and made the codebase consistent, refer to the same default values, etc.
Fixed an issue where errors were showing twice
Went through all cli commands and made sure they use the cli.MessageAndReturn function. Updated the function to wrap a custom error type we later unwrap in ExitNicely so we can handle message outputs better.
Reduced the details input of an GRPC error to description only
Renamed --repo to name when doing repo register for consistency with the rest of the repo commands
Added a new context for each batched profile call
Converged on using the basic table rendering framework, at least for the time being. We can revisit this but let's not stretch this PR more.

What's left:

[x] Rebase the latest changes related to context and config files
[x] Finalise the tables usage
[x] Ensure a separate context is used for all CLI commands that batch GRPC calls
[x] Ensure and update docs to reflect the latest state
[x] Context.Project is UUID, not name. Either implement or fix that in the docs. (Added it in the description, the switch from UUID to name would be a separate PR)

Fixes: #1758
Fixes: #1754
Fixes: #1751
Fixes: #1749
Fixes: #1748

@rdimitrov what changed?

@rdimitrov what changed?

@JAORMX - Jakub merged his PR so I had to rebase. Also there was a diff in the make gen output.

Trying this locally, the tables look good! the only one that looks a little off is the whoami list which looks a bit too slim. Is there a way for us to enforce minimal width?
+---------------+--------------------------------------+
|      KEY      |                VALUE                 |
+---------------+--------------------------------------+
| Minder Server | api.stacklok.com:443                 |
+---------------+--------------------------------------+
| Project       | jaormx /                             |
|               | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx |
+---------------+--------------------------------------+

Added a minimum width for tables of type key/value 👍


Windows test failure with Debug Linux build

When running the Windows OE tests using a Linux build with -DCMAKE_BUILD_TYPE=Debug, the following unit test fails consistently:
8 - tests/backtrace (Failed)

The full reason behind this has been explained here.
We ended up adding -DUSE_DEBUG_MALLOC=OFF as a temporary workaround to avoid the failure, but it's not desirable.
This pull request https://github.com/Microsoft/openenclave/pull/1408 removes the temporary workaround and reproduces the current issue.

Hello @mikbras,
This issue is related to the oe_backtrace_symbols function added by you in this commit for v0.4.1 release.
The failure is happening when running the tests in Debug mode on Windows. These are the pieces of code from tests that fail:

https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/tests/backtrace/enc/enc.cpp#L99-L100
https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/tests/backtrace/enc/enc.cpp#L120-L121

The exact piece of code do not fail on Linux (using Debug mode as well) and oe_backtrace_symbols correctly returns the symbols, thus the assertion doesn't fail.
Digging further into the oe_backtrace_symbols implementation, we found out that the oe_call with OE_OCALL_BACKTRACE_SYMBOLS from:
https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/enclave/core/sgx/backtrace.c#L105-L110
correctly sets the args->ret, thus the function returned value is correct.
However, when running the tests on Windows, the args->ret value is always empty, even after executing oe_call with OE_OCALL_BACKTRACE_SYMBOLS.
Do you know if the function oe_backtrace_symbols is supposed to work on Windows as well ?
NOTES:

This failure is isolated to the unit tests executed with a Linux elf build on Windows, and it doesn't impact Windows cross-compiled builds.
This scenario is currently skipped by the CI, otherwise the failure will be blocking.

cc @johnkord @CodeMonkeyLeet @andschwa

@ionutbalutoiu No, I don't believe it works on Windows, see #961

@ionutbalutoiu No, I don't believe it works on Windows, see #961

Thank-you for the reference! We might need to add some extra ifdef condition for checking if not WIN32 here:
https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/tests/backtrace/enc/enc.cpp#L96-L103
and here:
https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/tests/backtrace/enc/enc.cpp#L119-L124
Or simply wait for the proper support of backtrace on Windows.
So far, our workaround for the CI is to force USE_DEBUG_MALLOC to OFF:
https://github.com/Microsoft/openenclave/blob/171f2c29906e1963e6cca3f25ae8ba562a334f05/.jenkins/Jenkinsfile#L95
And we have the flag disabled, regardless of the CMake logic from here:
https://github.com/Microsoft/openenclave/blob/4969be9fe68bdb6e8e76af04e1f53a9f989c331e/CMakeLists.txt#L144-L150

I enabled backtrace support on Windows and it works fine for me with debug-malloc. The PR is here #1860.
I enabled tests/backtrace, which now runs and I allocated and un-freed block in tests/echo (which I've since taken back out). It prints a proper backtrace.

@mikbras This is awesome news!
I just tested removing our workaround for the CI (force USE_DEBUG_MALLOC to OFF) on top of your PR https://github.com/microsoft/openenclave/pull/1860, and everything worked as expected.
Once your PR to enable backtrace on Windows is merged, we can have this issue fixed.

Fixed by https://github.com/microsoft/openenclave/pull/1860


How do i create a reward?

Where is the create reward command?
And no Copy & pasting the default YAML and removing the UUID and permission like it said does not in regenerate a new random one at all?
So... HOW?

Hi there, at the time of writing this comment there is no create reward command in game.
There was a bug in v3.1.2 with the reload command not working due to some changes I made. Could you try again with v3.1.3? Thanks