[neurips] Media Support (Videos + Slides)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a new site support request
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that none of provided URLs violate any copyrights or contain any DRM to the best of my knowledge
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and am willing to share it if required

Region
Global
Example URLs

Single video: https://videos.neurips.cc/video/slideslive-38922815
Single video: https://videos.neurips.cc/topic/machine learning/video/slideslive-38923348
Playlist: https://videos.neurips.cc/category/2142/
Playlist: https://videos.neurips.cc/topic/machine learning/

Provide a description that is worded well enough to be understood
Site has slides side by side video, would appreciate if slides can be downloaded side by side of videos by default.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
❯ ytf 'https://videos.neurips.cc/video/slideslive-38922815' -v
[debug] Command-line config: ['-F', '--list-subs', 'https://videos.neurips.cc/video/slideslive-38922815', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b64402] (pip)
[debug] Python 3.10.6 (CPython arm64 64bit) - macOS-13.0.1-arm64-arm-64bit (OpenSSL 1.1.1q  5 Jul 2022)
[debug] exe versions: ffmpeg 5.1.1 (setts), ffprobe 5.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] [generic] Extracting URL: https://videos.neurips.cc/video/slideslive-38922815
[generic] slideslive-38922815: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] slideslive-38922815: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://videos.neurips.cc/video/slideslive-38922815
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1485, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1561, in __extract_info
    ie_result = ie.extract(url)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 674, in extract
    ie_result = self._real_extract(url)
  File "/opt/homebrew/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 2721, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://videos.neurips.cc/video/slideslive-38922815

Related: #5784
Looks like neurips uses the videoken API instead of the standard slideslive embed JS though. Will need an extractor for that

Site has slides side by side video, would appreciate if slides can be downloaded side by side of videos

The current version of the slideslive extractor PR will download all video slides by default, and download all image slides if --write-all-thumbnails is passed (the majority of slides are still images). The slides are numbered to coincide with the primary video's chapter numbers. I think actually having the slides displayed side-by-side with the video would require a custom postprocessor for the site. Not sure how to go about doing that or how feasible it is. Suggestions welcome

slides ... downloaded ... by default.

@pukkandan Is this something that would be acceptable to do? The only way I can think of is to have the extractor set the write_all_thumbnails param to True. & then would there need to be something like a skip_slides extractor-arg since --no-write-thumbnail would not work?

and download all image slides if --write-all-thumbnails is passed (the majority of slides are still images).
I think actually having the slides displayed side-by-side with the video would require a custom postprocessor for the site.

This is out of scope. But, we could download all the images as a single mhtml file (like storyboards). This will put all images in a single file, but will still keep the videos separate from images - so I'm not sure whether it's worth it

Is this something that would be acceptable to do? The only way I can think of is to have the extractor set the write_all_thumbnails param to True

No

The site's API changed in the past week and now I can't get any of these example URLs to play in browser anymore (error 400 from API). @Ashish0804 are you still able to play any of these neurips videos?

Thanks for the update on the SlidesLive support. Download the video now is made super easy. However, how can I download the slides? When passing --write-all-thumbnails, I get 403's:
$ yt-dlp --write-all-thumbnails https://slideslive.com/38983994
[SlidesLive] Extracting URL: https://slideslive.com/38983994
[SlidesLive] 38983994: Downloading webpage
[SlidesLive] 38983994: Downloading player info
[SlidesLive] 38983994: Downloading slides JSON
[SlidesLive] 38983994: Downloading m3u8 information
[SlidesLive] 38983994: Downloading MPD manifest
[info] 38983994: Downloading 1 format(s): hls-4+dash-5
[info] Downloading video thumbnail cover ...
[info] Writing video thumbnail cover to: Zero-Shot AutoML with Pretrained Models [38983994].cover.jpg
[info] Downloading video thumbnail 022 ...
WARNING: Unable to download video thumbnail 022: HTTP Error 403: Forbidden
[info] Downloading video thumbnail 021 ...

When passing --write-all-thumbnails, I get 403's as shown below.

This will fix that:
diff --git a/yt_dlp/extractor/slideslive.py b/yt_dlp/extractor/slideslive.py
index 4268bfeaf..73bc51823 100644
--- a/yt_dlp/extractor/slideslive.py
+++ b/yt_dlp/extractor/slideslive.py
@@ -431,7 +431,7 @@ def _real_extract(self, url):
         slides_version = int(self._search_regex(
             r'https?://slides\.slideslive\.com/\d+/v(\d+)/\w+\.(?:json|xml)',
             slides_info_url, 'slides version', default=0))
-        if slides_version < 4:
+        if slides_version < 3:
             slide_url_template = 'https://cdn.slideslive.com/data/presentations/%s/slides/big/%s.jpg'
         else:
             slide_url_template = 'https://slides.slideslive.com/%s/slides/original/%s.png'

I didn't have enough sample data for v3 slides at the time. Now it looks like .png is the way to go with them. I'll fix

Back in the old days, we were able to access the user-view video and slides-view video

Are you talking about with the extractor? Looking over the history of slideslive.py I am not seeing when this was ever possible

Thanks for the fix; tested and works! Sorry, with "back in the old days" I meant that SlidesLive would provide us links to separate videos for user-view and slides-view. What would be your suggestion how to best merge both video and slides into one video? I used to merge two videos (already in-sync) with ffmpeg but with the .png's the syncing with the video becomes an issue as I'm not sure how to access the chapter meta info in the video through ffmpeg.

btw
yt-dlp --embed-chapters --write-all-thumbnails https://slideslive.com/38983994
yields the error
ERROR: unsupported operand type(s) for *: 'NoneType' and 'int'

ERROR: unsupported operand type(s) for *: 'NoneType' and 'int'

@ferreirafabio Thanks for bringing these issues to my attention, they should be fixed with #6024
Merging the slides with the video is out-of-scope for yt-dlp but it could possibly be done with a postprocessor plugin. When I have time I'll try looking into it

@bashonly I just wanted to quickly follow-up on the mentioned postprocessor plugin and ask, whether you had time to look into this to allow merging both video and slides into one video. Essentially, a large portion of the machine learning community (which submits their talks to slideslive.com) would immensely welcome this functionality :) Thanks a lot for the great work!


pyffmpeg FFprobe list index out of range

Hello
I tried pyffmpeg on Linux Mint 20.1
pip install pyffmpeg
First test:
from pyffmpeg import FFprobe
fp = FFprobe("/home/user/video.mp4")

Result:
  File "APP.py", line 721, in get_video_duration
    fp = FFprobe("/home/user/video.mp4")
  File "/home/user/.local/lib/python3.8/site-packages/pyffmpeg/pseudo_ffprobe.py", line 52, in __init__
    self.probe()
  File "/home/user/.local/lib/python3.8/site-packages/pyffmpeg/pseudo_ffprobe.py", line 236, in probe
    self._extract_all(str(stdout, 'utf-8'))
  File "/home/user/.local/lib/python3.8/site-packages/pyffmpeg/pseudo_ffprobe.py", line 81, in _extract_all
    all_streams = all_streams.split('Input')[1]
IndexError: list index out of range

I haven't been able to replicate the problem, please try another file and lets see.

I have the exact same problem.

Hi,
I have a similar problem, but I'm working on windows 10.
I use PIP on windows.
from pyffmpeg import FFprobe
xxx = FFprobe('Qc06.000_Qd00.000_h0.50_w2.00_NoDroplet_A.avi')
output:
2023-01-02 12:14:42,370 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - FFprobe initialised
2023-01-02 12:14:42,371 - pyffmpeg.misc.Paths - INFO - bin folder: C:\Users\User.pyffmpeg\bin
2023-01-02 12:14:42,371 - pyffmpeg.misc.Paths - INFO - Inside load_ffmpeg_bin
2023-01-02 12:14:42,372 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - ffmpeg bin: C:\Users\User.pyffmpeg\bin\ffmpeg.exe
2023-01-02 12:14:42,373 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - Inside probe
2023-01-02 12:14:42,373 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - Probing file: "Qc06.000_Qd00.000_h0.50_w2.00_NoDroplet_A.avi"
2023-01-02 12:14:42,374 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - Issuing commads ['C:\Users\User\.pyffmpeg\bin\ffmpeg.exe', '-y', '-i', 'Qc06.000_Qd00.000_h0.50_w2.00_NoDroplet_A.avi', '-f', 'null', 'nul']
2023-01-02 12:14:42,421 - pyffmpeg.pseudo_ffprobe.FFprobe - INFO - Inside _extract_all
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Temp\ipykernel_13272\4088098422.py", line 1, in <cell line: 1>
xxx = FFprobe('Qc06.000_Qd00.000_h0.50_w2.00_NoDroplet_A.avi')
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pyffmpeg\pseudo_ffprobe.py", line 63, in init
self.probe()
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pyffmpeg\pseudo_ffprobe.py", line 271, in probe
self._extract_all(stdout)
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pyffmpeg\pseudo_ffprobe.py", line 100, in _extract_all
all_streams = all_streams.split('Input')[1]
IndexError: list index out of range
Is there anything I can do to solve it? Thank you.
(I'm sorry i see the problem is closed but i don't understand what should i do with it, if someone could help me)


How to display another 3D model in a pop-up window

Open a new 3D model in a pop-up window while keeping the source model unchanged, and instead of opening a new port with the new ViserServer, display it directly in the pop-up window.If you could reply to me, I would feel very honored.

Hello!
Unfortunately, I don't think what you're describing is supported. If you have a specific implementation idea we could review a PR.


awk: Space around END

Description
As promised.  I've made sure there is no space around all ENDs in awk statements.

LGTM :+1:


actions: fix sync_account_from_stripe_data

mccabe complains now.

'sync_account_from_stripe_data' is too complex (11)

workaround:
diff --git a/pinax/stripe/actions/accounts.py b/pinax/stripe/actions/accounts.py
index 9d379fb..c3e1686 100644
--- a/pinax/stripe/actions/accounts.py
+++ b/pinax/stripe/actions/accounts.py
@@ -113,9 +113,8 @@ def sync_account_from_stripe_data(data, user=None):
     else:
         top_level_attrs = common_attrs

-    for a in top_level_attrs:
-        if a in data:
-            setattr(obj, a, data.get(a))
+    for a in [x for x in top_level_attrs if x in data]:
+        setattr(obj, a, data.get(a))

     # that's all we get for standard and express accounts!
     if data["type"] != "custom":

I feel like this is actually not better though.
Let's split the function, or ignore mccabe here?

I am closing this, since it conflicts with https://github.com/lock8/pinax-stripe/pull/73 by itself, and we should have a single PR including both probably?!

@blueyed I like McCabe as a guide but we don’t need to be slave to it. If it’s not obvious a way to easily make better let’s ignore.

Hard to ignore it though if the lint job fails then..
But I guess you mean to ignore it for this function then only, which makes sense.


Remove unused code

When following the instructions on flatbuffers-rs it fails to compile because of unused parameter 'code_ptr_post', and warnings are treated as errors by default. This fixes that problem.

Closed in favor of #2 (same pull-request, but from a different branch. Sorry for the noise)


As Sophie, I want show her waiting spinner while loading so that she can recognize state of loading.

GIVEN: none
WHEN: She open home page
THEN: She sees spinner while preparing.

Done


WIP(feat): Store Isolate inside API struct which can be reused

Having the library user keep track of the Isolate instance improves performance by 5x from my tests in the benches. This could be further improved by multi-threading since the user can create a new Isolate (SSREnvironment) for each thread. We need to figure out an example to provide. There's a few more performance tweaks to be had:

[ ] Reusing CachedData for script compilation
[ ] Reusing Context

Codecov Report

Merging #8 (d0cde87) into main (09a13e0) will increase coverage by 9.22%.
The diff coverage is 97.91%.

:exclamation: Current head d0cde87 differs from pull request most recent head 3078924. Consider uploading reports for the commit 3078924 to get more accurate results

@@            Coverage Diff             @@
##             main       #8      +/-   ##
==========================================
+ Coverage   89.28%   98.50%   +9.22%
==========================================
  Files           2        2
  Lines          84       67      -17
==========================================
- Hits           75       66       -9
+ Misses          9        1       -8

Impacted Files
Coverage Δ

src/ssr.rs
97.22% <97.14%> (+13.88%)
:arrow_up:

tests/lib_check.rs
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09a13e0...3078924. Read the comment docs.

As we discussed, using this in a multi-threaded environment is actually quite difficult because of the way that async runtimes (ie. tokio) handle requests. There's no guarantee that the thread that starts handling a request will follow it all the way through to response if there's a yield as another thread from the thread pool might pick it up after the yield. Each v8 HandleScope requires Entry and Exit to be called when accessed by a new thread. We need to figure out how to handle that :o

Hey @Auspicus , I'm closing this because stale.


ENH: delegate computing wheel tags to the packaging module

Use the wheel contents only to determine whether the wheel contains python ABI dependent modules or other platform dependent code.
Fixes #189, #190.

For testing purposes, it would be great to have this fix as a pip-installable package. Is it too much to ask?

pip install git+https://github.com/dnicolodi/meson-python.git@packaging should work.
You can use something like this:
[build-system]
requires = [
    "wheel",
    "meson >= 0.63.3",
    "meson-python @ git+https://github.com/dnicolodi/meson-python.git@packaging",
]
build-backend = "mesonpy"

in a pyproject.toml file.

Thanks. I forgot that pip can install from git repos...

I can confirm that with this fix I am able to install scipy 1.9.3 on an M1 macOS 13. Thanks!

something funny with pypy tags, as far as I can see from failing tests here:
AssertionError: assert 'pp38' == 'pypy38_pp73'

Indeed, although it looks like a bug in wheel. The wheel filename parsing is confused by filenames in the form purelib_and_platlib-1.0.0-pp38-pypy38_pp73-manylinux_2_31_x86_64.whl while packaging is fine with it:
>>> import packaging.utils
>>> name, version, build, tags = packaging.utils.parse_wheel_filename('purelib_and_platlib-1.0.0-pp38-pypy38_pp73-manylinux_2_31_x86_64.whl')
>>> t = next(iter(tags))
>>> t.interpreter
'pp38'
>>> t.abi
'pypy38_pp73'
>>> t.platform
'manylinux_2_31_x86_64'

Although, I am not sure what is the canonical form for pypy tags.

Ah, no. Much simpler. I inverted interpreter and ABI tags in the test. The naming is confusing... Can someone please restart the tests?

Some other tests fail because bugs in the tests, namely they look for the string any anywhere in the wheel filename and that triggers on manylinux. However, this highlights the fact that packaging uses something like manylinux_2_31_x86_64 as platform tags while wheel usually generates something that looks like linux_x86_64. Is there a standard for these platform tags? I don't see why one version or the other should be preferred, except that we are not checking against the manylinux specification, thus putting that in the platform tag may be misleading.  Should we simply use linux_$architecture?

Windows test were failing for an actual bug. Fixed. Can someone please kick the tests again?

CC @rgommers

please mark this as fixing #142 also

Thanks @rgommers The two test failures seem unrelated, but I don't have access to the CI artifacts to check what exactly went wrong. Can someone please copy the relevant log portion out?

I think these 3 Sage jobs are known failures that we need to get around to fixing. Failing tests:
  [meson_python-git]   FAILED tests/test_wheel.py::test_local_lib - subprocess.CalledProcessError: C...
  [meson_python-git]   FAILED tests/docs/examples/test_spam.py::test_build_and_import - subprocess.C...

Can you really not access GitHub Actions logs? They should be public.

On centos-stream-8-python3.9:
ERROR: spam-0.0.1-cp39-cp39-manylinux_2_28_x86_64.whl is not a supported wheel on this platform.

This seems a case of packaging not agreeing with himself about what's the correct platform tag.

The test failure on gentoo-python3.9 and opensuse-15.4 seem to be unrelated, indeed.

Can you really not access GitHub Actions logs? They should be public.

I do, but I was under the impression that the complete logs are uploaded as artifacts. What is available in the GitHub Actions logs is not very helpful to see what exactly went wrong in the case of the Gentoo job, for example.

I don't think PEP517 is intended to support cross-compilation: it is assumed that the interpreter controlling the build is the one for which the packages are being built. meson-python enforces that passing the current interpreter as the python program to build for to meson via the patform declaration file. meson in turn derives the extension modules file names by running the python interpreter. I don't think this allows for cross-compilation.

Can you really not access GitHub Actions logs? They should be public.

I do, but I was under the impression that the complete logs are uploaded as artifacts. What is available in the GitHub Actions logs is not very helpful to see what exactly went wrong in the case of the Gentoo job, for example.

Gentoo has its own issues with the new scipy: https://bugs.gentoo.org/878883 - not sure if that's the cause of the problem here, though.

I think these 3 Sage jobs are known failures that we need to get around to fixing. Failing tests:
  [meson_python-git]   FAILED tests/test_wheel.py::test_local_lib - subprocess.CalledProcessError: C...
  [meson_python-git]   FAILED tests/docs/examples/test_spam.py::test_build_and_import - subprocess.C...

How can one get more details on Sage failures? E.g. one would like to see the main Sage's config.log, etc.

Can you really not access GitHub Actions logs? They should be public.

Gentoo has its own issues with the new scipy: https://bugs.gentoo.org/878883 - not sure if that's the cause of the problem here, though.

This is a different problem. I think it is due to the fact that the version of packaging vendored by the pip available on Gentoo and the one installed in the test environment do not agree on what is the right platform tag.
Definitely, using packaging for generating the platform tag results in very specific tags. I can find a way to relax them, but I don't know how how relevant is the issue of using an old pip to install a package build with the latest release of the build tools.

I don't think PEP517 is intended to support cross-compilation: it is assumed that the interpreter controlling the build is the one for which the packages are being built. meson-python enforces that passing the current interpreter as the python program to build for to meson via --native-file. meson in turn derives the extension modules file names by running the python interpreter. I don't think this allows for cross-compilation.

I don't think there's such an assumption, rather it's not considered at all. The only thing PEP 517 does is provide a way to invoke a build system; what you do after that is completely up to you, as long as you produce an sdist or wheel.

Or, from another prospective, how would you run a cross compilation with build or pip or other interface to PEP517 backends?

Here's an example: https://github.com/scipy/scipy/blob/6912c638fc13c4eb67b09412c33557f53a821055/.github/workflows/wheels.yml#L205-L220
I'll note that the separate line
meson setup --cross-file $(pwd)/tools/wheels/cross_arm64.txt build

is a workaround for meson-python not support config_settings yet (see gh-54, gh-167). Once that lands, I'd expect to be able to use --cross-file directly. Either way, I think this PR breaks that SciPy cross-compilation setup.

I can find a way to relax them, but I don't know how how relevant is the issue of using an old pip to install a package build with the latest release of the build tools.

Unfortunately, quite relevant. If the produced wheels don't work with pip versions packaged in Linux distros, there's going to be a lot of complaints.

I can find a way to relax them, but I don't know how how relevant is the issue of using an old pip to install a package build with the latest release of the build tools.

Unfortunately, quite relevant. If the produced wheels don't work with pip versions packaged in Linux distros, there's going to be a lot of complaints.

If packaging does no agree with himself on some platforms there is little that can be done, expect dumb down the platform tags. I can work on doing this, but having to wait days to get the tests run is not an efficient way to do it...

@rgommers I amended the path to rewrite the tags to be more generic. This should restore compatibility with pip back to version 8.1.0 and auditwheel 1.0.0. Can someone please spin the tests once more?

It definitely does but there is no way to obtain correct wheel tags without being able to execute the Python interpreter for which the wheel are being built.

Well it works in main now - so I guess that's an advantage of the current approach. Maybe we need a hybrid of the two methods? Get both, and when they mismatch warn, when one method has support but the other does not, then continue. E.g. Pyston is an unknown interpreter for meson-python now, but is supporting in packaging. Cross-compilation is the other way around.

Also, how does the Meson python module even work without being able to execute the Python interpreter it is building for?

It doesn't need to execute it, only to introspect it somehow. Sane cross-compilation support is one of the major advantages of using Meson, so we should keep it in mind (and add a few tests for it in this repo at some point).

Ops, forgot to adjust the tests. At least I know that the platform tags are indeed modified as expected. @rgommers can you spin the tests once more, please?

It doesn't need to execute it, only to introspect it somehow.

It does introspect it by executing it, see https://github.com/mesonbuild/meson/blob/97ec20e90142c229f62a6d20371f44df0b8dd41e/mesonbuild/modules/python.py#L327

Pyston is an unknown interpreter for meson-python now, but is supporting in packaging. Cross-compilation is the other way around.

It is not possible to generate correct wheel tags only looking at the extension module filenames. Some required information about the interpreter is not encoded in the file names. Also the extension file names are not required to contain any information about the interpreter for which they are compiled. For example, the Meson python module does not add the ABI tag to the extension filenames for Python 3.7. With current Meson, meson-python does not work at all for Python 3.7.

Argh! Forgot the type annotations... Fixed, but I'll wait the current test run to finish before pushing the fix.

Remaining test failures are most likely unrelated to this PR: two ImportError: dynamic module does not define module export function (PyInit_example) and one with unspecified multiple test errors.

The only way I see to support cross compilation is to have a way to specify the wheel tags explicitly instead than computing them. I am not sure whether there is a way that works across build frontends to pass configuration options. As a last resort, environment variables can be used. Looking at the scipy example, this is anyway more or less what it is done already.

The only way I see to support cross compilation is to have a way to specify the wheel tags explicitly instead than computing them. I am not sure whether there is a way that works across build frontends to pass configuration options. As a last resort, environment variables can be used. Looking at the scipy example, this is anyway more or less what it is done already.

Yes, I think you are right. It's not done manually there, but it shouldn't be fully robust - we're probably getting it right a little by accident. Here's a CI log that shows:
Building wheels for collected packages: scipy
  Created temporary directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-5kel266f
  Destination directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-5kel266f
  Building wheel for scipy (pyproject.toml): started
  Running command Building wheel for scipy (pyproject.toml)
  Building wheel for scipy (pyproject.toml): finished with status 'done'
  Created wheel for scipy: filename=scipy-1.10.0.dev0-cp38-cp38-macosx_11_0_arm64.whl size=21059121 sha256=878586e93ccc537df2e101857924b860d5386df745882fd263cd5240b371266c
  Stored in directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-ephem-wheel-cache-r8_tyzmx/wheels/e4/9c/9e/cfec93e95f4b4c7f6445dfaa96cf45ba0d6be751a6c69cd040
Successfully built scipy

The build log has
...
Program python3 found: YES (/Users/runner/hostedtoolcache/Python/3.8.14/x64/bin/python)
...
Linking target scipy/_lib/_test_ccallback.cpython-38-darwin.so
...
Copying files to wheel...
...
  [3/1234] scipy/_lib/_test_ccallback.cpython-38-darwin.so

So there is no indication that Meson knows that the build is for an arm64 Python. I think things work out just by accident. I also don't see how the wheel ends up being named scipy-1.10.0.dev0-cp38-cp38-macosx_11_0_arm64.whl as there is nothing in the extension modules filenames to indicate that they are built for arm64.

I found the trick: https://github.com/scipy/scipy/blob/6912c638fc13c4eb67b09412c33557f53a821055/.github/workflows/wheels.yml#L168
>>> import sysconfig
>>> sysconfig.get_platform()
'macosx-10.15-x86_64'
>>> import os
>>> os.environ['_PYTHON_HOST_PLATFORM']="macosx-12.0-arm64"
>>> sysconfig.get_platform()
'macosx-12.0-arm64'

So even the current solution uses the python used to run meson-python and thus meson to determine the wheel tags, simply Python is tricked into reporting the "right" information.

Substantially, the introspection used to find out information is running a python script that imports a data file referenced by the environment variable _PYTHON_SYSCONFIGDATA_NAME, which is basically a very fancy python dict.
It happens to be that the python script which gets run is shipped with the python stdlib (and defaults to finding the data file bundled alongside it. The script is called "sysconfig" but I suspect at this late point in my post, you've figured that one out. ;)
The state of the art in python cross compilation is, AFAIK, mostly to set that variable.
...
The other approach would be to specify to meson that the python interpreter you're building for isn't called /path/to/cross/python but rather called qemu-user /path/to/python. That will run fine, albeit a bit of a heavyweight approach. It also doesn't fully generalize, because it assumes that qemu supports your OS. There may be a similar tool for whatever other OS one does use, however.

Thanks @eli-schwartz
It seems that setting _PYTHON_SYSCONFIGDATA_NAME would work with the approach for generating the wheel tags used here... as long as all the things the packaging module introspects to generate the wheel tags are correctly covered. I'm not sure if this is the case. At a cursory look, the introspection of the running interpreter and of the surrounding system is quite deep in some cases.

The failed tests are the known broken ones.
From what I can understand this patches do not make cross compilation any less supported than what it is now: it requires setting some undocumented environmental variables to coerce the host machine Python to lay about its identity. From what I can tell, the scipy cross compilation example @rgommers brought up would continue to work as it is with this PR merged.
This PR fixes at least three know issues (#142, #189, and #160 for good) and removes quite a bit of complexity.
I think there is nothing holding this PR back and I would like to ask for it to be merged.

I'm okay with following up on cross-compilation separately (thanks for investigating - the info in this thread will be useful!), and having this move forward. @FFY00 wrote most of the current _tags.py code, so let's wait for their review.

Thank you for the work @dnicolodi, this is great! I have some comments I'd like to see address before merging. Let me know if you agree and/or have any feedback.

Test failures seem due to changes in Meson 0.64

Hmm, that seems bad:
found pkg-config 'C:\\Strawberry\\perl\\bin\\pkg-config.BAT' but it is Strawberry Perl and thus broken. Ignoring...
Found Pkg-config: NO
Could not find Python3 library 'D:\\a\\meson-python\\meson-python\\.nox\\test-3-10\\python310.dll'
Run-time dependency python3 found: NO (tried sysconfig)

..\..\meson.build:6:0: ERROR: Dependency lookup for python3 with method 'pkgconfig' failed: Pkg-config binary for machine 1 not found. Giving up.

The Perl thing is a bit of a red herring it seems: https://github.com/mesonbuild/meson/issues/9480. So what we're left with is python3 not being found for some reason that needs investigating.

Reported as mesonbuild/meson#11017

@rgommers Fixed in #198

I'm coming around on some of the objections. I had a better look at the API provided by packaging and, assuming that all functions not explicitly marked as private are to be considered public, not only the ones documented in the manual, we can get most of the things we need without re-implementing too much. I'll give it a try later today.

I've been looking into this some more and the main piece missing without re-implementing parts of packaging into meson-python is packaging.tags.interpreter_abi() see pypa/packaging#611 Unfortunately the bit and pieces necessary to implement this simple function are private and it may be better to do not rely on them. It really seem that packaging is not meant to provide the functionality we require, but I don't know of a better match. wheel would be the best candidate, but it does not have (yet) a stable API.
I see two possibilities: either continue to abuse the packaging API like in the approach in the current patch, or bite the bullet and start re-implementing the required pieces here, with the risk of the implementations diverging because of bugs or reinterpretations of the PEPs or new platforms appearing. The first approach is not that bad, but the second would be a step into providing meson-python functionality directly from Meson (because of the Meson's "no dependencies outside the standard library" policy).
I'll go with the first approach, for now.

I think the comments have all been addressed. This is ready for another round of review.
Despite there isn't in Meson yet an easy way to generate extension modules targeting the stable Python ABI, I added some synthetic tests checking that the handling of the stable ABI tag works as intended. Unfortunately, it turns out that, as I was fearing, the stable ABI tag in the extension modules filename is not only just advisory, it is not even supported on Windows: a CPython interpreter on windows does not load extension modules with filename suffix .abi3.pyd. Therefore, the meson-python handling of the stable ABI tag is "broken by design". I decided to keep the current logic in place, but a better solution has to be found if we want to support this use case. By the way, does anyone know of any project that builds against the stable ABI?

That's correct, on Windows you are not supposed to add the .abi3 part of the extension, that's by design (why, I don't know). You also need to link against python3.dll instead of python3.X.dll when building. You can see some stable ABI users here: https://discuss.python.org/t/lets-get-rid-of-the-stable-abi-but-keep-the-limited-api/18458/11?u=henryiii

Thanks @henryiii I also found https://github.com/pypa/cibuildwheel/issues/78 with some information. In general there isn't much documentation about how this is to be used. I think the only build tools that supports this is setuptools and its documentation on this is very poor.
Clearly detecting the stable ABI from the extension modules filenames is not a solution, if it cannot be done on all platforms. Also, it seems that one of the goals is to have extension modules compatible with multiple Python versions, thus using the stable ABI wheel tag and the current Python interpreter version for the interpreter tag is not going to be enough. The only solution I see is a configuration parameter in pyproject.toml. Something like
[tool.meson]
python-api = "cp32"

As this is an existing separate issue and Meson does not have yet an easy way to produce stable ABI extensions, I would leave solving this issue to another PR.

Given the mention of cryptography on that issue, I'm guessing Maturin supports it. I've got initial support in scikit-build-core, too, though FindPython from CMake doesn't support it so I'm only testing it on Windows. Yes, you can't auto-detect it, and you need to tell the system what your minimum version is for the tagging. In scikit-build-core:
[tool.scikit-build]
wheel.py-abi = "cp32-abi3"

Would target ABI3. It can also be used to build a "Pythonless" wheel (one that doesn't use Python, like ninja, cmake, or clang-format, or just has things you load via ctypes):
[tool.scikit-build]
wheel.py-abi = "py3-none"

SKBUILD_SOABI is set to "abi3" on non-Windows if this is set to *-abi3.

I'm a bit fuzzy on the connection between the limited API and the stable ABI. I thought that using the limited API always produces a stable ABI extension module. Doesn't it?

Yes. The reason I settled on requesting both API and ABI tags was to have a way to specify binary tags that do not depend on Python at all; currently I think those are just as if not more common than limited API / stable ABI extensions. You could do just the api tag and cp32 would be limited ABI and py3 or py2.py3 or py37 would be a wheels that doesn't depend on the C api. I was a little worried about that being unclear; setting py37 or cp37 here for example means you'd support anything 3.7+, but it looks like you are locking a specific version of Python instead. Maybe it's fine? (scikit-build-core is still experimental and can be changed, if we have similar settings that would be nice)

I think #202 is a better approach in the long run.


No permissions to access default examples after install

Make sure these boxes are checked before submitting your issue - thank you!

[Y] I have checked the superset logs for python stacktraces and included it here as text if any
[Y] I have reproduced the issue with at least the latest released version of superset
[Y] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
superset-0.15.1
Expected results
Should have been granted permissions to access the example dashboards.
Actual results
Permissions were denied
Steps to reproduce
Just follow instructions on http://airbnb.io/superset/installation.html and install on a Ubuntu 16.04 machine. Note that my preference is for using Conda instead of Virtualenv, so I created a python 3.5 conda environment, and did the pip install superset in that conda environment.
The problem I'm facing is after finishing the installation and running the app server, visiting the dashboards for all of the 3 examples, I get permission denied. For example, accessing the second dataset "Births", I get the error: This endpoint requires the datasource birth_names, database or all_datasource_access permission. I'm not sure why this is the case. It appears that issue #1799 was asking about a similar problem, but nothing from that post is helping me so that's why I am posting here.
Thank you.

Could you please try latest master?

Ok I've just cloned the latest master and re-installed Superset, by using pip install -e ~/superset (I cloned superset into my home dir). So now I do not get permission errors, but somehow the dashboards are not displaying properly. Clicking on them, I get brought to a page that just shows nothing, like what I've attached here:
[dashboard] Misc Charts.pdf
At the same time, when this is happening, I get flask errors like
/sqlalchemy/sql/compiler.py:575: SAWarning: Can't resolve label reference 'username asc'; converting to text() (this warning may be suppressed after 10 occurrences)
util.ellipses_string(element.element))and/flask_appbuilder/forms.py:245: FlaskWTFDeprecationWarning: "flask_wtf.Form" has been renamed to "FlaskForm" and will be removed in 1.0.
form = self(obj=obj)`. What is happening now?

If you install from source to have to build the assets like documented here http://airbnb.io/superset/installation.html#making-your-own-build

Ah I see. From the instructions on that link, it looks like building from source would mean I would be installing Superset within my machine's global environment. Is there anyway that I can do this in a Conda environment?

You can do it within a conda environment fine. Npm will install the assets
locally fine. If you're asking about conda, I'm guessing you're on Windows
where I've found it more painless to run in either vagrant or virtual box.
On Mon, 16 Jan 2017, 15:03 Augustine Koh<EMAIL_ADDRESS>wrote:

Ah I see. From the instructions on that link, it looks like building from
source would mean I would be installing Superset within my machine's global
environment. Is there anyway that I can do this in a Conda environment?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/airbnb/superset/issues/1987#issuecomment-272885178,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEdFuETtx9Tkcj2As4dMQtV9hvcMeB2Dks5rS4awgaJpZM4LkJbt
.

--

Alan Cruickshank
E<EMAIL_ADDRESS>T: +44<PHONE_NUMBER>

@xrmx , i am running into exactly same issue, any other way to fix other than manually rebuild master?
centos 6.6, python 3.4, superset 0.18.4

I got exactly the same issue after installing it on Mac OS X.

run these:
file=airflow.cfg
cp $file $file.bk
sed -i 's#load_examples = True#load_examples = False#g' $file
bin/airflow resetdb
bin/airflow initdb
restart your airflow, you should be good without examples.
Thank you!
BR,
Alex

发件人: Wei Hai<EMAIL_ADDRESS>发送时间: 2017年6月29日 2:38
收件人: ApacheInfra/superset
抄送: bronzels; Comment
主题: Re: [ApacheInfra/superset] No permissions to access default examples after install (#1987)
I got exactly the same issue after installing it on Mac OS X.
―
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/ApacheInfra/superset/issues/1987#issuecomment-311749336, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAQMA-NKCdfrGJM0nmMPDcpJPOhC4d6vks5sIp2WgaJpZM4LkJbt.

Hello all, running into a similar frustration here:
MacOS: 10.11.6
Superset: 18.5

Some help or a point in the right direction to poke around in the source would be greatly appreciated.

Does the user you're logged in as have either of the permissions listed in
the red error message? You can find out which roles your user has, and
which permissions that role has in the security tab.
On Tue, 4 Jul 2017, 18:32 Aaron Myatt<EMAIL_ADDRESS>wrote:

Hello all, running into a similar frustration here:
MacOS: 10.11.6
Superset: 18.5
[image: screen shot 2017-07-05 at 01 28 53]
https://user-images.githubusercontent.com/11920225/27839137-98200c2a-6121-11e7-91fb-6c61a0ef064e.png
[image: screen shot 2017-07-05 at 01 29 24]
https://user-images.githubusercontent.com/11920225/27839138-98254ca8-6121-11e7-9c20-f1635a839548.png
Some help or a point in the right direction to poke around in the source
would be greatly appreciated.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ApacheInfra/superset/issues/1987#issuecomment-312923922,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEdFuHT0jl-X7yuPa3zjLJC5Wyaz6xppks5sKncPgaJpZM4LkJbt
.

--

Alan Cruickshank
E<EMAIL_ADDRESS>T: +44<PHONE_NUMBER>

@alanmcruickshank You were right, I had never needed to add explicit permissions using the default admin user in previous versions I installed. Thanks for clearing this up.

Notice: this issue has been closed because it has been inactive for 292 days. Feel free to comment and request for this issue to be reopened.

A note for future googlers:
I was having the same errors above when connecting to tables in my postgres database and the solution was to specify that i was accessing the public schema, where my tables were stored.
So, in SQL lab I put select * from public.tbl_nm and in the tables pages, I specified public as the schema.

I have faced same issue ,then ran superset init : which created all default permissions

superset init solved my problem. thank you @snehilp

How to execute superset init when using Docker?


[Full-stack Signals] make default value optional

Calling a signal endpoint requires providing initial client-side value. Coming up with initial value for complex types is inherently complex. A full value could end up as a lengthy literal:
const person = new PersonEndpoint.personSignal({defaultValue: {name: '', dob: '1970-01-01'}});

This complexity could be mitigated with some help from a generated model:
const person = new PersonEndpoint.personSignal({defaultValue: PersonModel.createEmptyValue()});

However, we could reduce the complexity even further by using the model data on the Hilla end, so that using the signals don't require default value:
const person = new PersonEndpoint.personSignal(); // initialized using the model

From where does 1970-01-01 end up as the date of birth in PersonModel.createEmptyValue()?

For nullable signals, we could use undefined as the default value instead of the model's value.


fix: bump kong idle timeout to 150s

What kind of change does this PR introduce?
Bug fix https://github.com/supabase/cli/issues/2434
What is the current behavior?
Please link any relevant issues here.
What is the new behavior?
Feel free to include screenshots if it includes visual changes.
Additional context
Add any other context or screenshots.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
7 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.02%) to 60.288%

Files with Coverage Reduction
New Missed Lines
%

internal/storage/rm/rm.go
2
89.53%

internal/gen/keys/keys.go
5
11.76%

Totals

Change from base Build<PHONE_NUMBER>:
-0.02%

Covered Lines:
6900

Relevant Lines:
11445

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.03%) to 60.285%

Files with Coverage Reduction
New Missed Lines
%

internal/gen/keys/keys.go
5
11.76%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
6899

Relevant Lines:
11444

💛  - Coveralls


Issues when Jaeger tracing is enabled for Springboot application

Hi,
I am new to Opentracing and Jaeger. I am trying to implement Jaeger to my spring boot microservices application. I am able to see traces successfully but I have some questions.
Below is the flow of the application I am working on:
Microservice A (uses SOAP webservice) calls Microservice B (uses Restful webservices) which again calls another Microservice C.
Can someone please help me understand below.

Once the server is up for Microservice A, I am seeing GET as a request type for just first trace and subsequent traces are POST, instead POST for all requests. Is there any reason as to why this happens?
Since I have added Jaeger tracing for Microservice A, I can only see traces of A making calls to B. But, in order to see B making calls to C, do I need to integrate Jaeger for Microservice B as well?
Microservice A makes a call to microservice B, both are running on my local host but on different ports. I am not able to see any traces of A making calls to B in this scenario. Can someone please help me understand why?

Hi,
For tracing to work properly, all involved microservices  must be support tracing

Hi,
For tracing to work properly, all involved microservices must be support tracing

Thank you! Will add tracing in all Micro services. Any idea on other 2 issues?

For 2, the answer is yes, you need to enable tracing.
For 3 I don't really know, I haven't tested that scenario


Visual issue with diagnostic on-hover popup when the diagnostic message is too short

Does this issue occur when all extensions are disabled?: No, technically

VS Code Version: 1.86
OS Version: Windows 10 19045

Steps to Reproduce:

Have any diagnostic/"problem" with a short message length
Hover over said problem

Description:
When a diagnostic message length is very short, the "Quick fix" and "View Problem" buttons are unexpectedly cut off.
Here's an image showcasing the issue:

The diagnostic comes from this extension, but this fact is irrelevant as the issue is with VSCode's UI as far as I can tell.

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.86.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

With the recent work, there should be an ellipsis after truncated labels in the editor hover status bar

Wouldn't a better solution to this be setting a minimum width to the hover popup, so the buttons would fit no matter what? I recognize that it may be a bit more difficult to account for localization, but still.

Hi @robotboy655 thank you for filing this issue. There is already an issue filed for this, and I will consider that.

Could you please link the mentioned issue?

Yes its here: https://github.com/microsoft/vscode/issues/235334


Can we install other Docker files?

docker.fireeye
It appears there is support for amass docker file, can we pull down others?

@opticoax747  Yes, we probably can pull other docker images as well. Please specify which image and their purposes in a separate ticket so we can track them better.


[Improve] Suggestions for the code coverage options after the v2.0.0 refactor

1) Do we need the  coverage.reportPath option ?
I know I mentioned that we should be able to choose which html report to display on the monocart test results reporter with #69.
However, given the awesome work you've done on the release v2.0.0 of the monocart-reporter package I was wondering if the coverage.reportPath option is even needed?
What I'm thinking is that you know all the possible reporters that will generate an html report, they are:

the v8 report
the istanbul reporters that produce an html report: html or html-spa (don't think there are others)

Since you know the reporters that can produce an html report then my suggestion would be to include them all by default in under the monocart report attachments global:

Other considerations:

Consider adding an option that enables selecting the display name for the html reports. Otherwise you'd have to choose some kind of default attachment name.
Consider adding an option that controls whether to attach or not an html report to the global attachments. Would default to true.

Example of a playwright.config.ts with this suggestion would look like:
// playwright.config.ts

export default defineConfig({
  ...
  reporter: [
    ...
    [
      'monocart-reporter',
      {
        name: 'monocart playwright test results report',
        ...
        coverage: {
          outputDir: './tests/test-results/code-coverage',
          // reportPath: ... // this option would be removed
          reports: [
            ['v8', {
              name: 'v8 code coverage',
              ...
            }],
            ['html', {
              name: 'istanbul html code coverage'.
              subdir: 'html-report',
            }],
            ['html-spa', {
              includeInGlobalAttachments: false,
              subdir: 'html-spa-report',
            }],
           ...
          ]
        }
      }
  ]
  ...
});

Note that in the above example the html-spa report wouldn't show under the global attachments of the monocart report because includeInGlobalAttachments is set to false but since includeInGlobalAttachments defaults to true, then the v8 and html reports would be included and named according to the name property.
Note: the names I'm suggesting are just that, suggestions, feel free to choose better names or a better structure for this issue.
2)  Consolidate the v8 report options under the coverage.reports[v8] section
What do you think about removing the coverage.outputFile and move that option inside the coverage.reports[v8] report options? That to me would make a bit more sense and keep it inline with the other reporters?
So that instead of the playwright.config.ts config shown in the comment above you would do:
// playwright.config.ts

export default defineConfig({
  testDir: './tests',
  outputDir: './tests/test-results'
  ...
  reporter: [
    ...
    [
      'monocart-reporter',
      {
        name: 'playwright code coverage with monocart reporter',
        outputFile:  './tests/test-results/monocart-report.html',
        coverage: {
          outputDir: './tests/test-results/code-coverage',
          reportPath: './tests/test-results/code-coverage/v8/index.html',
          reports: [
            ['v8', {
              file: 'v8/index.html' // Use this instead of coverage.outputFile
            }],
           ...
          ]
        }
      }
  ]
  ...
});

If you agree with that then perhaps the other v8 related options like inline and assetsPath could also be moved to options of the v8 report:
// [V8 only](Boolean) Inline all scripts to the single HTML file. Defaults to false.
inline?: boolean,

// [V8 only](String) Assets path if not inline. Defaults to "./assets"
assetsPath?: string

What are your thoughts?
3) Do we still need the coverage.lcov option?
Looking at monocart-coverage-reports/lib/index.d.ts, you've still kept the lcov option on the CoverageOptions.
// (Boolean) Generate lcov.info file, same as lcovonly report. Defaults to false.
lcov?: boolean,

Does it still make sense to keep it? With the current v2.0.0 release you can generate an lcov only report by specifying it in the reporters array as:
coverage: {
  reports: [
    ['lcovonly', {
      file: 'lcov/code-coverage.lcov.info',
    }],
  ]

Is there any reason to keep the coverage.lcov option? If there's no benefit to having this option then I think for a user it can sometimes be confusing to have competing options. And for you as a maintainer, if it's less code for you to maintain then great right?

Note that I've made other suggestions to the structure of the code coverage options in #69 on this comment.

1) Do we need the coverage.reportPath option ?
I understand your explanation and I can see how it is useful but I'm not sure I was able to explain myself properly.
My question/suggestion was that if you were to consider including all html reports by default as global attachments then you wouldn't need the coverage.reportPath option right? If you did this it would also mean users could configure a v8 report, and then a html and html-spa reporter and all 3 would be included in the global attachments instead of having to choose only one right?
That's why I showed a potential config for this where you could also then provide a name to control the display text for the global attachment and a includeInGlobalAttachments that could default to true but the users could set to false if they didn't want that html report included in the global coverage options.
Perhaps you understood this already from my initial explanation and you feel that it isn't worth the change?
2) Consolidate the v8 report options under the coverage.reports[v8] section
I didn't notice that the v8 report already had some available options. I only checked the https://github.com/cenfun/monocart-coverage-reports/blob/main/lib/index.d.ts to understand the available options and at the time it didn't have those options but you already updated it for v2.0.1 👏
I've tested the options you mentioned above and they work as expected.
I still think that the fact that you left the coverage.outputFile, coverage.inline and coverage.assetsPath is a bit confusing and for a user it's a better API if you don't have the same option in two different places. I understand that one set of options has higher priority than others but I still think removing the ones these 3 in favor of leaving them in the v8 report options would be better.
3) Do we still need the coverage.lcov option?
Right, I understand that the coverage.lcov is a shortcut. For the same reasons explained for the point number two above, I believe it would be better for the user to NOT have two very similar ways of doing the same thing. I think that just leads to confusion and questions about what's the pros/cons on using one way versus the other, when for these options were discussing, there's really no pros/cons that I can see, it's just "shortcuts".
Summary
All of these are feedback on simplifying the coverage options. I feel that having the "duplicate/shortcut" options like coverage.outputFile, coverage.inline, coverage.assetsPath and coverage.lcov hurt more than they help, specially for new users trying to learn how to use the coverage options.
I also think that the coverage.reportsPath is an improvement made to v2.0.0 but given all the other work you did to improve the coverage.reports I think it becomes a limiting option because it ONLY allows the user to select ONE html type report.

[!Note]
All of the above points are my feedback as a user. Ultimately,  if you feel it's better/ok to leave as is then it's fine, you have your vision for how things should work and nothing is broken so it's fine if you don't think it should change. Feel free to close this if you don't think it's worth doing any changes.


Investigate high rate of replay upload timeouts

https://sentry.ppy.sh/organizations/ppy/issues/34904/

@bdach to follow up on this, we figured that the number being reported on datadog was much higher than actual, but I forget the reason why, or how i confirmed that this was the case. or if we actually decided this was still an issue..
root@db-master.osu.io:osu> select has_replay, count(*) from scores where preserve = 1 and passed = 1 and legacy_score_id is null and id between<PHONE_NUMBER> and 238700288
                        -> 0 group by has_replay;
+------------+----------+
| has_replay | count(*) |
+------------+----------+
| 1          | 22345    |
| 0          | 3669     |
+------------+----------+

That query is missing an approved in (1, 2).
Running the query did look significantly better but we never got to explain why the numbers were being overestimated like that. I believe we did rule out aborted submission.
Next step would be to take a sample of those timeout messages from logs or from sentry and look for possible patterns in database. Or just blindly try increasing timeout I guess.

That query is missing an approved in (1, 2).

aha, that's the one.
root@db-master.osu.io:osu> select has_replay, count(*) from scores join osu_beatmaps using (beatmap_id) where preserve = 1 and passed = 1 and legacy_score_id is null an
                        -> d id between<PHONE_NUMBER> and<PHONE_NUMBER> and approved in (1,2) group by has_replay;
+------------+----------+
| has_replay | count(*) |
+------------+----------+
| 1          | 21148    |
| 0          | 36       |
+------------+----------+

Let's close this one for the time being. As we discovered, the failure rate is actually very low.

I'd still want to investigate the metrics themselves and why they're overreporting the actual count. Maybe we can save on some unnecessary work in the server.


allow passing in keydir

What did you do?
Ejson allows a --keydir option to point to where the private key can be located for decrypting.
This PR allows setting the keydir for deploying through capistrano.
Why?
Private keys may be stored in a non-default location on the deploying machine.

@thegedge all yours!

I have never published a gem for this repo, so I'm not sure if I'll be able to. I'll figure that out tomorrow, @Drew-Goddyn. Wish me luck! 😅

Took a little longer than expected, but 1.1.0 is now released 🚀


Support String.formatted with FormatString Bug Pattern

Added in JDK 15 is the ability for callformatted on a String.
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html#formatted(java.lang.Object...)
"foo %s".formatted("bar");
It would be great if the https://errorprone.info/bugpattern/FormatString check was enforced on this.

SGTM, I'd be happy to take a PR for this
For the test coverage, we'll need something like the following to exclude the tests on Java < 15:
assumeTrue(RuntimeVersion.isAtLeast15());

@cushon any guidance on how you would fit that in with the current checks? FormatString seems pretty tied to it being a method  like .format(String pattern, Object... args) where this is on an instance of a String like"foo%s".formatted(Object... args)
Or would you create an entirely new check?

I think it would be possible to adapt the existing FormatString check.
It uses this method to get the list of arguments to e.g. String.format, and that list of expressions is passed through to FormatStringValidation.validate:
https://github.com/google/error-prone/blob/65eb009d1c7170ccc961471963def163f0c8ab4c/core/src/main/java/com/google/errorprone/bugpatterns/formatstring/FormatStringUtils.java#L55
Perhaps formatMethodArguments could be updated to recognize e.g. foo.format("bar", "baz"), and then convert that to the list of expressions starting with the LHS, and continuing with the arguments: [foo, "bar", "baz"]

@cushon Unfortunately I'm not able to figure out how to get Intellij to execute a unit test for debugging with everything on the classpath. Anyone have any hints or steps to make that happen?

@codylerum not sure what error you get, but if IntelliJ complains about duplicated classes on the classpath, then one workaround is to edit the run/debug configuration and click Modify Options -> Add before launch task -> Build, no error check, then drop the original Build step.
(You might also see an error about Stream#takeWhile being unknown. For that one raise the source level to 11.)

@Stephan202
What I get is
"C:\Program Files\Eclipse Foundation\jdk-<IP_ADDRESS>-hotspot\bin\java.exe" -agentlib:jdwp=transport=dt_socket,address=<IP_ADDRESS>:51014,suspend=y,server=n -ea -Xmx1g --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED -Didea.test.cyclic.buffer.size=1048576 -javaagent:C:\Users\cody.lerum\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-1\213.6461.79\plugins\java\lib\rt\debugger-agent.jar -Dfile.encoding=UTF-8 @C:\Users\cody.lerum\AppData\Local\Temp\idea_arg_file644136494 com.intellij.rt.junit.JUnitStarter -ideVersion5 -junit4 com.google.errorprone.bugpatterns.formatstring.FormatStringTest,stringFormatted
Connected to the target VM, address: '<IP_ADDRESS>:51014', transport: 'socket'

java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'

	at com.google.errorprone.scanner.ScannerSupplier.defaultSeverities(ScannerSupplier.java:64)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerInfos(ScannerSupplier.java:85)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerClasses(ScannerSupplier.java:74)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerClasses(ScannerSupplier.java:55)
	at com.google.errorprone.CompilationTestHelper.newInstance(CompilationTestHelper.java:116)
	at com.google.errorprone.bugpatterns.formatstring.FormatStringTest.<init>(FormatStringTest.java:32)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:250)
	at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:260)
	at org.junit.runners.BlockJUnit4ClassRunner$2.runReflectiveCall(BlockJUnit4ClassRunner.java:309)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

which also shows as a compile error in intellij for ScannerSupplier on line 64

@codylerum looks like the classpath somehow contains an older version of Guava. Does your checkout contain any modifications that could cause an older Guava dependency to be pulled in? (The buildOrThrow method was introduced in version 31.0, while Error Prone currently uses 31.0.1. So this should just work.)
(This issue reminds me of https://github.com/google/error-prone/issues/2138#issuecomment-766314550 and https://github.com/google/error-prone/issues/2745#issuecomment-991583157, but there the issue was with using Error Prone, while you're trying to build Error Prone itself. :thinking:)

@Stephan202 Thanks for the pointers I went through and made intellij invalidate all it's caches and it seems to be compiling now.
Intellij seems to frequently have issues when a dependency version changes and it doesn't blow away the old one out of the build.

I made some progress, but I'm out of my wheelhouse when it comes to walking around the complier here.
At line 116 I'm into the validator and I have my two expected arguments for my pattern of %s %s".formatted("foo", "baz");
https://github.com/codylerum/error-prone/blob/string_formatted/core/src/main/java/com/google/errorprone/bugpatterns/formatstring/FormatStringValidation.java#L116
However I'm lost at how to get the value of the string which is the "owner" of this method. I'm guessing this is a solved problem somewhere else.

@codylerum I won't have time for an in-dept analysis, but the formatMethodSymbol available in that method lacks the relevant information: it's the symbol associated with the String#formatted method, not linked to a particular string instance.
In the context of that specific method, when running FormatStringTest.stringFormatted, we have that state.getPath().getLeaf() points to the associated MethodInvocationTree, so the "%s %s" string can be retrieved as follows:
ASTHelpers.getReceiver((ExpressionTree) state.getPath().getLeaf())

However, this approach is very fragile (certainly that cast should be guarded, and there are likely other issues with it) and validate's signature/contract should likely be updated instead. Unfortunately I don't have more time to look into this now. But hopefully this helps somewhat :)

Thanks @Stephan202 and @cushon
I was able to work up a PR that implements this which is now linked. Let me know if that direction works or if you want me to look at another way


Add @Flag initializer for Optional

Per discussion, add an initializer for Flag<Optional<Bool>> that
allows the default value to be nil, enabling capacity to distinguish
the lack of a supplied flag vs the default value from user.

@dduan This looks good! For historical purposes, could you fill out this template, either as the PR intro or as a new comment?
https://raw.githubusercontent.com/apple/swift-argument-parser/master/.github/PULL_REQUEST_TEMPLATE/NEW.md
Thanks!

@natecook1000 done. Let me know if you want me to update documentation somewhere.

thanks for the discussion and reviews @natecook1000!


Malware scan execution error in VMR

The 1ES PT: AntiMalware ClamD Linux Scanner (Binary) step from the 1ES templates shows Pass but with the following error in latest build:
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.000 sec (0 m 0 s)
Start Date: 2024:07:25 10:33:56
End Date:   2024:07:25 10:33:56
ERROR: Could not connect to clamd on LocalSocket /var/run/clamav/clamd.ctl: No such file or directory
##[error]1ES PT Warning: Malware scan execution error in '/mnt/vss/_work/1/a/BuildLogs' with exit code '2'. This could be because the 'freshclam' could not update the virus databases.

From an internal discussion with the 1ES folks.
"This is a known issue. Unfortunately we cannot influence clamd behavior as it is a third party tool. Because of the amount of requests to clamd database it gets throttled. We are actively working on replacing it with Defender for Linux."
Given this isn't actually failing our builds I am going to close this issue as there is no action for us to take.


Rename repository to opentelemetry-website

Currently the link on the organization page (for all projects) doesn't work. It points to probably old location http://opentelemetry.dev .
The new fixed one is https://opentelemetry.netlify.com/ I assume.
In addition this project could be renamed. Maybe some domain neutral name would be good, such as opentelemetry-website.

@vitaliis What's the rationale behind using a repo name like *-website?

To keep it domain-independent (read abstract enough).
And by the way it is fixed by now... (both the repo name and the link on the org page)
At least for now. My intention was to keep it abstract enough so that new domain name doesn't break this repo name and therefore all the links pointing here.


frequent multiple error messages in Homebridge Log

Some weeks ago I updated my entire Raspberry / Homebridge environment. So NPM. node.js, Homebridge and every PlugIn is uptodate (LTS) now. With this I also updated the camera-ui Plugin (I used a very old version before). I have another Raspberry camera and used this plugin to "push" the stream to HomeKit. Worked great. After the Update all configurations were perfectly migrated and the stream running fine in Home.App
If I am "entering" the room in Home.App where the camera is located in HomeKit I got a refreshed Snapshot (fine) but I can see a lot of error messages in Homebridge.
This is happening as long as I am on that dedicated room in Home.App. After switching to another room or exiting Home.App the error messages stopp.
The config looks as follows:
{
            "name": "CameraUI",
            "port": 8282,
            "theme": "auto",
            "language": "de",
            "debug": false,
            "mqtt": {
                "active": false
            },
            "http": {
                "active": false,
                "localhttp": false
            },
            "ssl": {
                "active": false
            },
            "cameras": [
                {
                    "name": "Emmi",
                    "videoConfig": {
                        "source": "-i rtsp://192.168.X.XX:8554/",
                        "maxWidth": 960,
                        "maxHeight": 540,
                        "maxFPS": 20,
                        "vcodec": "copy"
                    }
                }
            ],
            "platform": "CameraUI"

Any tips or hints to fix it? Or is it a bug?
Thx for help.

Can u try without "vcodec": "copy" pls

The same behavior even without the vcodec-statement.

@WillyWurst
can u try following camera source pls (without vcodec:copy)
"source": "-rtsp_transport tcp -i rtsp://192.168.X.XX:8554/"
if this doesnt work, try this
"source": "-rtsp_transport udp -i rtsp://192.168.X.XX:8554/"
(replace with your camera address)

Hello @SeydX,
thank you very much for your effort and support!
With

"source": "-rtsp_transport tcp -i rtsp://192.168.X.XX:8554/"

neither the snapshot nor the stream is working. Homebridge is throwing the following error messages:
[22.9.2021, 13:20:33] [CameraUI] [Plugin] Emmi: Starting video stream: 960x540, 20 fps, 299 kbps
[22.9.2021, 13:20:33] [CameraUI] [Plugin] Emmi: [rtsp @ 0x372b700] [error] method SETUP failed: 461 Client error
[22.9.2021, 13:20:33] [CameraUI] [Plugin] Emmi: [error] rtsp://192.168.X.XX:8554/: Protocol not supported
[22.9.2021, 13:20:33] [CameraUI] [Plugin] Emmi: ffmpeg exited with code: 1 and signal: null (Error)
[22.9.2021, 13:20:33] [CameraUI] [Plugin] Emmi: Stopped video stream.
[22.9.2021, 13:20:36] [CameraUI] [Plugin] Emmi] [Snapshot: [rtsp @ 0x34ee270] method SETUP failed: 461 Client error
[22.9.2021, 13:20:36] [CameraUI] [Plugin] Emmi] [Snapshot: rtsp://192.168.X.XX:8554/: Protocol not supported
[22.9.2021, 13:20:36] [CameraUI] [Plugin] Emmi: Failed to fetch snapshot.

Maybe it helps to share how I initiated the stream on the other Raspberry PI (it's a Zero W with the raspberry camera module v2):  raspivid -o - -t 0 -n -w 960 -h 540 -fps 30 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264 :h264-fps=30 > streamoutput.txt 2>&1 & exit

Can u try v5 pls
sudo npm i -g<EMAIL_ADDRESS>

Hej @seydx,
sorry for my late response but needed to update node-js version to test with v5.
I installed v 5.0.16 and the "problem" is solved. No error messages like described above.
Thank you!!!
Great support 🥇

Same issue here. But I am pretty much sure that with the current version (5.0.16) with this plugin it was running before. Maybe an issue with iOS 15.4 update????


Basic support for non-integral const (e.g. bitvector)

This implements STEP 1 of #935 (towards fixing #932)

hmm @ranjitjhala, this still seems to trip up on annotations like this:
pub const START: BV32 = BV32::new(0xFFFF_FFFF);

#[sig(fn (BV32[@addr]) -> bool[addr == START])]
pub fn test_start_eq(address: BV32) -> bool {
    address == START
}

error[E0999]: refinement type error
   |
   |     address == START
   |                ^^^^^^^^^ a postcondition cannot be proved
   |
note: this is the condition that cannot be proved
   |
   | #[sig(fn (BV32[@addr]) -> bool[addr == START])]
   |                                ^^^^^^^^^^^^^

Looks like this new test generates mir like
fn test2(_1: BV32) -> bool {
    debug address => _1;
    let mut _0: bool;
    let mut _2: &BV32;
    let mut _3: &BV32;
    let _4: BV32;
    let mut _5: &BV32;

    bb0: {
        StorageLive(_2);
        _2 = &_1;
        StorageLive(_3);
        _5 = const test2::promoted[0];
        _3 = &(*_5);
        _0 = <BV32 as std::cmp::PartialEq>::eq(move _2, move _3) -> [return: bb1, unwind: bb2];
    }

which maybe means this has to do with that promotion business @nilehmann UGH.
https://github.com/flux-rs/flux/issues/886
Maybe this last bit will have to wait for STEP 3.

Oh yeah, annoying. == goes through  PartialEq so address == START  is equivalent to PartialEq::eq(&address, &START) which causes &START to be const promoted. The workaround without implementing support for const promotion is to write it as
#[sig(fn (BV32[@addr]) -> bool[addr == START])]
pub fn test_start_eq(address: BV32) -> bool {
    let s = START;
    address == s
}

@nilehmann's tmp fix works... it was pretty annoying to do but it definitely unblocked me - checking annotations that used to hang seemingly infinitely now check in a few seconds!

Wow. Surprising that

That’s why the promotion happens AND
One can work around it with a local AND
That’s it mitigates the bitvec issues!!!

Ranjit.

On Thu, Dec 12, 2024 at 6:26 PM Vivien Rindisbacher <
@.***> wrote:

@nilehmann
https://urldefense.com/v3/__https://github.com/nilehmann__;!!Mih3wA!DxMpSVRcImSGRWdG7kiDkrA7XxAy3jnkit26L-EbqxEEnCzwzahbrU2ohXh3Dus7bhpwOUUYGGM3CSzElzC9BBPX$'s
tmp fix works... it was pretty annoying to do but it definitely unblocked
me - checking annotations that used to hang seemingly infinitely now check
in a few seconds!
—
Reply to this email directly, view it on GitHub
https://urldefense.com/v3/__https://github.com/flux-rs/flux/pull/941*issuecomment-2540397304__;Iw!!Mih3wA!DxMpSVRcImSGRWdG7kiDkrA7XxAy3jnkit26L-EbqxEEnCzwzahbrU2ohXh3Dus7bhpwOUUYGGM3CSzEl0Sm6Hlx$,
or unsubscribe
https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAMS4OHLCJQEGA2QGGK7UQL2FJAW3AVCNFSM6AAAAABTO6J5M6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBQGM4TOMZQGQ__;!!Mih3wA!DxMpSVRcImSGRWdG7kiDkrA7XxAy3jnkit26L-EbqxEEnCzwzahbrU2ohXh3Dus7bhpwOUUYGGM3CSzEl5_Kw5iK$
.
You are receiving this because you were mentioned.Message ID:
@.***>

I may have been over-optimistic. It definitely helped a lot with some functions but there are still some others that take forever. I'll open an issue for these.

Opened an issue here: #944. I don't think it's bit vector related though - it seems like this helped a ton. For example ldmia_w would also take forever to check until I converted everything to bitvectors.

Ok I think this is ready to go. I'm not happy with the "HACK" which is a very special case where the const has generic args, but maybe that's a fight for another day.

I probably going to take a look at this tomorrow

Ok I think I did the lot, except for

more explicit about what stuff we support

ok I think i took care of those changes, pls see!


RC README Individual Train and Eval Commands

aferritto created the following on Apr 26:
Original issue: https://github.ibm.com/ai-foundation/PrimeQA/pull/157

PR merged.


Modal dialog trigger does not work on touch events until portal is unmounted

Current Behavior
When a modal dialog is closed, there is a period of about 2-3 seconds (the time it takes to completely unmount the portal) where the the dialog trigger does not work and does not re-open the modal if it is pressed on a touchable device. It works for mouse clicks but not touch events.
This only applies to modal dialogs and not sheets.
Expected Behavior
No response
Tamagui Version
1.15.17

Reproduction
https://tamagui.dev/docs/components/dialog
https://user-images.githubusercontent.com/1728215/233246308-c6b12fcf-677e-4c5e-97f2-04fa20955fbd.mp4
System Info
No response

This has gotten worse now. Every time the trigger is pressed, the "timeout" time for the portal to be unmounted is reset, so if you keep clicking the dialog never opens. You have to wait a couple of seconds before pressing for it open.

I know this isn't the ideal answer, but we've started just switching everything over to use state for open/close instead of the triggers.

I am experiencing something very similar which I described in another issue here. In my use case, I have "nested" dialogs and pressing somewhere after closing the second one, would close the first one.
I am also using open/close state for all of those dialogs, but it doesn't improve much.
It got a bit better with v1.27.1, but still a bit buggy.

I wasn't able to reproduce it anymore, seems already resolved.
if you still have the issue let me know to re-open the ticket


New post: large numbers in SKI and Xi function records

Using new methods, I named large numbers in SKI and set greatly improved lower bounds on the Xi0 function.
By nature of the post, there isn't much to review in terms of content, though please do comment on anything notable or which needs changing.
I can't expect my reviewers to all understand ordinals very well, but please at least check that the combinator counts are correct. I did my best to ensure all information is accurate, and I will be doing some recounts myself, but still I am just a little worried I've made a mistake around the w^w level due to using ski_tools.py before the parser bug was fixed.
Also, please play around with ski_tools.py and ensure there's no obvious bugs.

Actually I should be more worried that I slipped up in the process of translating some of the larger functions to SKI, especially for the Goodstein sequence ones, where a mistake is likely to go unnoticed. The script does its best, and I don't think the bug actually affected me since I worked only in normal form (minimal brackets) as I recall.


Javadoc Fixes

I've fixed some issues in Javadoc that were preventing JCS from compiling on my machine. Also a change to allow a different error code for SQL table already exists errors.

Had a quick look and changes look OK. I also had issues building with Java 8 (though looking at BUILDING.txt looks like it requires Java 7?).
But before anything, would be nice if we could fix the conflicting files. Could you have a look, please @mjaggard?
Cheers
Bruno

The PR was auto-closed because trunk was deleted; please re-open against master if it still applies


[opentelemetry-cpp] Add missing features #21124

Add missing opentelemetry-cpp features otlp and zpages.
Fixes #21124
For the next release update, the vcpkg maintainer should update the OTEL_PROTO_VERSION variable and the SHA512 signature for the opentelemetry-proto submodule dependency

@juandemanjon ,Thanks for your pr, I tested the features in the x64-linux triplet but failed, this is the error log:
CMake Warning at /home/test1/vcpkg/scripts/buildsystems/vcpkg.cmake:788 (_find_package):
  By not providing "FindgRPC.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "gRPC", but
  CMake did not find one.

  Could not find a package configuration file provided by "gRPC" with any of
  the following names:

    gRPCConfig.cmake
    grpc-config.cmake

  Add the installation prefix of "gRPC" to CMAKE_PREFIX_PATH or set
  "gRPC_DIR" to a directory containing one of the above files.  If "gRPC"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  CMakeLists.txt:305 (find_package)

CMake Error at /home/test1/vcpkg/downloads/tools/cmake-3.21.1-linux/cmake-3.21.1-linux-x86_64/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR)
Call Stack (most recent call first):
  /home/test1/vcpkg/downloads/tools/cmake-3.21.1-linux/cmake-3.21.1-linux-x86_64/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /home/test1/vcpkg/downloads/tools/cmake-3.21.1-linux/cmake-3.21.1-linux-x86_64/share/cmake-3.21/Modules/FindProtobuf.cmake:650 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  /home/test1/vcpkg/scripts/buildsystems/vcpkg.cmake:788 (_find_package)
  CMakeLists.txt:319 (find_package)

@JonLiu1993 I have added grpc and protobuf dependencies to OTLP feature in the vcpkg.json

@JonLiu1993  Now both x64_linux and x64_osx succeeded but windows is failing. I think it is not related to the port. Please would you have a look?

Except for ‘etw’, all features have been successfully tested under the 64-linux triplet

@JonLiu1993 Thank you for pushing a new port-version


Support missing data

See issue #2, which is a prerequisite.  The algorithm is described in the same paper, Friedman, 1991, "Estimating functions of mixed ordinal and categorical variables using adaptive splines".

It is actually possible to do this without implementing full support for categorical variables.  I am working on it now.

Just added a pull request for this: #88.  Still needs to be benchmarked, as I'm worried speed was affected.


Ldanzinger/raster function fix

adding additional null checks, adding the offline table to the doc, and renaming duplicate variable name in QML.
@nandinirao  can you please review/merge
cc @bsnider

I've added some checks to make sure the file exists @nandinirao


Support node labels

Firstly, thanks a lot for the nice project! Is there any plan to support custom label or annotations for servers, since it's very useful for separating the deployments between nodes.

Does this need to happen during deployment? I add additional labels and taints aftrewards.

Hi @sananguliyev, the tool doesn't add any labels to the nodes. What kind of labels where you thinking?

Thanks for your answer. Let's assume that I am using the same type of servers for 2 different purposes.

API gateway
SQL Engine

It's very useful to add labels to nodes so when I deploy my gateway I can specify in my k8s deployment that run this app only on the nodes which has labelled api-gateway.

I understand. However I think this goes beyond the scope of the tool which is creating clusters with a common configuration. What you are asking is for specific use cases. Would you want to add the same labels on all the nodes in a pool or how?

Yes, exactly. I would like to have those labels on all nodes in the pool, otherwise it would be too specific.
Desired configuration:
worker_node_pools:
- name: sql-engine-small
  instance_type: cpx21
  instance_count: 3
  labels:
    purpose: sql-engine
    size: small

Ok, I'll see what I can do.

Perfect. Thanks in advance.

@vitobotta if you add this you can also go ahead and add the same for taints, i.e.
worker_node_pools:
- name: sql-engine-small
  instance_type: cpx21
  instance_count: 3
  labels:
    purpose: sql-engine
    size: small
  taints:
    key: value

Ok

Implemented in v0.6.2. Please give it a try and let me know how it goes.


fix(sec): encode URI when DOM text is reinterpreted as HTML

Description
Fixing DOM text reinterpreted as HTML alert
Issue(s)
export const hideAndClickDownloadLink = (url: string, a: HTMLAnchorElement) => {
  a.style.display = "none";
  a.href = URL;

is reinterpreted as HTML without escaping meta-characters.
Fixes
DOM text reinterpreted as HTML
How to test

Go to https://deploy-preview-2593--oak-web-application.netlify.thenational.academy
Click on /teachers/curriculum/
You should be able to download curriculum plans
CodeQL should remove errors when scanning

:tada: This PR is included in version 1.477.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Missing Arundina Sans font

Arundina Sans is supplied on Ubuntu 14.04 by the fonts-sipa-arundina package.

font-family-reunion only includes default operating system fonts, not those installed after-market with other packages. Please reopen if you think Arundina Sans fits this mold.


Failed to Fetch https://azure.archive.ubuntu.com Not Found

Description
We are facing currently issues with installing curl package from the azure.archive.ubuntu.com repository resulting in a fail for all of our build jobs. The package itself can be found but it seems that the installation files are missing.
$ sudo apt-get install curl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libcurl4
The following packages will be upgraded:
  curl libcurl4
2 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
Need to get 484 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Ign:1 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 curl amd64 7.81.0-1ubuntu1.17
Ign:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcurl4 amd64 7.81.0-1ubuntu1.17
Err:1 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 curl amd64 7.81.0-1ubuntu1.17
  404  Not Found [IP: <IP_ADDRESS> 80]
Err:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcurl4 amd64 7.81.0-1ubuntu1.17
  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/c/curl/curl_7.81.0-1ubuntu1.17_amd64.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/c/curl/libcurl4_7.81.0-1ubuntu1.17_amd64.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[X] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[ ] macOS 14 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
20240811.1.0
Is it regression?
no
Expected behavior
Install the packages using the azure.archive.ubuntu repository
Actual behavior
Packages can be found but the installation package doesn't seem to be available.
Repro steps
Use sudo apt-get install curl
or curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

@FunLow -  Thank you for bringing this issue to our attention. We will look into this issue and will update you on this issue after investigating.

This is also happening for libkrb5-3 and its dependencies on jammy

Ditto for our self-hosted build agents installing packages, e.g. libpython3.10
working yesterday:
2024-08-14T11:31:42.2010323Z Get:56 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10-*** amd64 3.10.12-1~22.04.5 [4762 kB]
...
2024-08-14T11:31:44.5494110Z Preparing to unpack .../00-libpython3.10_3.10.12-1~22.04.5_amd64.deb ...
2024-08-14T11:31:44.6206333Z Unpacking libpython3.10:amd64 (3.10.12-1~22.04.5) over (3.10.12-1~22.04.3) ...

failing today
2024-08-15T13:32:57.3480473Z Get:56 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10-*** amd64 3.10.12-1~22.04.5 [4762 kB]
...
2024-08-15T13:32:57.5939383Z Err:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3.10 amd64 3.10.12-1~22.04.5
2024-08-15T13:32:57.5940224Z   404  Not Found [IP: <IP_ADDRESS> 80]

The source is working again today. Hurrah!

404's still , seems the repo is broken at http://azure.archive.ubuntu.com/ubuntu
seems a loop when trying to navigate into the ubuntu folder as per:

Can confirm that the issue has also been fixed for us.
@hemanthmanga
I tried Solution 1 and that one didn't worked for us.
Solution 2 may have been possible but didn't tried that.
Solution 3 would have been quite hard as the identification would require that we test each package and also have to evaluate if the relevant packages are compatible. Most of the packages installed were coming from some other packages requiring these as dependencies.
As we are using an automated setup for dependencies installed on short living Agents manual installation wasn't an option.
Still thank you for your suggestions.

Hi @FunLow,
Thank you for your update and detailed feedback.
I understand that Solution 1, which involved using --fix-missing, didn’t resolve the issue, and Solution 2, switching mirrors, was considered but not applied. Given your automated setup and the short-lived nature of your agents, it’s clear that manual installations were not feasible.
As an alternative solution, you might consider using a different package manager like snap to install curl. This can serve as a temporary workaround:
Install curl via snap:
sudo snap install curl
This approach can bypass issues with the package repository and may offer a quick fix.
If you encounter any further issues or have additional questions, please don’t hesitate to reach out.

@FunLow this is not a solution, the solution is to fix the mirror, not paste some chatgpt response.

Hi @FunLow, Since we haven't heard back, we'll assume your issue is resolved and will close this issue for now. Feel free to reach out us for any other queries. Thanks.

This isn't fixed and unsurprisingly the chatGPT bot @hemanthmanga isn't too much help