[Deutsche Welle] An extractor error has occurred. (caused by KeyError('media_title'))

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.09.01 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
Germany
Provide a description that is worded well enough to be understood
I'm trying to download this Deutsche Welle podcast (only audio!) https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677
with ArchLinux youtube-dlp (up-to-date): It fails with the KeyError('media_title') error.
This URL doesn't contain characters which need escaping.
The error seems to be the same like in the otherwise unrelated issue https://github.com/yt-dlp/yt-dlp/issues/2606
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[papa@main podcast]$ youtube-dl -vU https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677
[debug] Command-line config: ['-vU', 'https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6]
[debug] Python 3.10.7 (CPython 64bit) - Linux-5.19.8-arch1-1-x86_64-with-glibc2.36 (glibc 2.36)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-107098-g4d45f5acbd (setts), ffprobe N-107098-g4d45f5acbd, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.12.0, brotlicffi-<IP_ADDRESS>, certifi-2022.09.14, mutagen-1.45.1, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [dw] Extracting URL: https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677
[dw] 63153677: Downloading webpage
ERROR: 63153677: An extractor error has occurred. (caused by KeyError('media_title')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 670, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/dw.py", line 53, in _real_extract
    title = hidden_inputs['media_title']
KeyError: 'media_title'

Please provide a new example url, this one has expired (unless it's geo-blocked)
NOTE: the expired url is not what caused this issue, because yt-dlp correctly detects the 410 error
[debug] Command-line config: ['https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677', '--verbose']
[debug] User config "C:\Users\jaybu\AppData\Roaming\yt-dlp\config.txt": ['--ffmpeg-location', 'C:\\Users\\jaybu\\ffmpeg\\bin', '-P', 'C:\\Users\\jaybu\\youtube.dl', '--update', '--audio-quality', '0']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffmpeg" -bsfs
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffprobe" -bsfs
[debug] exe versions: ffmpeg N-106498-g854615adf2-20220405 (setts), ffprobe N-106498-g854615adf2-20220405, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [dw] Extracting URL: https://www.dw.com/de/wenn-der-strom-aus-der-luft-kommt/av-63153677
[dw] 63153677: Downloading webpage
ERROR: [dw] 63153677: Unable to download webpage: HTTP Error 410: Gone (caused by <HTTPError 410: 'Gone'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 670, in extract
  File "yt_dlp\extractor\dw.py", line 51, in _real_extract
  File "yt_dlp\extractor\common.py", line 1079, in _download_webpage
  File "yt_dlp\extractor\common.py", line 1030, in download_content
  File "yt_dlp\extractor\common.py", line 864, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 821, in _request_webpage

  File "yt_dlp\extractor\common.py", line 803, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 3656, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 410: Gone

❯ yt-dlp -v https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733#spark_wn=1
[debug] Command-line config: ['--ignore-config', '-v', 'https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733#spark_wn=1']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d656] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: dab284f80
[debug] Python 3.10.7 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-107787-gc469c3c3b1-20220814 (fdk,setts), ffprobe N-107787-gc469c3c3b1-20220814, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] Loaded 1673 extractors
[debug] [dw] Extracting URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733#spark_wn=1
[dw] 63121733: Downloading webpage
ERROR: 63121733: An extractor error has occurred. (caused by KeyError('media_title')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "D:\Programs\Source\yt-dlp\yt-dlp\yt_dlp\extractor\common.py", line 672, in extract
    ie_result = self._real_extract(url)
  File "D:\Programs\Source\yt-dlp\yt-dlp\yt_dlp\extractor\dw.py", line 53, in _real_extract
    title = hidden_inputs['media_title']
KeyError: 'media_title'

In yt-dlp and yt-dl, --force-generic captures this HTML5 source.

In yt-dlp (...), --force-generic captures this HTML5 source.

Indeed!
yt-dlp -v -F "https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733" --force-generic =>

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733', '--force-generic']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2<IP_ADDRESS>9 [a1f3c39] (win32_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] Checking exe version: "..\ffmpeg" -bsfs
[debug] Checking exe version: "..\ffprobe" -bsfs
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.14, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1672 extractors
[debug] [generic] Extracting URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733
[generic] av-63121733: Downloading webpage
WARNING: [generic] Forcing generic information extractor
[generic] av-63121733: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
[html5] av-63121733: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Identified a html5 embed
[info] Available formats for av-63121733-1:
ID       EXT RESOLUTION FPS |   TBR PROTO | VCODEC        VBR ACODEC    ABR
---------------------------------------------------------------------------
hls-386  mp4 480x270     25 |  386k m3u8  | avc1.42c015  386k mp4a.40.2  0k
hls-631  mp4 512x288     25 |  631k m3u8  | avc1.4d4015  631k mp4a.40.2  0k
hls-1137 mp4 640x360     25 | 1138k m3u8  | avc1.4d401e 1138k mp4a.40.2  0k
hls-2632 mp4 960x540     25 | 2632k m3u8  | avc1.4d401f 2632k mp4a.40.2  0k
hls-5064 mp4 1280x720    25 | 5065k m3u8  | avc1.64001f 5065k mp4a.40.2  0k
hls-6500 mp4 1920x1080   25 | 6500k m3u8  | avc1.640028 6500k mp4a.40.2  0k

This doesn't mean, of course, that the dwIE doesn't require fixing... 😉
If I understood things correctly,  --force-generic has been deprecated in yt-dlp in favour of  --ies generic; but using that instead does not yield the same result as the deprecated flag...

yt-dlp -v -F "https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733" --ies generic =>
[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733', '--ies', 'generic']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2<IP_ADDRESS>9 [a1f3c39] (win32_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] Checking exe version: "..\ffmpeg" -bsfs
[debug] Checking exe version: "..\ffprobe" -bsfs
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.14, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1 extractors
[debug] [generic] Extracting URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733
[generic] av-63121733: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] av-63121733: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
ERROR: Unsupported URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1477, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1553, in __extract_info
  File "yt_dlp\extractor\common.py", line 672, in extract
  File "yt_dlp\extractor\generic.py", line 3089, in _real_extract
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733

A second bug discovered inadvertently here?

If I understood things correctly,  --force-generic has been deprecated in yt-dlp in favour of  --ies generic; but using that instead does not yield the same result as the deprecated flag...

https://github.com/yt-dlp/yt-dlp/blob/19b4e59a1e1bf368078f90e7f735fa4576f97b64/README.md#L2010

To add some clarification that I personally needed, --ies generic doesn't enable the HTML5 extractor, and the extraction fails. Equally neither --ies html5,default nor just --ies html5 work because the yt-dlp HTML5 extractor depends on the generic extractor.

 --ie generic restricts yt-dlp to use ONLY generic extractor while  --ie generic,default tells it to prioritize generic extractor

... Well, we already know the default extractor (=dw) fails, due to the filed bug...
... Using standalone generic one also fails, as proved by the log I posted...
So, why will  --ies generic,default indeed work?
[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733', '--ies', 'generic,default']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] ytdl-patched/yt-dlp version 2<IP_ADDRESS>9 [a1f3c39] (win32_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] Checking exe version: "..\ffmpeg" -bsfs
[debug] Checking exe version: "..\ffprobe" -bsfs
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.14, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1672 extractors
[debug] [generic] Extracting URL: https://www.dw.com/en/under-construction-indonesias-new-capital-nusantara/av-63121733
[generic] av-63121733: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] av-63121733: Extracting information
[debug] Looking for Brightcove embeds
[debug] Looking for embeds
[html5] av-63121733: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Identified a html5 embed
[info] Available formats for av-63121733-1:
ID       EXT RESOLUTION FPS |   TBR PROTO | VCODEC        VBR ACODEC    ABR
---------------------------------------------------------------------------
hls-386  mp4 480x270     25 |  386k m3u8  | avc1.42c015  386k mp4a.40.2  0k
hls-631  mp4 512x288     25 |  631k m3u8  | avc1.4d4015  631k mp4a.40.2  0k
hls-1137 mp4 640x360     25 | 1138k m3u8  | avc1.4d401e 1138k mp4a.40.2  0k
hls-2632 mp4 960x540     25 | 2632k m3u8  | avc1.4d401f 2632k mp4a.40.2  0k
hls-5064 mp4 1280x720    25 | 5065k m3u8  | avc1.64001f 5065k mp4a.40.2  0k
hls-6500 mp4 1920x1080   25 | 6500k m3u8  | avc1.640028 6500k mp4a.40.2  0k

(apologies if the answer is clear to Python-savvy persons, it just isn't to me... 😢 )

It's just the extraction logic.
Normally (default) yt-dl[p] tries to match the to-be-extracted URL (possibly also checking the page that the URL gives) against the available extractors. The --ies .../--force-generic options change that by forcibly applying an extractor that wouldn't have been selected.
Thus, by default:

match DW URL against extractors
select DWIE by URL match
fail to extract anything
give up (never trying Generic).

While in yt-dlp with --ies generic, default:

forcibly select GenericIE (which matches any URL)
find the embedded HTML5 using the HTML5EmbedIE extractor (enabled because of default)
return extracted HTML5
that's it.

For yt-dl with --force-generic it's similar, but the HTML5 extraction is (for now) in-line in the GenericIE.

Thanks for that very detailed explanation 👍 .

find the embedded HTML5 using the HTML5EmbedIE extractor (enabled because of default)

It was that very bit that eluded me, sadly 😞 ; i.e., unlike youtube-dl, html5IE is only invoked when defaultIE is also specified alongside genericIE...
BTW, I printed myself a list of the available info-extractors, 1673 (!) in total at the moment, and there it was at place no. 525: html5; so, in yt-dlp it's called just html5 (not html5embed).

neither  --ies html5,default nor just --ies html5 work because the yt-dlp HTML5 extractor depends on the generic extractor.

Thanks! According to this,  --ies generic,html5 would work equally as good (in the case of the DW show in question) as  --ies generic,default (and it does 😄 ) ...

I am hitting the same problem:
yt-dlp_linux -v -F  "https://www.dw.com/es/noticias-%C3%BAltima-hora/av-18529525"
[debug] Command-line config: ['-v', '-F', 'https://www.dw.com/es/noticias-%C3%BAltima-hora/av-18529525']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.06 [6becd25] (linux_exe)
[debug] Python 3.10.8 (CPython x86_64 64bit) - Linux-6.0.0-6-amd64-x86_64-with-glibc2.36 (OpenSSL 3.0.7 1 Nov 2022, glibc 2.36)
[debug] exe versions: ffmpeg 5.1.2-1 (setts), ffprobe 5.1.2-1
[debug] Optional libraries: Cryptodome-3.16.0, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1760 extractors
[dw] Extracting URL: https://www.dw.com/es/noticias-%C3%BAltima-hora/av-18529525
[dw] 18529525: Downloading webpage
ERROR: 18529525: An extractor error has occurred. (caused by KeyError('media_title')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp/extractor/common.py", line 680, in extract
  File "yt_dlp/extractor/dw.py", line 53, in _real_extract
KeyError: 'media_title'

This page has what would be HTML5 video except that the tag is <video-js> instead of <video>. This is used for DW's player which appears to be based on video.js.
This structure hasn't been targeted in any existing yt-dl extractor AFAIK. In the BandaiChannel extractor the same tag introduces a Brighcove player that doesn't use <source> elements. Of course video.js is a BC project. The documentation of its getPlayer() method says:

@param   {string|Element} id

     An HTML element - `<video>`, `<audio>`, or `<video-js>` -

     or a string matching the `id` of such an element.

So this patch to the HTML5 parser, which extracts DW's Spanish news, as above, with --force-generic in yt-dl, and I expect in yt-dlp, doesn't seem unreasonable:
-        _MEDIA_TAG_NAME_RE = r'(?:(?:amp|dl8(?:-live)?)-)?(video|audio)'
+        _MEDIA_TAG_NAME_RE = r'(?:(?:amp|dl8(?:-live)?)-)?(video(?:-js)?|audio)'

Now the patch is in the branch: https://github.com/ytdl-org/youtube-dl/commit/ded2fbd635d2d1dedc86887cc95eddc5ab93796b.

see also the uncannily similar suggested solution here: https://github.com/yt-dlp/yt-dlp/issues/6764#issuecomment-1502489642


Luigi Visulizer Stuck

Hi
I am having an issue with the Luigi Visulizer web page getting stuck as shown in the image below:

I am running on Mac OS X 10.11.6 (El Cap.) using Python 2.7.12 and Luigi 2.3.0.
I stopped "luigid" and restarted it but that does not fix the issue.  I have rebooted my machine and restarted but the issue continues.
Is there some file somewhere that I have to clean up ?
EDIT
The URL I am using is "http://localhost:8082/"
Also, this did initially work when I first installed it.
Thanks.

Actually. I have this issue for some reason too. It happens for me when the scheduler isn't running at all, because I think the browser remembers "luigi webui" in the cache or something. It's not re-downloading the frontend stuff on a page refresh. But whenever I restart the scheduler data starts to get populated again.

Unfortunately restarting the scheduler didn't work for me (for Chrome).  I restarted my laptop (where the scheduler was running) and that didn't fix it either.  However, I just launched "luigid" and tried it in Safari and it is NOT having the same problem.  I think you might be right about the browser caching.

I just "fixed" the issue in Chrom.  Instead of reloading with "Command-R" I did "Command-shift-R". That should NOT have fixed it as Command-shift-R is the short cut for "Show Reader", so I am not sure why this is working now.
Thanks for your comments.

Is this something that can be handled ("fixed"?) in the web interface ?

@steveb-cirina, probably there's some setting somewhere telling the browser how long to cache stuff. But actually I have no idea, I know very little about web development. If you look into this and find a fix, please send a PR. :)

Thanks for the info about Firefox producing this issue consistently.  I am new to luigi so I don't know how often this issue occurs or what will cause it to occur.  I will see if I can find a potential fix when it occurs again.

I had this issue too. I recommend to check your browser decoding setting when this happens.
In my case, the browser(Chrome, Firefox) fails to parse URI.js(at line 216 on v2.6.0), because of wrong detection (was Shift-JIS for me) for decoding luigi visualizer's files. Setting 'UTF-8' manually helps to fix this issue.
I guess the fundamental solution is to put <meta charset="utf-8"> on luigi/static/visualiser/index.html.
Thank you:)

@colspan Thanks for digging into this!  I haven't seen this issue since the day I posted this.

closing this issue.

[x] It has been inactive for +4 months.
[ ] It's not about luigi core, so not as many users are affected about this.
[ ] The change seems quite big, it's unlikely to be sporadically picked up.
[ ] The owner haven't responded or disappeared.
[ ] I don't understand what this issue is about.
[ ] There exists a reasonable workaround for this.
[x] We need to check if this hasn't been fixed by now (for old issues).
[ ] This is kind of by design and not a bug.
[ ] Resolving this would probably add a lot of complexity.
[ ] This is a question not an issue. Use SO or the mailing list.

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.
Feel free to reopen this issue at any point if you have the intent to continue to work this. :)


[BUG] Fix main/tip regressions

CI is failing on main due to several regressions in recent merges. Use official release from pypi or release tags until we fix this.
Target fix: Within 12 hours of this issue creation.

Fixed with https://github.com/ModelCloud/GPTQModel/pull/533


transform: Send Slack message with future dates

Sometimes we receive data from GISAID with collection dates in the
future. This causes a big headache for the Nextstrain nCoV build
maintainers when left uncaught until the end of a multi hour long build.
Send a Slack message containing the GISAID EPI ISL ID and collection
date for every strain that has a collection date from the future (e.g.
a date > now). Emma specifically requested that this Slack message be
formatted differently from the others so that it stands out.
Related issue(s)
#33

Here's an example of the current format of this "future dates" message (using fake dates I manipulated):

 ncov-ingest: warn about future dates

I haven't tested (not even sure I can!) but from a quick look over the code, this looks great @kairstenfay , thank you so much!! This will be a huge help to the runs!

I made the requested changes by Tom and also incorporated feedback from the conversation in #33.
This PR now runs an exclude script after transform and uses the existing notify-slack script to upload files to Slack. Exclusions are automatically saved to a new S3 file called exclude.txt that will be used by ncov along with the manually created exclude.txt.

Looks good to me. A couple minor things to clean up.
We'll need to sort out the fate of nextstrain/ncov#401 before merging this, right?

@tsibley Yes. I think the simplest resolution is to make pulling exclude.txt from S3 optional.

I like this idea, if there's a way we can use today's date in the call? (I am sure this is possible, but unsure the most robust way to do this). This would be nice as it would mean as dates were updated, they'd be auto-included without having to update any files (always a plus).
Would still be good to get Slack notifications of the future dates though, so we can flag them to submitters!

I like this idea, if there's a way we can use today's date in the call? (I am sure this is possible, but unsure the most robust way to do this). This would be nice as it would mean as dates were updated, they'd be auto-included without having to update any files (always a plus).
Would still be good to get Slack notifications of the future dates though, so we can flag them to submitters!

I pushed a commit to nextstrain/ncov#401 that uses augur filter --max-date to exclude future dates.  Instead of creating a new script called exclude here, I wonder how we can semanitcally reflect that we're not going to store sequences with future dates in an automatically generated exclude.txt for augur exclusions, but we'll still track them for Slack notifications.

I wonder if we should just not call it exclude.txt but instead future_dates.txt or similar. This shows exactly what we're tracking, and can help us to alert submitters, but doesn't imply that it going to go into the exclude file used by filter. (As they won't need to, with the max-date option.) Or, if we think we may want to keep track of a larger variety of files, we could call it something like flagged_sequences.txt (though we'll want to note this is different from ncov/results/flagged_sequences.tsv which stores the results of the sequence diagnostics...

I wonder if we should just not call it exclude.txt but instead future_dates.txt or similar. This shows exactly what we're tracking, and can help us to alert submitters, but doesn't imply that it going to go into the exclude file used by filter. (As they won't need to, with the max-date option.) Or, if we think we may want to keep track of a larger variety of files, we could call it something like flagged_sequences.txt (though we'll want to note this is different from ncov/results/flagged_sequences.tsv which stores the results of the sequence diagnostics...

I like where you're going with flagged_sequences.txt. Naming things is hard and I agree we don't want to confuse this file with ncov/results/flagged_sequences.tsv. I'll keep thinking on this.

I like where you're going with flagged_sequences.txt. Naming things is hard and I agree we don't want to confuse this file with ncov/results/flagged_sequences.tsv. I'll keep thinking on this.

What are your thoughts on flagged_metadata.txt? This file would contain any and all sequences that have problematic metadata and be sent to slack. These sequences won't be dropped in ingest but possibly filtered out downstream (like in augur filter --max-date xx-xx-xxxx).
One thing to keep in mind is this will eventually live alongside exclude. In a new PR, I will re-create the exclude script and the exclude.txt file for automatic exclusions (see #45 for an example).

What are your thoughts on flagged_metadata.txt? This file would contain any and all sequences that have problematic metadata and be sent to slack. These sequences won't be dropped in ingest but possibly filtered out downstream (like in augur filter --max-date xx-xx-xxxx).

I think that's a good name, but it'll be a bit confusing if it also contains sequences which will be excluded due to sequence attributes (like length, in the example) you linked to. I think the other thing is we'll want to ensure this file only contains what's 'new' if it's going to the slack... getting a long list of everything ever won't be helpful in the Slack! (Too much info!) (Though might be useful to store somewhere.)
However if we're using this mostly for metadata stuff, like dates, I think the name is good! Perhaps the 'latest additions' can go to the slack, and all can be included in a file with reasons for why it's 'bad'. Hope I'm understanding!

I think that's a good name, but it'll be a bit confusing if it also contains sequences which will be excluded due to sequence attributes (like length, in the example) you linked to.

It won't, ideally. The example I linked to would put sequences in an exclude.txt file like this PR originally proposed.

I think the other thing is we'll want to ensure this file only contains what's 'new' if it's going to the slack... getting a long list of everything ever won't be helpful in the Slack! (Too much info!) (Though might be useful to store somewhere.)

I see two solutions here.

We could append all "flagged metadata" to one file and send only the diff via Slack (like what we do for metadata.tsv).
We could overwrite a "flagged metadata" file every ingest that contains only bad metadata from new or changed rows.

I think 1 may be simpler, but I definitely agree we don't want to clutter Slack unnecessarily.

However if we're using this mostly for metadata stuff, like dates, I think the name is good! Perhaps the 'latest additions' can go to the slack, and all can be included in a file with reasons for why it's 'bad'. Hope I'm understanding!

Sounds good, thanks. I would like to format this file in the same style as exclude.txt where each sequences has an inline comment describing the problem.

We could append all "flagged metadata" to one file and send only the diff via Slack (like what we do for metadata.tsv).

I think this would be great, if it's not too much trouble to do! If sequences were updated (to have a correct date), would they disappear from this file?

Sounds good, thanks. I would like to format this file in the same style as exclude.txt where each sequences has an inline comment describing the problem.
Sounds perfect!

I think this would be great, if it's not too much trouble to do! If sequences were updated (to have a correct date), would they disappear from this file?

Not in the way I described, but this is even better and could certainly be arranged.

This PR now adds strains with future dates to a file called flagged_metadata.txt that is in the same style as the ncov exclude.txt. It sends Slack notifications for any new exclusions not in the old file stored on S3 and overwrites that file at the end of each ingest.

I rebased this onto master, squashing commits in the process to reduce code churn, and re-pushed. If the workflows run without error, I'll merge.


Looks like the submit data form is not working correctly

When you try and submit data from our form the page opens as shown in the attachment.
Looks different that before, parts are missing.

Missing many of the fields that use to be there. second attachment is a final form with missing entries highlighted.

not sure what happened.

When we switched to plss.utah.gov a path to the coord parameters was broken.

Thanks, you made quick work of that.


CART_MAX undefined

Trying to compile the current master branch at revision 627ca5f73e3f39a945bf5b2e5b1057e046aeb1bb, I get the error
/home/work/pyscf/pyscf/lib/gto/ft_ao.c: In function ‘_g2c_index_xyz’:
/home/work/pyscf/pyscf/lib/gto/ft_ao.c:138:18: error: ‘CART_MAX’ undeclared (first use in this function)
  138 |         int i_nx[CART_MAX], i_ny[CART_MAX], i_nz[CART_MAX];
      |                  ^~~~~~~~

It looks like this macro was removed by 392b827491e7c0b9b13b1283a3deb97ab79e482f, but I can't find a definition for CART_MAX anywhere in the source tree. It is still used in pyscf/lib/gto/ft_ao.c and pyscf/lib/gto/nr_ecp.c

Oh looks like the issue was that the definition was moved inside libcint and I just had to remove the compiled copy to trigger a recompile.


Migration form Version 3.2 to 7.0

Which version and edition of Flyway are you using?
3.2
Problem Statement
We want to use the capability added in V7.0 to use S3 bucket to have the script files.
I know we cannot migrate directly from 3.2 to 7.
Can you please guide how we go about with the migration from 3.2 to latest version?

The only supported path is to migrate from 3.2 to 4.x, then 5.x, 6.x, 7.0; so install each version in turn and run at least one command (so that the history table is updated to the newer version).
As this is likely to be quite tedious, an alternative might be to remove the schema history table, upgrade to v7, baseline at the appropriate version and continue. This obviously loses past Flyway history, which will probably not be acceptable to you.


USN-3464-1: Wget vulnerabilities

Package wget should be upgraded in quay.io/deis/slugbuilder image due to USN-3464-1

I made an issue on Heroku stack images repository: https://github.com/heroku/stack-images/issues/84

This issue was moved to teamhephy/slugbuilder#1


is there an install syntax to specify package version?

I would trying to install a specific version of gcc (gcc-9) in an action setting, but could not find any examples to do so - is this supported?
additionally, I am looking for the package that provides static library libgomp.a, I only found that mingw-w64-x86_64-gcc-libs provides libgomp.so, but not the static library.
any help is appreciated!

It is not possible to install old packages like that. Old packages can be found in https://mirror.msys2.org/mingw/ repository. But installing them may break the whole setup and you have to manually install those packages with pacman -U <full-link-here> command.

I would trying to install a specific version of gcc (gcc-9) in an action setting

In the actions log, there seems to be some path conflict between the mingw toolchains in chocolatey and msys2. If you have any issue with latest gcc toolchain it may be fixed faster than using old gcc.

thanks for your quick reply and helpful comments. I realized that pacman and msys2 was made for rolling releases, which is aimed to only serve the latest.

In the actions log, there seems to be some path conflict between the mingw toolchains in chocolatey and msys2. If you have any issue with latest gcc toolchain it may be fixed faster than using old gcc.

I agree. The actual issue I am trying to solve is the missing of static libraries: libz.a, libgomp.a and libpthread.a when I used the default Mingw64 8.1 gcc toolchain provided by the windows-2019 runner. here is the log
https://github.com/fangq/mmc/actions/runs/4356395089/jobs/7614344016
I searched and only found commands to install packages via msys2, if I am able to locate the packages in msys2 that provides these .a files, I should be able to set shell: msys2 and use the mingw64 gcc installed on msys2 (12.x) to complete the compilation;
I just tried to mix the mingw64 gcc (9.0.2) with the .a files extracted from msys2, but it failed with undefined reference errors
https://github.com/fangq/mmc/actions/runs/4357600651/jobs/7617111580
I thought that if I could downgrade the msys2 gcc version, I may have a better luck to link.
alternatively, I may be able to solve this by finding an action script to install libraries for mingw64 instead of msys2, but I failed to find such command :-(

if I am able to locate the packages in msys2 that provides these .a files

You can search files with pacman. For example, here I am searching for libpthread.a in ucrt64
$ pacman -Fq libpthread.a | grep ucrt
ucrt64/mingw-w64-ucrt-x86_64-winpthreads-git

libpthread.a: https://packages.msys2.org/base/mingw-w64-winpthreads-git
libz.a: https://packages.msys2.org/base/mingw-w64-zlib
libgomp.a: https://packages.msys2.org/base/mingw-w64-gcc

In the actions log, there seems to be some path conflict between the mingw toolchains in chocolatey and msys2. If you have any issue with latest gcc toolchain it may be fixed faster than using old gcc.

@Biswa96, you were spot-on! turned out most of my struggle was a result of mixing gcc versions - I added path to Octave's bin folder which also contains a bundled gcc. After appending mingw64's gcc path in GITHUB_ENV, I was able to run static linking without needing to install any additional package (or msys2 environment).
closing the ticket. thanks again for all the helpful comments!


[SourceKit] Make sure to propagate the SIL options from the compiler invocation when doing performSILGeneration() to get the SIL diagnostics

master: https://github.com/apple/swift/pull/9854
rdar://32329669

@swift-ci Please test

Explanation: SIL options from the compiler invocation are not propagated by sourcekitd when retrieving the SIL diagnostics. Due to this issue the '-enforce-exclusivity=' option is getting ignored for diagnostics emitted by sourcekitd.
Scope: Only affects sourcekitd, the code path that retrieves SIL diagnostics.
Radar: rdar://32329669
Risk: Low
Testing: Added regression test, verified with test project


Out of date BERT Tutorial Link

Thank you for submitting a TensorFlow documentation issue. Per our GitHub
policy, we only address code/doc bugs, performance issues, feature requests, and
build/installation issues on GitHub.
The TensorFlow docs are open source! To get involved, read the documentation
contributor guide: https://www.tensorflow.org/community/contribute/docs
URL(s) with the issue: https://www.tensorflow.org/guide/distributed_training
Description of issue (what needs changing): Out of date link to BERT tutorial
Clear description
The above URL contains an out of date link:
BERT Tutorial: https://github.com/tensorflow/models/blob/master/official/bert/run_classifier.py
This is the correct link: https://github.com/tensorflow/models/blob/master/official/nlp/bert/run_classifier.py
Are you planning to also submit a pull request to fix the issue?  Yes

https://github.com/tensorflow/docs/pull/1113


nano GPT preparation

Thank you for sharing the code for this excellent work!
Do the following data/shakespeare_char/prepare.py and data/text8/prepare.py come from this fork repo of the original nano GPT?
# Prepare NanoGPT data
python data/enwik8/prepare.py
python data/shakespeare_char/prepare.py
python data/text8/prepare.py

Ahh, fixed thanks! The folder was in our .gitignore and forgot to fix that for the release

Cool! Data preparation now works!
# Set up NanoGPT baseline run
python templates/nanoGPT/experiment.py --out_dir run_0

This on the other hand does not. Had to change the path in experiment.py line 387:
    # poor man's data loader
    if out_dir == "run_0":
        #data_dir = os.path.join("../../data", dataset)
        data_dir = os.path.join("data", dataset)

    else:
        data_dir = os.path.join("../../../data", dataset)

Changing the command in the README to this will avoid many unnecessary path problems. @AI-Guru @luchris429
before
python templates/nanoGPT/experiment.py --out_dir run_0
after
cd templates/nanoGPT && python experiment.py --out_dir run_0

fixed -- thanks!

First, this is brilliant, thank you for open-sourcing.  Second, why do we need to set up an LLM API and also nanoGPT?

I have been trying to get AI Scientist up and running on my machine. For quite a few of the instructed steps, I had do fix some things, mostly directory names that do not fit.
Would the authors mind doing a fresh checkout of the repo and then try the instructions? :D

Here is an example:
Processing idea: adaptive_block_size
Failed to evaluate idea adaptive_block_size: [Errno 2] No such file or directory: 'templates/nanoGPT_lite/run_0/final_info.json'
Processing idea: adaptive_dropout
Failed to evaluate idea adaptive_dropout: [Errno 2] No such file or directory: 'templates/nanoGPT_lite/run_0/final_info.json'

Please check the instructions in: Create baseline runs (machine dependent)!

Thanks!
I went through the instructions from top to bottom. I believe that as they are written, they do not work 100% 🤗

Cloning into 'aider'...
info: please complete authentication in your browser...
remote: Repository not found.

Why are you cloning into aider? Please see the installation instructions in Requirements - Installation :)

Slowly getting there! I pulled and followed the updated instructions.
Now I am getting this:
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hordak/Development/AI-Scientist/results/nanoGPT_lite/20240813_211123_adaptive_block_size/experiment.py", line 511, in train
    X, Y = get_batch("train")  # fetch the very first batch
           ^^^^^^^^^^^^^^^^^^
  File "/home/hordak/Development/AI-Scientist/results/nanoGPT_lite/20240813_211123_adaptive_block_size/experiment.py", line 393, in get_batch
    data = np.memmap(
           ^^^^^^^^^^
  File "/home/hordak/miniconda3/envs/ai_scientist/lib/python3.11/site-packages/numpy/core/memmap.py", line 229, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '../../data/shakespeare_char/train.bin'

This is what I started:
python launch_scientist.py --model "gpt-4o-2024-05-13" --experiment nanoGPT_lite --num-ideas 2

Have you run the prepare steps? Please verify you have data/shakespeare_char/train.bin!

 Please verify you have data/shakespeare_char/train.bin!

Yes I did. I copied it to ../.. to make it work.
It now runs experiments. I guess not every experiment yields a PDF?

Yes, please see our paper for success rates. With GPT-4 you may expect something like 20 PDFs per 50 runs.

why do we need to set up an LLM API and also nanoGPT? Couldn't we just use the LLM API?

The LLM API writes the code, nanoGPT is the base model that is being edited.

Using GPUs: []
Using Anthropic API with model claude-3-5-sonnet-20240620.
Generating idea 1/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 2/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 3/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 4/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 5/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 6/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 7/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 8/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 9/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 10/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 11/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 12/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 13/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 14/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 15/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 16/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 17/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 18/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 19/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 20/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 21/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 22/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 23/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 24/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 25/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 26/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 27/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 28/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 29/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 30/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 31/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 32/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 33/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 34/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 35/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 36/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 37/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 38/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 39/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 40/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 41/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 42/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 43/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 44/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 45/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 46/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 47/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 48/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 49/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Generating idea 50/50
Iteration 1/3
Failed to generate idea: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Checking novelty of idea 0: adaptive_block_size
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Checking novelty of idea 1: layerwise_learning_rates
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
Error: "Could not resolve authentication method. Expected either api_key or auth_token to be set. Or for one of the X-Api-Key or Authorization headers to be explicitly omitted"
All ideas evaluated.

Set the anthropic api key, see instructions in the read me.


Show sender machine in web gui?

Great product, can I make a feature request?
It'd be a huge timesaver to show which machine the web interface were accepted from. A filter using that would help a lot too. Thanks.

Hi @ian-kent do you accept bounties for features?

@macau23 Can you explain a little more about what you are asking for?

@tyndyll I can't remove the waiting label from this issue, do you need more info?

@tyndyll I know someone who would be willing to sponsor this

@macau23 Drop me an email. My profile should have the email address


feat: add faq section

update DefaultLayout to allow customized margin
add FAQ section

Blocker
#179

@ctrlc03 so this is actually a template for those who wanna fork our repo and do their own FAQ, so actually these contents are just example text. However I agree it would be confusing. Maybe let's make it more reasonable to be example FAQ content? (but I have no idea what kind of content could be example)

@ctrlc03 so this is actually a template for those who wanna fork our repo and do their own FAQ, so actually these contents are just example text. However I agree it would be confusing. Maybe let's make it more reasonable to be example FAQ content? (but I have no idea what kind of content could be example)

yea I think we could have one faq each around what maci and maci-rpgf are, then requirements, what is public goods funding and method of reaching out for more questions (discord public channel for now)

since this would be hard to rebase, would open another PR for this faq integration branch from current feat/design-ui, keep this PR until I open the new one.

close by #190


GDScript: Fix native class not set with inheritance

This is a bug that may make compilation be wrong in certain cases.

Thanks!


trace_async broken for async_trait usage

Hi 👋
I am trying to use the trace_async macro to add span tracing on a tonic Grpc server using async functions. I am getting an error of error[E0599]: no method named `in_new_span` found for struct `Pin<Box<impl futures_util::Future>>` in the current scope when I try applying the trace_async macro to one of these async functions.
It appears that maybe in_new_span was renamed to in_local_span in this pr, but the usage on line 105 of crates/minitrace-macro/src/lib.rs was missed in the refactoring?
I am new to Rust, so I don't know if there are any implications for fixing this beyond just changing the name to in_local_span, but I would be happy to help if I can.

It appears that maybe in_new_span was renamed to in_local_span in this pr, but the usage on line 105 of crates/minitrace-macro/src/lib.rs was missed in the refactoring?

You are right.

I am new to Rust, so I don't know if there are any implications for fixing this beyond just changing the name to in_local_span, but I would be happy to help if I can.

Just to change the name then it will be fixed.

@zhongzc thanks for the answer. I have made the change and am operating off of my own fork, I am planning on making a PR when I can verify it is working as expected.
One question I have is that when I annotate an async function with the trace_async macro, I appear to end up with multiple traces for one function call. I think I might be ending up with a trace per poll, which I do see mentioned in the docbloc on in_local_span. Is this really expected, to have multiple traces per function call? Is there some way to combine these traces?
Here is an example of a span within Jaeger where there is a root span that calls an async service function, which itself calls a different async function, each annotated with trace_async. I suppose I would only expect to see three nested spans, one for root, one for mailbox_ack and one for datastore_ack_mailbox_public_key.

There isn't a good way to achieve your requirement with the current version.
[trace_async("mailbox_ack")] adds code for futures automatically as follow:
a_future.in_local_span("mailbox_ack").await;

Changing it to
a_future.in_span(Span::from_local_parent("mailbox_ack")).await;

will partially meet your requirement, but it's not easy as just adding an annotation [trace_async]. Besides, the parent-child relationship between mailbox_ack and datastore_ack_mailbox_public_key will be missed.
I'm not trying to convince you, but that outcome is really what I am hoping for as from that graph I can figure out that:

When were futures suspended and when were they resumed
How long was the real execution time and how long was the waiting time, not just the complete time

Thanks for the explanation. That makes sense, no sense in indicating that the code was actually doing anything when really it was just waiting on i/o. Also, I agree that the parent-child relationship is more important than these separated spans.
Is there anything that could be used to link/group those spans together (i.e. to indicate that a collection of spans were resulting from one function call, vs. multiple calls to the same function one after another, each resulting in multiple spans)?

Is there anything that could be used to link/group those spans together (i.e. to indicate that a collection of spans were resulting from one function call, vs. multiple calls to the same function one after another, each resulting in multiple spans)?

AFAIK, there is no such thing defined by opentracing spec. You can manually specify a parent span.
For example, we have a loop run some code multiple times; every time generates a span. Then we can group these spans by letting them have the same parent:
let _parent = LocalSpan::enter("a group of spans");

while running {
  let _child = LocalSpan::enter("some code");
  // some code
}

We will get finally:
[                            a group of spans                                 ]
 [ some code ]
                        [ some code ]
                                               ......
                                                                 [ some code ]

Thanks for the response again @zhongzc, you are very helpful, I really appreciate it.
I just put the fix for the async_trait usage in a PR here https://github.com/tikv/minitrace-rust/pull/65
I guess maybe as a feature request, in my mind it would make sense if there could be some way for there to be an automatically-generated parent span for the async polls that wouldn't involve adding code for the parent span around function calls (i.e. just adding the trace_async macro to a function would automatically lead to a parent span grouping the child polls). I don't know enough about how this might be architected, or how it would need to be designed to make sense in all situations (i.e. would each child span have a prefix of poll: or something?), but just thought I would put that out there in case it leads to something useful.

@scirner22 WIth default behavior #[trace] in the latest 0.2.0 traces the entire lifetime of async fn including polling and parking. The previous behavior of starting a new span on polling is enabled by a new macro attribute#[trace(event, enter_on_poll = true)].

Ah, just to mention,#[trace_async] is removed, and #[trace] handles asyncness and async_trait automatically now.

That's awesome. Thanks!


GPII-1259: Implementation of standalone micromodule deduping infusion

Matched pull request for universal is https://github.com/GPII/universal/pull/402

Thanks @colinbdclark , pushed version with feedback addressed


Updating a terminal's width affects its status icon

I wasn't sure if there was already an issue for this: https://github.com/microsoft/vscode/issues/155163#issuecomment-1184872077
Here's the recording:

It's interesting that I only noticed it last Friday. Is it a coincidence?

@joaomoreno I think it's more likely to happen here because of the reconnect, pretty sure it's the same underlying cause though.

Makes sense thanks!


Examples are automatically pulled if not present before any example command is run

…li command is run
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[x] If my change requires a change to the documentation, I have updated the documentation accordingly.
[x] I have added tests to cover my changes.

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Describe changes
Examples are automatically pulled if not present before any example command is run

In this context pulled means that the examples are present in the global zenml folder. As we are not sure from where the user is running zenml example * we don't know if he actually wants it copied to his current working directory. In this case I will add a better lof message telling the user to run zenml example pull with the desired destination folder (zenml_examples being the default)

From the users perspective, doesnt it make sense to just work? Like just let it run and pull to the default?

You're right.
How about if without -f, the user needs to confirm that it will be copied into his cwd.
With -f, its pulled, copied into cwd and then run.

that also works. We should reflect the example readme and all the individual examples readmes to reflect this if we take that route though :-)

Implemented the functionality of pulling if not present. Now if you do zenml example run quickstart it will use the cli context to forward to zenml example pull quickstart with the same parameters if there is no example present locally.
Do you think its alright to use cli context for this, or should I go back and put the functionality of the pull command into a separate callable function?


Confluence audit

Proposed Changes

Adding Confluence Audit Connector

@averbn : could you please resolve the comments.
Thanks,
Aditi Maheshwari

@averbn , Please add azure function zip package in data connector folder. Thanks!.

@averbn : We have made required changes for logAnalyticsUri and also add the zip file in the PR. Could you please verify the changes.
Thanks,
Aditi Maheshwari


apt upstream broken in ubuntu-latest (22.04.2 version 20230219.1)

Description
it looks like something has gone wrong with the upstream packaging / repo signing (not technically the image, but, impacting the image). this appears to be intermittent, and has occured in march in the past https://github.com/actions/runner-images/issues/2919, so perhaps one of the repository mirrors is out of sync / hasn't picked up updated keys?
calling apt update or equivalent results in:
Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
...
Get:23 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [13.3 kB]
Reading package lists...
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy Release' is no longer signed.

this may be a recurrence of https://github.com/actions/runner-images/issues/5656
Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[X] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[X] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
20230219.1
Is it regression?
yes (this is an upstream package repo issue, the same image worked yesterday)
Expected behavior
apt update should work
Actual behavior
apt update fails with E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy Release' is no longer signed.
Repro steps
create an action using ubuntu-latest with a step invoking apt

@ryankurte thank you, we will take a look.

looks like it has been healed on the repo side (+ network)

I got this again, today, on Github Action with runs-on: ubuntu-latest  @ddobranic
Could you have a look?

Faced the same problem as  @terrynguyen255 - Github Action and ubuntu:latest.
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease' is no longer signed.

I believe yesterday everything was fine 🤷♂

Started facing the same issue on CI runs today
E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease' is no longer signed.

Hi @mikhailkoliada. Sorry to ping you here directly but could I draw your attention to this issue? This is also happening for us on our build and judging from the activity above this seems like a widespread issue atm.


Not change in deye inverter parameters

Maybe some can help me I added this SOLARMAN to my home assistance for automatically charge battery in depend from forecast weather.
Added automatic action, but without any result nothing change on deye dashboard if I run auto or try to set soh parameters from home assistant.
maybe I need add any addons or change some in my inverter

Which inverter do you have the one from deye

Deye 12 kw sg04lp03
Roman Lambruh
+421915529840
+380503727783
пт, 11 жовт. 2024 р. о 23:31 CrazyUs3r @.***> пише:

Which inverter do you have the one from deye
—
Reply to this email directly, view it on GitHub
https://github.com/davidrapan/ha-solarman/issues/207#issuecomment-2408084273,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFKVG6SJIJOA7GDALKOVXNDZ3AYS5AVCNFSM6AAAAABPZO2DQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYGA4DIMRXGM
.

Please write me directly to my mail
@.***
I will be very appreciate if you will help me solve it
Roman Lambruh
+421915529840
+380503727783
сб, 12 жовт. 2024 р. о 14:34 David Rapan @.***> пише:

You are probably doing something wrong, show me the yaml of automation you
are using.
—
Reply to this email directly, view it on GitHub
https://github.com/davidrapan/ha-solarman/issues/207#issuecomment-2408531592,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFKVG6XHJ2RGK6VXEIPYWY3Z3ECN7AVCNFSM6AAAAABPZO2DQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYGUZTCNJZGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

Finaly today i found some part solution.
If i change number.inverter_program_1_soc manualy  from entery - charge parameters in Deye change well.
Problem that i can't undarstand how work automatisation, i dont have next sensore -
sensor.solcast_pv_forecast_forecast_today, in my entery.
I made copy to Yaml for creating auto.

This is my automation:
alias: Inverter Grid Charging Program
mode: single
triggers:
  - at: "03:00:10"
    trigger: time
  - at: "04:00:00"
    trigger: time
  - at: "05:00:00"
    trigger: time
actions:
  - sequence:
      - action: solcast.update_forecasts
        data: {}
      - action: number.set_value
        metadata: {}
        target:
          entity_id: number.inverter_program_1_soc
        data:
          value: "{{ states('sensor.inverter_battery_precharge') }}"

and sensor.inverter_battery_precharge is template sensor which has some if - elif - else statements according to the sensor.solcast_pv_forecast_forecast_today.

and for forecasts use other extension like this,
https://github.com/bjReplay/ha-solcast-solar
for whether forecast,
чт, 17 окт. 2024 г. в 20:43, David Rapan @.***>:

This is my automation:
alias: Inverter Grid Charging Program
mode: single
triggers:

at: "03:00:10"
trigger: time
at: "04:00:00"
trigger: time
at: "05:00:00"
trigger: time
actions:
sequence:

action: solcast.update_forecasts
data: {}
action: number.set_value
metadata: {}
target:
entity_id: number.inverter_program_1_soc
data:
value: "{{ states('sensor.inverter_battery_precharge') }}"

and sensor.inverter_battery_precharge is template sensor which has some if

elif - else statements according to the
sensor.solcast_pv_forecast_forecast_today.

—
Reply to this email directly, view it on GitHub
https://github.com/davidrapan/ha-solarman/issues/207#issuecomment-2420132412,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFKVG6RQP3TQ6EQSSNPSUA3Z37ZL3AVCNFSM6AAAAABPZO2DQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRQGEZTENBRGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Roman Lambruh
+421915529840
+380503727783

Yes I do use solcast integration.


dynamic redirectTo rule with comeFrom url (and params)

Hello, I have a question
I need to pass to the redirectTo state a paramenter for where you are come from (that parameter must contain also the parameter of the state you are redirected from in order to be redirected correcly after the login ) is this possible?
example
if i enter to state1 without be logged
redirect to login state with a param ?goto=state1/param
from state2
redirect to login state with a param ?goto=state2?id=id
i tried to handle using dynamic redirection rules
but the transitionProperties object is a string (the permission definition) and the toState, fromState,toParams are undefined
data: {
                    permissions: {
                        only: ['logged'],
                        redirectTo: function(transitionProperties){
                            console.log("transition properties");
                            console.log(transitionProperties);
                            console.log(transitionProperties.toState);
                            console.log(transitionProperties.toParams);
                            console.log(transitionProperties.fromState);
                            return 'login'
                        }
                    }
                }

actual state definition
$stateProvider
    .state('state1', {
      url: "/state1/:param",
      templateUrl: "partials/state1.html",
                data: {
                    permissions: {
                        only: ['logged'],
                        redirectTo: 'login'
                        }
                    }
                }
    })
    .state('state2', {
      url: "/state2/?{id:string}",
      templateUrl: "partials/state1.html",
                data: {
                    permissions: {
                        only: ['logged'],
                        redirectTo: 'login'
                        }
                    }
                }
    })
    .state('login', {
      url: "/login",
      templateUrl: "partials/login.list.html",
      controller: "loginCtrl"
      }
    })

estratto bower.json
"angular-ui-router": "^0.2.18",
"angular-permission": "^3.0.0",

i was forced to not use the angular-ui-router 1.0.0-alpha.4 because the the state permission are ignored with that version

Thanks you very mutch for now i finded a very simple workaround
remove te redirect to from the state and add a custom redirect to the $stateChangePermissionDenied event

i will try, but i think will not possible by today

oh i get it, the problem is in the documentation that need to be corrected


Comparison with Google's Closure Compiler?

Great project! I was wondering if there could be a comparison with Closure Compiler? Willing to do it if you think it'd be worth.

Couldn't they be used together? Complementing each other

Sure, I'd be interested in seeing a comparison. Please feel free to compare the two and then write a blog post about it, and I'll be glad to link to it from the README. :)


More descriptive results

Instead of "Eligible" and "Ineligible", a sentence like "You are eligible to have your conviction expunged." would be nice. If the app deals with expungement vs. sealing (as in #7), it could be expanded to "You are ineligible to have your conviction expunged, but you may be able to have your records sealed." or something like that.

Agreed. A short sentence explaining the reason would also be good (though more a nice to have).


unblock pushbullet-types

pushbullet-hs now builds, following
https://github.com/tsani/pushbullet-hs/commit/da06bce
Checklist:

[x] Meaningful commit message - please not Update build-constraints.yml

[x] At least 30 minutes have passed since Hackage upload

[x] On your own machine, in a new directory, you have successfully run the following set of commands (replace $package with the name of the package that is submitted, $version is the version of the package you want to get into Stackage):
stack unpack $package
cd $package-$version
stack init --resolver nightly
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks

Hey @tsani looks like pipes-concurrency got dropped from your version of build-constraints.yaml; if you accept tsani/stackage#1 that should fix everything and clear this to be merged

Thanks @jkachmar ! I accepted your PR and squashed it into my original commit.


Download URL text is wrong when version is first added.

When I first add a version of a mod the Download URL text seems to be corrupt. The url of the link itself is correct because if I click on it it correctly downloads the mod. It's just the text as you can see in the image below.

If I refresh the issue goes away, though.

Strange - I just installed solder fresh on a new VPS and I don't have that issue.
I am using MySQL, PHP 5.6 and Apache though

Hm. I'm using sqlite, php7, and nginx if that helps.

It could be a combination of PHP7 & ngix - maybe. I did notice with some of the commits in dev branch there was some work on PHP7 being done - not sure if its ready for PHP7 or not.  Someone else will have to answer that.

Hang on, your right @adamk33n3r.  It just happened to me as well.
So far it happens on

Apache or nginx
SQLite or MySQL
PHP7 or 5.6

For the record my VPS that I just installed to is CentOS 7.
My apologies for jumping the gun on you @adamk33n3r.

@spannerman79 no worries :)
Mines installed on an arch box for the record.

I also have this issue.

Ubuntu 14.04 LTS Server
MySQL (Percona Cluster to be exact)
PHP 7.0
Nginx (Latest)

I also just tested it on an AWS VPS and saw the issue there as well.

There's an extra { in code, it's a very simple fix.
https://github.com/TechnicPack/TechnicSolder/blob/master/app/views/mod/view.blade.php#L149
Change the line from:
$("#add-row").after('
	' + data.version + '	' + data.md5 + '	name }}/{{ $mod->name }}-' + data.version + '.zip" target="_blank">{mods/{{ $mod->name }}/{{ $mod->name }}-' + data.version + '.zip	' + data.filesize + '
');
to:
$("#add-row").after('
	' + data.version + '	' + data.md5 + '	name }}/{{ $mod->name }}-' + data.version + '.zip" target="_blank">' + mirror_url + 'mods/{{ $mod->name }}/{{ $mod->name }}-' + data.version + '.zip	' + data.filesize + '
');
And the issue fixes itself.

Will close issue once dev is merged into master.


Feature #40 for testing.

This is a fix for #40.
Instructions:

Install this version of i2pd,
test on Android versions before Lollipop;
and leave it to work during several days on Lollipop or later. It should not be killed by Android, hopefully.

Test session 1

Device 1: physical Android 9. API LEVEL > 21.
Device 2: emulator Android 4.1 API LEVEL 16 rev (7 or 8).

1.1 Device 1 Phases:

After approx. 6 hours of work, it had been online and working OK, IRC connection through it had been working OK.
I pressed Open Webconsole, the app has disappeared and immediately quit.

1.2 Device 2 Phases:

After approx. 6 hours of work, all is OK;
I opened Webconsole, the visuals were garbled and App's "back arrow" button invisible (emulator artifacts???).

2 Test session 2
2.1 Device 1 Phases

2.1.1 Test Started Sunday Oct. 9, 7 am

2.2 Device 2 Phases

2.2.1 Test Started Sunday Oct. 9, 7 am

2.1.2 test failed:

--------- beginning of system
2022-10-09 07:05:49.694  7128-7128  ActivityThread          org.purplei2p.i2pd                   W  Application org.purplei2p.i2pd is waiting for the debugger on port 8100...
--------- beginning of crash
2022-10-09 11:01:53.197 27226-27226 AndroidRuntime          org.purplei2p.i2pd                   E  FATAL EXCEPTION: main

java.lang.NullPointerException: Attempt to invoke virtual method
    'void org.purplei2p.i2pd.DaemonWrapper.addStateChangeListener(
        org.purplei2p.i2pd.DaemonWrapper$StateUpdateListener)'
    on a null object reference

    at org.purplei2p.i2pd.MyJobService.onStartJob(MyJobService.java:29)
    at android.app.job.JobService$1.onStartJob(JobService.java:62)
    at android.app.job.JobServiceEngine$JobHandler.handleMessage(JobServiceEngine.java:108)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loop(Looper.java:280)
    at android.app.ActivityThread.main(ActivityThread.java:6706)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
    ```

  * 2.2.2 test is still ok working

3 Test session 3, device 1

Test started at 2022-10-09 13:33:33.092 i2pdActvt               org.purplei2p.i2pd                   D  onCreate

<postsolarpunk_hypn> R4SAS, результаты уже чуть лучше - I2PDActivity уничтожилась и её скушал сборщик мусора, но i2pd JobService каждую минуту шлёт пинг и после этого
<postsolarpunk_hypn> т.е. тред пинговалки живёт после I2PDActivity.onDestroy

This experiment has failed: Android closed i2pd-android app after 50 hours of uptime.

logcat:
10-14 02:25:25.088  1694  1867 W BroadcastQueue: Background execution not allowed: receiving Intent { act=org.purplei2p.i2pd.MAKEMEAJOB_ACTION flg=0x10 } to org.purplei2p.i2pd/.receivers.MakeMeAJobReceiver

This experiment (not committed) has failed: Android has closed i2pd-android app after 10 hours of uptime.


[Android] RefreshServices causes reads to hang

Please check all of the platforms you are having the issue on (if platform is not listed, it is not supported)

[] iOS
[*] Android
[] tvOS
[] macOS
[] .NET Standard
UWP IS NOT SUPPORTED DURING THE BETA PERIOD - ONLY PRs ARE WELCOME FOR NOW

Version of OS(s) listed above with issue
6.0 and 7.1
Version of Library
6.2.3
Expected Behaviour
When RefreshServices Android configuration is set to true, I should be able to read characteristics as normal
Actual Behavior
If applicable, please include entire exception - type, message, stacktrace
With RefreshServices enabled, I can only manage a few reads on different characteristics (which seem to progressively get slower) before they all just start timing out.
Turning RefreshServices back off returns everything back to normal.
I turned it on as i need to do OTA updates but having it always on seems to not play nicely when trying to do a large number of reads at once.
Code of Conduct

No Library Update Requests
No Timeline Requests
Please read through the documentation and FAQs
If you have a complaint about the docs, FIX IT - DO NOT FILE IT HERE!  I won't guarantee everything is always 100% up-to-date since I am just 1 person.
Please be respectful of my time. I put a lot of effort into my libraries.  I try to support the issue requests within a day.  I'm not psychic, so please take the time to fill out the above form as complete as possible

Refresh services is only called when a discovery is requested.  Please include a full reproducible sample of what you are doing.
https://github.com/aritchie/bluetoothle/blob/dev/Plugin.BluetoothLE/Platforms/Android/Device.cs#L102

I added a test method to the sample app in DeviceViewModel like this
    private async void Test()
    {
        <relevant UUIDs defined here>
        for (var i = 0; i < 5; i++)
        {
            var writeChar = await device.GetKnownCharacteristics(serviceUuid, writeCharUuid);
            await writeChar.Write(new byte[] {1});
            var read1Char = await device.GetKnownCharacteristics(differentServiceUuid, read1CharUuid);
            var t1 = await read1Char.Read();
            var read2Char = await device.GetKnownCharacteristics(serviceUuid, read2CharUuid);
            var read3Char = await device.GetKnownCharacteristics(serviceUuid, read3CharUuid);

            var t2 = await read2Char.Read();
            var t3 = await read3Char.Read();
        }

and called from the WhenStatusChanged handler on connect.
I put a break point at the beginning and end of the loop. With RefreshServices enabled, it makes it through the loop once, hanging somewhere between the first and second read of read3Char.
This works just fine with RefreshServices disabled.
RefreshServices is enabled in MainActivity
Hope this helps


[RFC] An ability to refresh a list with the redux

We can give an ability to developers to refresh lists from redux actions. What do you think about it?

Still waiting for the <RefreshButton> component

Sorry for delay. I will try to implement on this weekend.

I prefer the withHandlers version, I think it's better. Do you mind reverting your last commit?
Also, you've broken the refresh button on the Edit page, since it still uses key and doesn't pass the resource to the RefreshButton component. Can you update this component just like you updated List?

Any update on this one?

Sorry, I don't have time. I'll do on this weekend probably.

Any update? You're not very far from something mergeable...

Sorry for late

Maybe it's not the best choice for using global "version" state. Do you have any idea?

I don't understand why you still use key in Edit while it's renamed as version in List.

tests fail :(

I'll check it.

Since we've moved to prettier for code formatting, your PR must now be rebased...

Bump

Sorry, I don't have time to for work with it.


Tooling: add image compression

Originally posted by @mic-max in https://github.com/open-telemetry/opentelemetry.io/issues/869#issuecomment-965749094

Making this part of the PR checks would be cool.
One way to implement it would be something like:

The PR either modifies or creates a PNG
Run a compression program on all the modified and created PNGs
If all the compressed images have the same file sizes in the PR, succeed. Otherwise fail.

Then you'd likely want to add some guidance that before uploading they run the same compression program locally to a README

There is a GitHub app called ImgBot that does this that I recently discovered


Issue 6642 - SysTime should not be hasUnsharedAliasing

http://d.puremagic.com/issues/show_bug.cgi?id=6642

Merged.


CASMINST-5679: Update cray-nls chart version to 1.4.55

Summary and Scope
Removes unused fields from IUF Product Manifest schema.
Issues and Related PRs

Resolves CASMINST-5679

Testing
Tested on:

Local development environment

Test description:
See https://github.com/Cray-HPE/cray-nls/pull/158
Risks and Mitigations
Low risk.
See https://github.com/Cray-HPE/cray-nls/pull/158
Pull Request Checklist

[x] Version number(s) incremented, if applicable
[x] Copyrights updated
[x] License file intact
[x] Target branch correct
[x] CHANGELOG.md updated
[x] Testing is appropriate and complete, if applicable
[x] HPC Product Announcement prepared, if applicable

/backport release/1.4


top_k and repetition_penalty

Can top_k and repetition_penalty be added? Thanks.

It works in batch mode

@deep-diver top_k and repetition_penalty are not applicable for batch mode?

it works now. but batch mode is deprecated for now.


Just a comment

Dude, this is a great tool. Please bring it to 1.0.0.
I would like to leave a few comments:

Make some kind of configuration so that I can define which files I want to check. For example, right now you're looking for ./deps.ts, ./dev_deps.ts and ./import_map.json, but I want to check ./src/**/deps.deno.ts or ./importMap.json, for example.
It would also be nice to make exceptions for some packages. For example, I know for a fact that package X version 1.17.1 is not compatible with package Y version 2.0.4, so I want to prevent automatic updates of package X. Or I want to allow only the deno.land/std packages to be updated.
Fix the fact that you replace links like<EMAIL_ADDRESS>with<EMAIL_ADDRESS>

I was testing this tool alone and I don't have many cases in mind to test it thoroughly, only mine :)
Your observations are noted and I'll take them into account for future improvements. Thank you for your feedback.

Also, note that hyphenated module names (drizzle-orm, for example) are not displayed correctly in the console (in my case, I see only orm without drizzle-)

We also have a suggestion for the table: modules that have a new version available can be displayed in red (any color), so that if there are a large number of packages, you can quickly find the problematic ones.


Update to latest Node to get rid of deprecation warnings

When you run the action, you get the following warning:

Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: cezarypiatek/VsixPublisherAction@759bd2ec02ac1b13dea56d8c84834e517278b5b1

Please update the Node version to latest (LTS) to get rid of this.

Any PR is welcomed ;)

Fixed

Thank you!


Error when renaming file

Hello,
I use the package on windows with go version 1.16. If I execute the following code, I get an error:
config := rollingwriter.NewDefaultConfig()
config.RollingPolicy = rollingwriter.VolumeRolling
config.RollingVolumeSize = "10M"
w, _ := rollingwriter.NewWriterFromConfig(&config)

for i := 0; i < 5000000; i++ {
  _, err := w.Write([]byte(fmt.Sprintf("This is a test %d", i)))
  if err != nil {
    fmt.Println(err)
    break
  }
}

The printed error is: "rename log/log.log log/log.log.202104121537: The process cannot access the file because it is being used by another process"
If I look deeper into the returned error, the inner errors is golang.org/x/sys/windows.ERROR_SHARING_VIOLATION (32).
Is this a known error and is there a workaround?
Thank you and best regards,
Niklas

@niklasfc hi, this is a known bug cause windows is not suit for go atomic api.
dump #11

fixed via #39


Running these samples with application verifier

I had an issue with some code and used application verifier to help me track down the issue.  Then I noticed there was an issue very early on.  The errors are from the full screen example, but they are in all the samples.
I tested even the hello world samples and the same issue was coming up.  I get two stops.
The first is on the line:
if (SUCCEEDED(D3D12CreateDevice(pAdapter, D3D_FEATURE_LEVEL_11_0, _uuidof(ID3D12Device), nullptr)))
the output window says:
VERIFIER STOP<PHONE_NUMBER>000900: pid 0xE98: A heap allocation was leaked.
0000021B71B3AFF0 : Address of the leaked allocation. Run !heap -p -a <address> to get additional information about the allocation.
0000021B6A866FF0 : Address to the allocation stack trace. Run dps <address> to view the allocation stack.
0000021B75492FD8 : Address of the owner dll name. Run du <address> to read the dll name.
00007FFD9D6B0000 : Base of the owner dll. Run .reload <dll_name> = <address> to reload the owner dll. Use 'lm' to get more information about the loaded and unloaded modules.

The second is in the file client.h from the comptr.
ref = temp->Release();
the last line of code before going to DXGI was
D3D12Fullscreen.exe!Microsoft::WRL::ComPtr::InternalRelease() Line 196
the output window says
VERIFIER STOP<PHONE_NUMBER>000303: pid 0xE98: NULL handle passed as parameter. A valid handle must be used.
0000000000000000 : Not used.
0000000000000000 : Not used.
0000000000000000 : Not used.
0000000000000000 : Not used.

Thanks for reporting this issue.  We'll take a look.

Just an update.  The first one is a leak in the debug layer and I filed a bug on that.  I have not been able to reproduce the second one.


DataHub CVE - Vulnerabilities identified

Describe the bug
We put the DataHub components (e.g. Front End React) through a Trivy vulnerability scanner and it identified over 200 High and Critical CVEs. These relate to different parts of DataHub including Spring Framework, Log4j, Jackson-Databind and lib python3.9.
To Reproduce
We used a Trivy scanner against the Datahub components.
Expected behavior
Vulnerability scanning that does not find quite so many CVE and particularly does not find High or Critical CVEs.
Where vulnerabilities are tolerated, some form of mitigation evidence as to why that vulnerability is not applicable or deemed to be a low risk based on the context in which it is operating.
Screenshots
I have a xls document showing all the CVE and each one includes a link to find further information. Examples include https://access.redhat.com/security/cve/CVE-2021-29921 and https://packetstormsecurity.com/files/166713/Spring4Shell-Code-Execution.html.
Additional context
The MITRE CVE page is here: https://cve.mitre.org. Each includes a CVSS which is their standard for rating the severity of vulnerabilities. https://nvd.nist.gov/vuln-metrics/cvss#

Hello @syrilsneer,
Thank you for submitting this ticket!
The core team is working on addressing CVEs, unfortunately we can not address all of them in short amount of time.
Could you share your xls document so that we can be aware of any new vulnerability that we might be missing?
Also, this is exactly the type of thing that contributing to the project would be extremely welcome if you have bandwidth for!

Hello @pedro93 and @shirshanka ,
Please find attached the list of Critical and High CVE that our Trivy tool picked up.
Datahub CVE.xlsx

Thanks for this @syrilsneer : we are looking into these and working through them as you might be seeing from the PRs :)

Hello,
we detected additional CVEs on the dathub image:
Let us know if you have any

Sorry. I skipped by error other related images.
Here you have an update of findings:
datahub-additional-cves-pr4804.csv

I have just ran the latest datahub-ingestion image through trivy and posted the report on this related issue
Hope this helps

Hello,
Commented to keep issue opened


Can't add cards with send to deck command.

markdown file
# Vscode to Anki Test

## How to use Vscod to anki?

don't know

## fdsafdsa ?

???

info log
{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "active Editor..",
  "time": "2020-08-10 13:17:32.838Z"
}
{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "{\"action\":\"createDeck\",\"version\":6,\"params\":{\"deck\":\"notes\"}}",
  "time": "2020-08-10 13:17:32.853Z"
}
{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "{\"action\":\"addNotes\",\"version\":6,\"params\":{\"notes\":[{\"deckName\":\"notes\",\"modelName\":\"BasicWithHighlightVSCode\",\"options\":{\"allowDuplicate\":false,\"duplicateScope\":\"deck\"},\"fields\":{\"Front\":\"
How to use Vscod to anki?
\\n\",\"Back\":\"
don't know
\\n\"},\"tags\":[\"Vscode_to_Anki_Test\"]},{\"deckName\":\"notes\",\"modelName\":\"BasicWithHighlightVSCode\",\"options\":{\"allowDuplicate\":false,\"duplicateScope\":\"deck\"},\"fields\":{\"Front\":\"
fdsafdsa ?
\\n\",\"Back\":\"
wtf
\\n\"},\"tags\":[\"Vscode_to_Anki_Test\"]}]}}",
  "time": "2020-08-10 13:17:33.059Z"
}

@flynngao for now you could try uninstalling the plugin. Reinstall it with dev tools open to see the console output. This should kick it to trying again. Can you paste the output here..
Also can you go into Anki and click Browse, then tell me if you have a model called BasicWithHighlightVSCode

{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "active Editor..",
  "time": "2020-08-10 15:01:57.252Z"
}
{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "{\"action\":\"createDeck\",\"version\":6,\"params\":{\"deck\":\"notes\"}}",
  "time": "2020-08-10 15:01:57.266Z"
}
{
  "label": "Anki for VSCode",
  "level": "info",
  "message": "{\"action\":\"addNotes\",\"version\":6,\"params\":{\"notes\":[{\"deckName\":\"notes\",\"modelName\":\"BasicWithHighlightVSCode\",\"options\":{\"allowDuplicate\":false,\"duplicateScope\":\"deck\"},\"fields\":{\"Front\":\"
How to use Vscod to anki?
\\n\",\"Back\":\"
don't know
\\n\"},\"tags\":[\"Vscode_to_Anki_Test\"]},{\"deckName\":\"notes\",\"modelName\":\"BasicWithHighlightVSCode\",\"options\":{\"allowDuplicate\":false,\"duplicateScope\":\"deck\"},\"fields\":{\"Front\":\"
fdsafdsa ?
\\n\",\"Back\":\"
???
\\n\"},\"tags\":[\"Vscode_to_Anki_Test\"]}]}}",
  "time": "2020-08-10 15:02:00.428Z"
}

No BasicWithHighlightVSCod model

@flynngao which version are you using?
If you've uninstalled and reinstalled it there should be some setup logging involved, im not seeing that in your output.
It doesn't look like it set itself up properly (i don't know why)

version 0.0.2

@flynngao in version 0.1.0 could you try running the force re-install with the output console open and paste here what you get?


suppresses all warnings reported by latest Xcode/AppleClang

There are a bunch of warnings reported by Apple LLVM version 8.0.0 (clang-8<IP_ADDRESS>).  This commit suppresses them,

Thanks for the fixes!