[Extractor Error] Getting extractor error when downloading videos from VIU.

Checklist

[x] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2020.09.20
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.viu.com/ott/sg/en-us/vod/13733/Coffee-Prince', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.3.1-essentials_build-www.gyan.dev, ffprobe git-2019-12-08-9f7b2b3
[debug] Proxy map: {}
[viu:ott] 13733: Downloading video info
[viu:ott] 13733: Downloading stream info
ERROR: An extractor error has occurred. (caused by KeyError('data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\extractor\common.py", line 532, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\extractor\viu.py", line 233, in _real_extract
KeyError: 'data'
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\extractor\common.py", line 532, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\extractor\viu.py", line 233, in _real_extract
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\YoutubeDL.py", line 797, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\extractor\common.py", line 545, in extract
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

This doesn't require any credentials so it is easy to duplicate. I am running Windows 10 and youtube-dl.exe and have begun seeing this error for about 1 week now. I am unsure as to myself doing anything wrong as I am doing the same as I did earlier when it used to work. Any help regarding this issue will be really great.

There's already a fixed for that. But the source code is not yet compiled.
Check this
https://github.com/ytdl-org/youtube-dl/pull/26775
https://github.com/ytdl-org/youtube-dl/pull/26723
https://github.com/ytdl-org/youtube-dl/issues/26701

There's already a fixed for that. But the source code is not yet compiled.
Check this
#26775
#26723
#26701

Thank you for responding, I was unable to find the threads, is there any ETA for the next update, that may fix this?

There's already a fixed for that. But the source code is not yet compiled.
Check this
#26775
#26723
#26701

Thank you for responding, I was unable to find the threads, is there any ETA for the next update, that may fix this?

I don't know. This is not their priorities.

Well, you could try this for now.
Make sure you have python installed on your sytem.

Download the source code and then
Save this as viu.py
https://raw.githubusercontent.com/ytdl-org/youtube-dl/ef8b06469e8dd9783cd05089ed039b958a71e497/youtube_dl/extractor/viu.py

3 Now go to extractor folder from the souce code "C:\bin\youtube-dl-2020.09.06\youtube_dl\extractor" and replace with this updated codes.

Inside this folder: C:\bin\youtube-dl-2020.09.06\youtube_dl
open your cmd and then paste this to go to that dir.

cd C:\bin\youtube-dl-2020.09.06\youtube_dl

Now paste this command on your cmd.
python main.py https://www.viu.com/ott/sg/en-us/vod/13733/Coffee-Prince

Well, you could try this for now.
Make sure you have python installed on your sytem.

Download the latest source code
https://github.com/ytdl-org/youtube-dl/releases/download/2020.09.20/youtube-dl

and then
2. Save this as viu.py (or ask @lkho for his updated code)
https://raw.githubusercontent.com/ytdl-org/youtube-dl/1e0f4c34608c7f5815e3833bf2bcf33cb50c9a7f/youtube_dl/extractor/viu.py
3 Now go to extractor folder from the souce code "C:\bin\youtube-dl-2020.09.06\youtube_dl\extractor" and replace with the updated codes you saved.

Inside this folder: C:\bin\youtube-dl-2020.09.06\youtube_dl
open your cmd and then paste this to go to that dir.

cd C:\bin\youtube-dl-2020.09.06\youtube_dl

Now paste this command on your cmd.
python __main__.py https://www.viu.com/ott/sg/en-us/vod/13733/Coffee-Prince

thank you. it's work but not working for premium videos like this. https://www.viu.com/ott/sg/en-us/vod/299246/Do-You-Like-Brahms
how can i download this link?

Well, you could try this for now.
Make sure you have python installed on your sytem.

Download the latest source code
https://github.com/ytdl-org/youtube-dl/releases/download/2020.09.20/youtube-dl

and then
2. Save this as viu.py (or ask @lkho for his updated code)
https://raw.githubusercontent.com/ytdl-org/youtube-dl/1e0f4c34608c7f5815e3833bf2bcf33cb50c9a7f/youtube_dl/extractor/viu.py
3 Now go to extractor folder from the souce code "C:\bin\youtube-dl-2020.09.06\youtube_dl\extractor" and replace with the updated codes you saved.

Inside this folder: C:\bin\youtube-dl-2020.09.06\youtube_dl
open your cmd and then paste this to go to that dir.

cd C:\bin\youtube-dl-2020.09.06\youtube_dl

Now paste this command on your cmd.
python __main__.py https://www.viu.com/ott/sg/en-us/vod/13733/Coffee-Prince

thank you. it's work but not working for premium videos like this. https://www.viu.com/ott/sg/en-us/vod/299246/Do-You-Like-Brahms
how can i download this link?

That's the changes from Viu. You'll need now premium token. On my personal tool that's what I did.
They're not yet added the premium future.

I tried it now and I get the streams. but you'll need premium account.
Like in this screenshot, the first result, the streams/m3u8 is NOT FOUND and then when I used premium account and get the cookies, I can gett the streams.

Here's what I've noticed @lkho @panda130
For SG url, sometimes you'll need to play first in your browser before trying to use youtube-dl. I don't know why, maybe it's required another cookies.
And in that can case, the ccs api, must include identity into query to get the premium videos correctly.
like this. https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=6855251b4b6fadafd26107dd05a2ca3761fe60b4

@lkho @panda130
Here's the demonstration for sg fix.
watch:
https://www.youtube.com/watch?v=j8kJQeOR1P8

I tried it now in my personal tool and I get the streams. but you'll need premium account.
Like in this screenshot, the first result, the streams/m3u8 is NOT FOUND and then when I used premium account and get the cookies, I can gett the streams.

Here's what I've noticed @lkho @panda130
For SG url, sometimes you'll need to play first in your browser before trying to use youtube-dl. I don't know why, maybe it's required another cookies.
And in that can case, the ccs api, must include identity into query to get the premium videos correctly.
like this. https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=6855251b4b6fadafd26107dd05a2ca3761fe60b4

i havent tool. what do i do? after get premium account how can i download like you?

I tried it now in my personal tool and I get the streams. but you'll need premium account.
Like in this screenshot, the first result, the streams/m3u8 is NOT FOUND and then when I used premium account and get the cookies, I can gett the streams.

Here's what I've noticed @lkho @panda130
For SG url, sometimes you'll need to play first in your browser before trying to use youtube-dl. I don't know why, maybe it's required another cookies.
And in that can case, the ccs api, must include identity into query to get the premium videos correctly.
like this. https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=6855251b4b6fadafd26107dd05a2ca3761fe60b4

i havent tool. what do i do? after get premium account how can i download like you?

Just wait for @lkho or @panda130 to edit the viu to add the identity query.

I tried it now in my personal tool and I get the streams. but you'll need premium account.
Like in this screenshot, the first result, the streams/m3u8 is NOT FOUND and then when I used premium account and get the cookies, I can gett the streams.

Here's what I've noticed @lkho @panda130
For SG url, sometimes you'll need to play first in your browser before trying to use youtube-dl. I don't know why, maybe it's required another cookies.
And in that can case, the ccs api, must include identity into query to get the premium videos correctly.
like this. https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=6855251b4b6fadafd26107dd05a2ca3761fe60b4

i havent tool. what do i do? after get premium account how can i download like you?

Just wait for @lkho or @panda130 to edit the viu to add the identity query.

ok. i wait for them to send here

Also I've noticed, when I changed the country code to PH. even the url is SG,
I can get the streams without using vpn.
URL: https://www.viu.com/ott/sg/en-us/vod/299246/Do-You-Like-Brahms
CountryCode = 5;
LanguageFlagId = 3;
ut = 0;
API:
CCS https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=8dda66eaa5af83badd40a509cf05af92b5009b76
AJAX https://www.viu.com/ott/ph/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=5&language_flag_id=3&product_id=299244&ut=0
Maybe that's the other way you can access the premium videos from your country. Because here's in PH. that given url is not premium that's why I can easily get the streams.

@touchhyuk could you try use hola vpn on chrome and change the country to PH?
and just check if your url is also premium. If not, you have a chance  to get the streams without premium account

Also I've noticed, when I changed the country code to PH. even the url is SG,
I can get the streams without using vpn.
URL: https://www.viu.com/ott/sg/en-us/vod/299246/Do-You-Like-Brahms
CountryCode = 5;
LanguageFlagId = 3;
ut = 0;
API:
CCS https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=8dda66eaa5af83badd40a509cf05af92b5009b76
AJAX https://www.viu.com/ott/ph/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=5&language_flag_id=3&product_id=299244&ut=0
Maybe that's the other way you can access the premium videos from your country. Because here's in PH. that given url is not premium that's why I can easily get the streams.

where input this code? API:
CCS https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=8dda66eaa5af83badd40a509cf05af92b5009b76
AJAX https://www.viu.com/ott/ph/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=5&language_flag_id=3&product_id=299244&ut=0
i can connect ph with express vpn

Also I've noticed, when I changed the country code to PH. even the url is SG,
I can get the streams without using vpn.
URL: https://www.viu.com/ott/sg/en-us/vod/299246/Do-You-Like-Brahms
CountryCode = 5;
LanguageFlagId = 3;
ut = 0;
API:
CCS https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=8dda66eaa5af83badd40a509cf05af92b5009b76
AJAX https://www.viu.com/ott/ph/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=5&language_flag_id=3&product_id=299244&ut=0
Maybe that's the other way you can access the premium videos from your country. Because here's in PH. that given url is not premium that's why I can easily get the streams.

where input this code? API:
CCS https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=49fc785360f200de94d202e94b458c29&language_flag_id=3&identity=8dda66eaa5af83badd40a509cf05af92b5009b76
AJAX https://www.viu.com/ott/ph/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=5&language_flag_id=3&product_id=299244&ut=0
i can connect ph with express vpn

they must update that API from the source code.

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

https://www.viu.com/ott/sg/en-us/vod/300009/Lie-After-Lie

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

https://www.viu.com/ott/sg/en-us/vod/300009/Lie-After-Lie

Here
https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

https://www.viu.com/ott/sg/en-us/vod/300009/Lie-After-Lie

Here. (I used SG API with premium account)
https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ

how can i use this?

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

https://www.viu.com/ott/sg/en-us/vod/300009/Lie-After-Lie

Here. (I used SG API with premium account)
https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ

how can i use this?

here:
youtube-dl --hls-prefer-native --output "OUTPUT FILENAME.mp4" "https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ"

It's already 1080p so you don't need to specify the resolution

or you can change to --hls-prefer--ffmpeg

You can add also this commands:
--hls-use-mpegts
--fixup warn
--no-part
you can previe while downloading

well, for now just give me the url you want to download and I'll give you the m3u8 url and then you can download it via youtube-dl

https://www.viu.com/ott/sg/en-us/vod/300009/Lie-After-Lie

Here. (I used SG API with premium account)
https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ

how can i use this?

here:
youtube-dl --hls-prefer-native --output "OUTPUT FILENAME.mp4" "https://d1k2us671qcoau.cloudfront.net/vodapi/viu_akm.m3u8?vid=c3f3b6132bd78fbaa293d94e61dd6fc8&layer=Layer4&ts=202010042137&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9kMWsydXM2NzFxY29hdS5jbG91ZGZyb250Lm5ldC92b2RhcGkvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYwMTgzMzA1Mn19fV19&Signature=Ui9hBFtBLX-QORfZk1IKD~9clt7-ZlhN7I4mu89IDFpxXdB2iWQ2Y3PlOVo4AIhSYB-p40TSmmqM1hcLdQY7yceuR4YbhTnXsL7~yPBb28xBegQk5OESYmv73esqUcc6hR0V7ggFtaDleM2a9r52YAcS8IEqFmOjyPNlvW2k6R6gXMw4~V1F2v54JtCHFTbBDBPJ2N7r6zv1~~4VTy6A-WfFp97woWr0OBqDUFJeuKBAcmf1p9NeJ6WuJe2dgNblbyLmIRPMDqbhWh0b4qh4lJd344qGa9QTSFFJWyOfvsh3CW7ZDPJjCRictTnUP7xwZb9Dq1svgBOuAH6YEmvl7Q__&Key-Pair-Id=APKAJ6Z4RF5IYK7Y3SQQ"

thank you. it's work fine
how can i get api link with premiunm account?

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

will need premium account after update source code?

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

will need premium account after update source code?

Yeah. but it's only required for the premium vidoes

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

please teach me how get m3u8
i dont know when they update source code

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

please teach me how get m3u8
i dont know when they update source code

Alright. here's the chrome extension that I modified. This is what I'm using before
The video tutorial is included. https://file.io/i4l47P5do8ED

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

please teach me how get m3u8
i dont know when they update source code

Alright. here's the chrome extension that I modified. This is what I'm using before
The video tutorial is included. https://www.upload.ee/files/12352350/cat-catch-1.0.12__Moded_by_zackmark29_.rar.html

thank you. you are best. it's work fine

n i get api link with premiunm account?

Just wait until they update the source code. If I'm just good at python I can compile it for you.
But I only know C#

please teach me how get m3u8
i dont know when they update source code

Alright. here's the chrome extension that I modified. This is what I'm using before
The video tutorial is included. https://www.upload.ee/files/12352350/cat-catch-1.0.12__Moded_by_zackmark29_.rar.html

thank you. you are best. it's work fine

Welcome. Also you could try this much better than "cat catch"
https://github.com/rowrawer/stream-detector
you can customize and copy directly with youtube-dl command

Just wait for @lkho or @panda130 to edit the viu to add the identity query.

I don't have premium account to test for. BTW, in my country (HK), their own website seems to be broken also if the video requires premium.
I also found that, if premium is required, but they offer a few minutes preview (although the website blocks after the preview), I can still get the streams. If the entire video requires premium, their website simply crashed.

@zackmark29 do you have the demo API response of the premium videos so I can try to incorporate the changes?

Just wait for @lkho or @panda130 to edit the viu to add the identity query.

I don't have premium account to test for. BTW, in my country (HK), their own website seems to be broken also if the video requires premium.
I also found that, if premium is required, but they offer a few minutes preview (although the website blocks after the preview), I can still get the streams. If the entire video requires premium, their website simply crashed.

What is your email? I'll send you SG Premium account

@zackmark29 but i don't have SG vpn. maybe you can give me the C# code of that part, I can try to translate from that first.

@zackmark29 but i don't have SG vpn. maybe you can give me the C# code of that part, I can try to translate from that first.

You could you hola vpn in your browser. Like what I'm using also for testing.
But, what parts of codes you want?

you mean the login codes?

Just wait for @lkho or @panda130 to edit the viu to add the identity query.

I don't have premium account to test for. BTW, in my country (HK), their own website seems to be broken also if the video requires premium.
I also found that, if premium is required, but they offer a few minutes preview (although the website blocks after the preview), I can still get the streams. If the entire video requires premium, their website simply crashed.

我都係香港人, 第一次用呢啲嘢, 想問下viutv用唔用到?

@wingcwf the official release don't, but my customized fork can. you can try this, but requires you to install Python locally:
https://github.com/lkho/youtube-dl/tree/pr/viutv

@wingcwf the official release don't, but my customized fork can. you can try this, but requires you to install Python locally:
https://github.com/lkho/youtube-dl/tree/pr/viutv

有冇得send個中文教學睇睇 A_>A

@wingcwf the official release don't, but my customized fork can. you can try this, but requires you to install Python locally:
https://github.com/lkho/youtube-dl/tree/pr/viutv

已經裝好同埋試用完原版嗰個, 但係你個個要點用?_?

你要自己裝python，用python黎run個script，因為我地無build到exe出黎

你要自己裝python，用python黎run個script，因為我地無build到exe出黎
https://github.com/ytdl-org/youtube-dl#developer-instructions

師兄有冇得拍條片嚟睇下, 方便快捷, 費事不斷問打攪你

咁你要等下了我現在沒電腦在身

咁你要等下了我現在沒電腦在身

係咪咁樣, 之後要點(咁佢兩下會自動閂)

咁你要等下了我現在沒電腦在身

我諗我應該差唔多成功

咁你要等下了我現在沒電腦在身

都係唔得-.- MPEG DASH

@wingcwf 我剛剛試了一下viutv網站最新的api，發覺他們已經加了DRM protection. 所以應該是不能再下載了

@wingcwf 我剛剛試了一下viutv網站最新的api，發覺他們已經加了DRM protection. 所以應該是不能再下載了

Oh no... 真係冇其他辦法咩😭

you mean the login codes?

please guide me for get videos from iqiyi with youtube-dl
i found this https://github.com/ytdl-org/youtube-dl/issues/16852
but cant run

you mean the login codes?

please guide me for get videos from iqiyi with youtube-dl
i found this #16852
but cant run

@zackmark29 maybe you can give me the C# code that you used for login and get the identity token, I can translate it to python.

This issue, was heavily discussed I see. I was unable to join in the discussion due to some personal issues, but I would like to know if we were able to get a workaround for this.

@zackmark29 maybe you can give me the C# code that you used for login and get the identity token, I can translate it to python.

Alright. I'll email you later.

@zackmark29 maybe you can give me the C# code that you used for login and get the identity token, I can translate it to python.

What is your email? I'll send you the project file
I already made the codes for login and getting streams.

@AlecHex Thanks for posting. We figured out how to get the non-premium Viu streams for Singapore in https://github.com/ytdl-org/youtube-dl/pull/26775. However, we have been unable to access the Viu streams for Thailand and other countries. It seems like the &ut part of the URL may have something to do with it. Also, access to premium videos is completely broken and no workaround has been found so far.

@AlecHex Thanks for posting. We figured out how to get the non-premium Viu streams for Singapore in #26775. However, we have been unable to access the Viu streams for Thailand and other countries. It seems like the &ut part of the URL may have something to do with it. Also, access to premium videos is completely broken and no workaround has been found so far.

I'll send to @lkho the C# source code when his online already. it might help you fix the premium videos.
I'll send here the video of how it's work

@AlecHex Thanks for posting. We figured out how to get the non-premium Viu streams for Singapore in https://github.com/ytdl-org/youtube-dl/pull/26775. However, we have been unable to access the Viu streams for Thailand and other countries. It seems like the &ut part of the URL may have something to do with it. Also, access to premium videos is completely broken and no workaround has been found so far.

Ah that is a little good news to hear. I am in Thailand, and I have never had any luck with Thailand VIU streams due to I think the change of language or something, but I did use Singapore all the time and that worked flawlessly.

@lkho and @panda130 here's the demo for the premium videos
https://www.youtube.com/watch?v=F_NBVsB2_Es&feature=youtu.be

@zackmark29<EMAIL_ADDRESS>

@zackmark29 maybe you can give me the C# code that you used for login and get the identity token, I can translate it to python.

What is your email? I'll send you the project file
I already made the codes for login and getting streams.

can you send for my email too?

<EMAIL_ADDRESS>
already sent

@zackmark29 maybe you can give me the C# code that you used for login and get the identity token, I can translate it to python.

What is your email? I'll send you the project file
I already made the codes for login and getting streams.

can you send for my email too?

You can wait for to release the final build. I'll just sending him the source code that might help a little to resolve the premium videos

already sent

received. thanks

already sent

received. thanks

I have a little changes on the subscription. I'll send you the new code

already sent

received. thanks

Already sent the updated code

@zackmark29 @panda130 from my observations:
ut query param is user_level, retrieved from GET /ott/<area>/index.php?r=user/info response: data.user.user_level

when anonymous, data is null
when logged in, data.user.user_level is integer > 0
this user_level field is also referenced in GET /ott/sg/index.php?r=vod/ajax-detail&product_id=<id>&ut=<user_level> response: data.current_product.user_level

for non-premium videos, data.current_product.user_level is 0, for premium videos, it is > 0

@zackmark29 @panda130 from my observations:
ut query param is user_level, retrieved from GET /ott/<area>/index.php?r=user/info response: data.user.user_level

when anonymous, data is null

when logged in, data.user.user_level is integer > 0

this user_level field is also referenced in GET /ott/sg/index.php?r=vod/ajax-detail&product_id=<id>&ut=<user_level> response: data.current_product.user_level

for non-premium videos, data.current_product.user_level is 0, for premium videos, it is > 0

Oh I see. thanks for that.
By the way, In my case. I don't think so if I still needed to login to get the token because I just found another way to download the premium videos with the preview but I just tried for ph url only.
I get the final m3u8 url and then replace the duration=180 from the url with duration=2000. and I get the original m3u8.
I thought the preview is only a trailer but still it's only have the limited duration.

By the way. I accidentally logout the account.  You may login again

something like this
         ```
	bool IsPreview = false;
	string ccs_api = "https://d1k2us671qcoau.cloudfront.net/distribute_web_{COUNTRY_CODE}.php?ccs_product_id={ccs_product_id}&language_flag_id={LANG_FLAG}";
	var duration_param = "&duration=180&duration_start=0";

bypass: var api_ccs = IsPreview ? ccs_api + duration_param : ccs_api;
	using (var doc = await ApiClient.ParseJson(api_ccs, HttpMethod.Get, cts).ConfigureAwait(false)) {
		var elem = doc.RootElement;
		var message = elem.Get("status")?.Get("message")?.GetString();

		if (message == "Access violated") {
		IsPreview = true; goto bypass; }

@lkho
i found the another solution for the sg url. we don't have now needed to play from the  browser first.
We can add the headers : request.Headers.Add("X-Forwarded-For", "<IP_ADDRESS>");

I get the final m3u8 url and then replace the duration=180 from the url with duration=2000. and I get the original m3u8.

this is what I did yesterday. but I think they will eventually block this also. so we better add the identity token properly to support premium videos. as previously I found that there are some videos that don't even allow previews.

I get the final m3u8 url and then replace the duration=180 from the url with duration=2000. and I get the original m3u8.

this is what I did yesterday. but I think they will eventually block this also. so we better add the identity token properly to support premium videos. as previously I found that there are some videos that don't even allow previews.

Yeah. I just did not remove the option to login just in case there will be an error.

I get the final m3u8 url and then replace the duration=180 from the url with duration=2000. and I get the original m3u8.

this is what I did yesterday. but I think they will eventually block this also. so we better add the identity token properly to support premium videos. as previously I found that there are some videos that don't even allow previews.

Do you still have the history of url that doesn't have previews? I'd like to see

Do you still have the history from your browser the url that doesn't have previews? I'd like to see

I can't find it currently, maybe they fixed the bug after I tried many times last week :P
https://github.com/ytdl-org/youtube-dl/issues/26788#issuecomment-703525961

ok I've finished the login function.
@zackmark29 @touchhyuk you can try this branch
https://github.com/lkho/youtube-dl/tree/pr/viuott
however, @zackmark29 I can't reproduce the "needed to play from the browser first" scenario. I have always succeeded regardless of playing in browser or not.

ok I've finished the login function.
@zackmark29 @touchhyuk you can try this branch
https://github.com/lkho/youtube-dl/tree/pr/viuott
however, @zackmark29 I can't reproduce the "needed to play from the browser first" scenario. I have always succeeded regardless of playing in browser or not.

Oh nice!
Yeah maybe in your country it doesn't needed.
I think only with PH country. but I resolved it already without playing first by adding the header X-Forward-For
By the way, I just see that you also have a PR for the changing the duration.
We can also get the exact duration by getting the from ['data']['current_product']['time_duration']

@zackmark29 i tested in both HK and SG, can't reproduce.

@zackmark29 i tested in both HK and SG, can't reproduce.

can you provide the url I'll just gonna test

i tested with many of the premium videos i can found on the website (SG), all can download successfully

i tested with many of the premium videos i can found on the website (SG), all can download successfully
https://lkho.me/dl/private/github_oq8nm40fpz9al/2020-10-12 00-45-19.mp4

Yeah I think only in PH that needed to play first in the browser (but this is not a problem anymore. I already fixed)
In your video, did you still add the bypass for the duration?

In your video, did you still add the bypass for the duration?

yes.

In your video, did you still add the bypass for the duration?

yes.

Ah alright. so we already fixed everything.
Yeah me too. I don't need to login anymore haha hopefully the bypass will always work

this is how it looks like if I don't use the duration hack:
https://lkho.me/dl/private/github_oq8nm40fpz9al/2020-10-12 01-24-17.mp4

this is how it looks like if I don't use the duration hack:
https://lkho.me/dl/private/github_oq8nm40fpz9al/2020-10-12 01-24-17.mp4

what does --netrc?

because i don't want to show the credentials on screen, so i put it in a separate file. it's just the same as putting it in the command args.
see https://github.com/ytdl-org/youtube-dl#authentication-with-netrc-file

because i don't want to show the credentials on screen, so i put it in a separate file. it's just the same as putting it in the command args.
see https://github.com/ytdl-org/youtube-dl#authentication-with-netrc-file

haha ah Ok. by the way. you can keep the credential that I included in the project file.
I have a back up accounts

The credentials will only work in that specific country, for me it is no use if I use it in HK 😂

The credentials will only work in that specific country, for me it is no use if I use it in HK 😂. unless you will always VPN to SG first when visiting viu.com.

Yeah only works for SG. it's only useful for your for testing purpose =D

I am still so out of the loop. Damn sorry y'all

I am still so out of the loop. Damn sorry y'all

are you still having problem with downloading?

There's already a fixed for that. But the source code is not yet compiled.
Check this
#26775
#26723
#26701

@zackmark29 @lkho
Hi guys, I see a lot of discussion after this comment. Does this still stand? I.e. is the fix ready just pending compilation of a new version? Or are you guys working on newly discovered related bugs?
If it is still merely awaiting compilation, any idea which of the contributors is coordinating that? Will be happy to lend a voice to urge action.

the corresponding PR is here https://github.com/ytdl-org/youtube-dl/pull/26775

There's already a fixed for that. But the source code is not yet compiled.
Check this
#26775
#26723
#26701

@zackmark29 @lkho
Hi guys, I see a lot of discussion after this comment. Does this still stand? I.e. is the fix ready just pending compilation of a new version? Or are you guys working on newly discovered related bugs?
If it is still merely awaiting compilation, any idea which of the contributors is coordinating that? Will be happy to lend a voice to urge action.

It's already fixed the bugs. I think it's just pending for compilation.

Just hoping to see a new update soon. I am not so good with technical stuff :(

Now that YT-DL is back and restored, what is the status for this?

I am still so out of the loop. Damn sorry y'all

are you still having problem with downloading?

Yes I still am :(

Hi AlexHex, (This comment has nothing to do with this issue)
I have seen you for this issue:
https://github.com/ytdl-org/youtube-dl/issues/24754
Did you find a way to download the Viki videos at 720p ?, I would really appreciate if you could tell me
When I try to download with credentials / login
I get an error "HTTP Error 404: Not Found (caused by HTTPError 404: 'Not Found'").
Thank you.


CSHARP-806 Drop netstandard1.5 and bump net45 to net452

Notable changes are the addition of json converters to graph and geometry types. Instead of these converters we could bump newtonsoft from 9.0.1 to 10.0.3 (where newtonsoft supports ISerializable in .net core) but I figured this could be annoying for OSS users (or even dse users that don't use graph) so the json converter does the trick without having to bump the version.
I enabled the graph serialization unit tests in .net core and added an Assert for json deserialize (existing tests only tested serialize). Also added these tests for GraphNode with GraphSON2 (previously had a test with GraphSON1 only).

Oh and I haven't changed the matrix yet because we have a couple of unit tests that depend on the runtime. I'm thinking of leaving the matrix unchanged for appveyor and travis but remove netcoreapp2.0 (which targeted driver's netstandard1.5 before this PR) integration tests from jenkins.
I'll also take this opportunity to remove DSE + Windows tests from jenkins but keep OSS + Windows


Proto2 and proto3 cannot be mixed up, or the model cannot be transformed

Not sure I understand the issue.
Can you give an example of what doesn't work?

It's unclear what the problem is and the author has been unresponsive. Closing.


Several bugfixes  and refactoring of dynamic generic reduction

This P.R fixes several visible bugs in dynamic generic reduction, and commits are also added to refactor the codes following the review from 54615.
Here are my explanations about three commits

2f4fe706435ea8918d0fda78587e60270dfec192 fixes  the issue that one passed type parameter of the template is not consistent with the dst buffer data type (compType) while using BlockWise and ThreadWise transfer
f098bfbe4cf415500f79509a8d9d3eedc7b361dc fixes the issue that GetZeroVal() is not correctly implemented with some binary operator on half type on both the kernel and host
ba91b9918e5d78155b9764f22998584b76f3e79f fixes the issue in the kernel that value of dstDataType is converted to compType before writing out, this could cause losing of precision when srcDataType and compType are both half, while the dstDataType is float,  even though this situation might not be common,  miopenReduceTensor() does support it

The commits have passed the following testing locally on MI100
bin/test_reduce_test --all
bin/test_reduce_test --all --half
bin/test_reduce_test --all --double
MIOpenDriver script 1 for non-indexable operations
#!/bin/bash

PRECISION=reduce  ## reducefp64 reducefp16

if test -n $PRECISION && test "$PRECISION" = "reducefp16"; then
   CTYPE="-C 1"
else
   CTYPE=""
fi

if [ $# -ge 1 ] ; then
    NREPEAT=$1
else
    NREPEAT=1
fi

for op in 0 5 6 7; do
    set -x
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 2 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,2 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 2,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,2 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,2 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,2,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,2,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,2,3 -O $op $CTYPE  -t 1 -i $NREPEAT
    set +x
done

MIOpenDriver script 2 for indexable operations
#!/bin/bash

PRECISION=reducefp16    ## reducefp64 reduce

if test -n $PRECISION && test "$PRECISION" = "reducefp16"; then
   CTYPE="-C 0"
else
   CTYPE="-C 1"
fi

if [ $# -ge 1 ] ; then
    NREPEAT=$1
else
    NREPEAT=1
fi

for op in 2 3 4; do
    for use_idx in 0 1; do
        set -x
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 2 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,2 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 2,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,2 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,2 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 1,2,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,2,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        bin/MIOpenDriver $PRECISION -D 64,3,280,81 -R 0,1,2,3 -O $op -I $use_idx $CTYPE -t 1 -i $NREPEAT
        set +x
    done
done

@qianfengz Thank you! Does this PR resolve some known SWDEV or github issue?

@qianfengz Any performance impact?

@qianfengz And last but not least -- shall we also add tests to cover the new functionality (i.e. improved precision)?

@qianfengz Thank you! Does this PR resolve some known SWDEV or github issue?

No.   While responding to SWDEV-302129,  I reviewed my codes and found some issues, but these commits are actually unrelated to SWDEV-302129.

@qianfengz Any performance impact?

Nothing to do with performance

@qianfengz And last but not least -- shall we also add tests to cover the new functionality (i.e. improved precision)?

Some situations could not be easily covered by CTEST  since for test_reduce_test,   our implementation assumes the srcDataType and dstDataType are the same,   and also test_reduce_test --all can only cover those cases that srcDataType is the same as compType.   To test those cases where srcDataType is half, compDataType is float ,  we have to use test_reduce_test  or MIOpenDriver reducefp16 manually with specific options.

Here are my explanations about three commits

commit ec192 fixes  the issue that one passed type parameter of the template is not consistent with the dst buffer data type (compType) while using BlockWise and ThreadWise transfer
commit 361dc fixes the issue that GetZeroVal() is not correctly implemented with some binary operator on half type on both the kernel and host
commit 3e79f fixes the issue in the kernel that value of dstDataType is converted to compType before writing out, this could cause losing of precision when srcDataType and compType are both half, while the dstDataType is float,  even though this situation might not be common,  miopenReduceTensor() does support it

[off-topic] @qianfengz Thanks for very clear explanations! I will update the PR description with these.
BTW you can do such updates yourself in the future. Refining a PR description is a normal and useful practice. Let's keep a description of what exactly was done in exactly one place.

There will be many commits in next several days

Commit 21cbb is a big commit,   it does the following

Re-order the input tensor into sequences of [invariant dimensions,  toReduce dimensions] and pass re-ordered lengths/strides to the kernel.  This can reduce the number of kernels to be considered
Split each second_call kernel wrapper file into two files of reduce_all_dims and reduce_partial_dims respectively
Remove unused tensor transformations in all reduce_all kernel wrappers.

@qianfengz And last but not least -- shall we also add tests to cover the new functionality (i.e. improved precision)?

Some situations could not be easily covered by CTEST since for test_reduce_test, our implementation assumes the srcDataType and dstDataType are the same, and also test_reduce_test --all can only cover those cases that srcDataType is the same as compType. To test those cases where srcDataType is half, compDataType is float , we have to use test_reduce_test or MIOpenDriver reducefp16 manually with specific options.

Some Correction to the above comments:

For both MIOpenDriver reduce<xxx>  and test_reduce_test,   the srcDataType and dstDataType are assumed to be same
For test_reduce_test --all --half,   MIN/MAX/AMAX still uses half as compType, while ADD/AVG/MUL/NORM1/NORM2 use float as compType, the reason is that ADD/AVG/MUL/NORM1/NORM2 could lead to overflow during the accumulating

Some situations could not be easily covered by CTEST ...   To test those cases where srcDataType is half, compDataType is float, we have to use test_reduce_test  or MIOpenDriver reducefp16 manually with specific options.

Let's add a custom test to test/CMakeLists.txt. If it is difficult for you to create it, then just give me a list of configs for reduce_test (not for the driver), and I will do it for you, right in this thread. Thanks!

@qianfengz And last but not least -- shall we also add tests to cover the new functionality (i.e. improved precision)?

No need.    Since

Issued fixed by Commit ec192  only occurred with codes implementing indiced operation (op = MIN/MAX/AMAX and --I 1) and when srcDataType/dstDataType is half, compType is float.  For example bin/MIOpenDriver reducefp16 -D 64,3,280,81 -R 0 -O 2 -I 1 -C 1 -t 1  can arouse the issue,  but we have no reason to use -C when the op is MIN/MAX/AMAX for half
Issue fixed by commit b361dc can only be reproduced if we either the host or the kernel has the issue, both not both of them.
Issued fixed by commit 3e79f can only occur if dstDataType has higher precision than the compType.  But with MIOpenDriver and test_reduce_all, we all assume dstDataType and srcDataType are same,  and compType should have not lower precision than srcDataType.   And so far,  we regard  dstDataType different than srcDataType as unusual situation.

@qianfengz Okay.

There will be many commits in next several days

Is PR ready for review/testing/merge?

@qianfengz Okay.

There will be many commits in next several days

Is PR ready for review/testing/merge?

Yes, please

I recommend improving clarity of the code. For example, adding explanations for all the non-trivial literals in a program.
@qianfengz It's up to you. I will approve this now.

I recommend improving clarity of the code. For example, adding explanations for all the non-trivial literals in a program.
@qianfengz It's up to you. I will approve this now.

Thanks.   I will add some comments in next P.R

I recommend improving clarity of the code. For example, adding explanations for all the non-trivial literals in a program.
@qianfengz It's up to you. I will approve this now.

Thanks. I will add some comments in next P.R

Let's wrap up this PR for now :) Priority has shifted and we can revisit reduction only after the urgent task.


Publish on WP.org

It would be nice to have it on WP.org to be able auto update the plugin.
Or at least composer support.

Yes, definitely!
I'm working towards publishing it on WP, hoping to get it ready in a week or so

I simply want to test it some more, before publishing.

Ehm, this ticket belongs to the Wordpress version of WebPConvert...
https://github.com/rosell-dk/webp-express

Just created the issue in WebPExpress:
https://github.com/rosell-dk/webp-express/issues/2

oops, thanks :)


Trace which collection(s) cause a lock timeout

In case of lock timeouts, it would be useful to know which collection(s) are causing the timeout.

Add some diagnostic log messages in the devel branch.
These will print the involved collections in case of a lock timeout or a deadlock.

Any chance to have this in 3.1.18?

Sorry, no. The code around this patch has moved a lot in the rise of the rocksdb engine. It will become available with the beta release in near future.

3.2 alpha 4 containing this bugfix is available for download now.


call?.AnswerAsync(mediaSession).ForgetAndLogExceptionAsync

Describe the issue
Error in bot configuration still allows the call to get accepted and the HeartbeatHandler continues to ping on a call that doesn't exist.
Code Snippet
Bot.cs: call?.AnswerAsync(mediaSession).ForgetAndLogExceptionAsync(
call.GraphLogger,
$"Answering call {call.Id} with scenario {call.ScenarioId}.");
Expected behavior
Say the Bot is missing some permissions. When the call comes in, AnswerAsync fails with an error like this:
Code: 7504
Message: Insufficient enterprise tenant permissions, cannot access this API.
However, since we swallow the exception the PolicyRecordingBot continues to pretend as if a call exists and the HeartBeatHandler continues to try to find the missing call resulting in a series of errors like this:
Code: 8522
Message: Call not found.

+1...........I'm experiencing this problem too (i think) and have raised an issue here.  I too noted the spurious Code:8522 errors from the GraphLogger.


Run (go vet) on all subpackages instead only of changed files

Apparently, it was never documented to use go vet $somefile.go (but go tool vet $somefile.go was).
go 1.10 seems to do more checks within packages, and $somefile.go is interpreted as a package with only that file (even if other files from that package are in the same directory), leading to spurious "undefined: $symbol" errors.
So, just run go vet on ./... (explicitly excluding skopeo/vendor for the benefit of Go 1.8). We only have three subpackages, so the savings, if any, from running go vet only on the modified subpackages would be small.
More importantly, on a toolchain update, ./... allows us to see the newly detected issues all at once, instead of randomly waiting for a commit that changes one of the affected files for the failure to show up.

:+1:


Bindings not found

First build of the application gives.
Severity	Code	Description	Project	File	Line	Suppression State
Error	CS1061	'MainPage' does not contain a definition for 'Bindings' and no accessible extension method 'Bindings' accepting a first argument of type 'MainPage' could be found (are you missing a using directive or an assembly reference?)	PhotoEditor	C:\Projects\WindowsAppSDK-Samples-main\WindowsAppSDK-Samples-main\Samples\PhotoEditor\cs-winui\obj\x64\Debug\net6.0-windows10.0.19041.0\win10-x64\MainPage.g.cs	1066	Active

We've just updated our dependencies and references to WindowsAppSDK 1.2 preview in preparation for the 1.2 stable release.  Can you confirm that the issue still exists on a current tree?  If so, please include a binlog and we'll take a look.

If you are still seeing this on 1.4, please let us know.


[TUBEMQ-231] Update URL link to contribution documents

Update link to new contribution docs.

Thanks @gosonzhang for reviewing and merging this.


feat(#276): add j prefix for java names

Add j$ prefix for java name to avoid naming collisions with EO reserved words.
Closes: #276.

History:

feat(#276): add JavaName and corresponding test for it
feat(#276): finish with raw implementation of JavaName
feat(#276): try to apply JavaName
feat(#276): try to fix some tests
feat(#276): handle constructors
feat(#276): fix all qulice suggestions

PR-Codex overview
This PR focuses on introducing a JavaName class to handle encoding and decoding of Java names.
Detailed summary

Added JavaName class to handle encoding and decoding of Java names.
Updated code references to use JavaName for encoding and decoding.

The following files were skipped due to too many changes: src/main/java/org/eolang/jeo/representation/JavaName.java

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

@rultor merge

@rultor merge

@volodya-lombrozo OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@volodya-lombrozo Done! FYI, the full log is here (took me 4min)

@rultor release, tag is 0.2.3, title is 0.2.3.

@volodya-lombrozo Done! FYI, the full log is here (took me 8min)


[DRAFT] Handle Pipewire Write Function

This is a draft for the write() method. I open this PR to review the pipewire backend implementation of write.

Closing, not needed anymore.


Alphabetise examples

To make things easier to find, and help avoid merge conflicts when new examples are added.

Thanks @hugovk, could you resolve the conflicts?
I'll try to be prompt about merging this once those are resolved so that more conflicts don't come up.

Done!


Ionic 2: Inappropriate behaviour when dismissing alert which is on top of a modal

I have a modal for editing or deleting item. When user want to delete the item, I show a confirm alert asking if user is sure about the delete.
Once user click "yes", I call the delete function and this.viewCtrl.dismiss(). Then the confirm alert dismissed. However, the modal is still there. And user have to click twice on the screen so that the interaction with the UI will recover. Maybe there is a transparent layout when dismissing the alert.

Hello, thanks for opening an issue with us! Would you be able to provide a plunker that demonstrates this issue? Thanks for using Ionic!

Are you dismissing both the Alert and the Modal?  Both are separate view controllers pushed on to the stack.
Thanks,
Dan

Hello all! As it seems it has been a while since there was any activity on this issue i will be closing it for now. Feel free to comment if you are still running into this issue. Thanks for using Ionic!


Stream Support

Hi.
Can you add an option to prevent a user from receive a shared media when you click you show all?
This is useful when you stream one user and don't want to shared image to show for the stream user.

tks

Hello @brunocalado
Can you be more specific please?
It should already be the case. There should be two buttons, "share to all players" and "share to some players".

Also, the window in the screenshot is not from share-media.

Yes. But, it's same from share media button.

If I share the embbed media all connected users will receive this imagem. But, there is one user used for stream that should not receive it.
I can just unselect it every time. But, I wonder if you can't make this easier.

Examples

Create an option in settings to always avoid the user
Let you pick a default for the check box

I'll have to think about that.
Out of curiosity, why don't you want your stream to show medias?

When make a safe stream you can use art without permission
When I can: It's annoying to go to OBS and remove the image from there. The foundry runs inside OBS, I'm not recording a browser window.

@brunocalado a blacklist feature is now live with 0.0.23

@brunocalado a blacklist feature is now live with 0.0.23

Thank you.


Internal Server Error while trying to upload image on post

Hi,
I am getting Internal Server Error while trying to upload image for one of the post, it was working fine earlier, but today I pulled latest changes from camaleon_cms repository, not sure if latest changes are cuasing problem, but now I am getting this error if I try to upload image.
Here is the exception.
 Hence closing the issue. Thanks.

yes, someone suggested to have a place where to change the upload directory name:
use site.slug instead of site.id like actually.
regards,

Sure, I will change as you suggested if required, but for now I am using as it is. Thanks.


[MINOR][SS][DOCS] Update doc for streaming deduplication

What changes were proposed in this pull request?

This patch fixes an error about streaming dedupliaction is Structured Streaming, and also updates an item about unsupported operation.
Why are the changes needed?

Update the user document.
Does this PR introduce any user-facing change?

No. It's a doc only change.
How was this patch tested?

Doc only change.

Today I looked for streaming deduplication in the document for customer question, and noticed an error. Also took the chance to add an item of unsupported operation.

cc @HeartSaVioR @xuanyuanking

Test build #142679 has started for PR 33801 at commit dda6d19.

Test build #142679 has finished for PR 33801 at commit dda6d19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142679/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47181/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47181/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47181/

Thanks @xuanyuanking @HyukjinKwon !

Merging to master/3.2/3.1.


SOLAS V2

After quite a bit of playtesting, there's some pretty drastic need of new boons - ones that cater to the central ideas of the 12 Dungeons and Dragons 5e classes. As a side effect, it also means restructuring how spells work, specifically how players acquire them.

Update concepts will be included in the preview document before being added. It's mainly a place to organize my own thoughts and ideas, but it can give some clarity to these changes before they're implemented.


Navigation click buttons ouside of the carousel

Hello,
I don't know if it was already asked (I've searched throw the issues and read the doc) but is there an option for creating custom navigation click buttons outside of the carousel ?
As you can see on the screenshot, I would like to remove navigation click buttons from both side of the carousel and put them outside on the top.

Thank you.
Ok I've just changed the navigation button position with CSS, sorry for this issue.

<carousel ref="car">
...
</carousel>
..
<script>
methods: {
nextPage(){
this.$refs.car.handleNavigation('forward');
}
</script>`


[INFRA-536][INFRA-1323] Implement voting in board election

This PR introduce following changes:

Add a view to display elections history
Add a view to display one election result
Store one user vote in one file INFRA-1323
Only allow a one-year-old user account to vote
Use single transferable vote to elect candidates

Add two following parameters:

Seniority: Define in month how old a user account need to be created before next election
Seats: Define number of seat for next election

Single Transferable Vote
The backend used this project on Github lourisda/stv.git
Test Environment
In order to test this PR, you can run the following command
make clean build fake: to deploy the accountapp locally with election enable and fake data
then go to http://localhost:8080 with one of the following user/password (alice/password) or (kohsuke/password)
Testing scenarios:
User kohsuke must be able to:

Vote
Generate old election result
Access old election result (if generated by an admin)

User kohsuke can't be allow to:

Generate current election result
Access current election result (if generated by an admin)

User alice must be able to:

Vote
Access old election result (if generated by an admin)

User alice can't be able to:

Generate election result
Generate current election result
Access current election result

USER/Password:

alice/password
kohsuke/password

@olblak I set about testing this while riding the bus, and then realized I needed to pull some docker images :man_facepalming:
Will have to take a look at this once I'm back on good internet :frowning:


[core] Add compression for ExternalBuffer.

External buffer spill could add compression.
This will compress bytes and reduce memory copy while read.
Choose lz4, reason:

lz4 is faster than zip zstd snappy， write spill need speed
lz4 compression ratio is acceptable

see https://indico.fnal.gov/event/16264/contributions/36466/attachments/22610/28037/Zstd__LZ4.pdf

+1


feat: add sherpa dist for android aarch64

Todo:

Fix bindgen creation error https://github.com/rust-lang/rust-bindgen/issues/1780

Failed to generate bindings: ClangDiagnostic("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:1022:2: error: Unsupported architecture

Related: https://github.com/rust-lang/rust-bindgen/issues/1229

Failed to run on Android arm64
Missing libpthread.so.0
ls
libcargs.so  libonnxruntime.so  libsherpa-onnx-c-api.so  libsherpa-onnx-cxx-api.so  libsherpa_rs.so  tts  tts-0498663e598a2018  tts-0498663e598a2018.d  tts.d
surya:/data/local/tmp/examples $ LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH" ./tts
CANNOT LINK EXECUTABLE "./tts": library "libpthread.so.0" not found: needed by /data/local/tmp/examples/libsherpa-onnx-c-api.so in namespace (default)
1|surya:/data/local/tmp/examples $

is there release for Android for the C API of sherpa-onnx?

Please use the script from sherpa-onnx to build it by yourself.
Hint: set
https://github.com/k2-fsa/sherpa-onnx/blob/be159f943ed83c607940958c654bde1c5fa82bec/build-android-arm64-v8a.sh#L97
to
ON

I suggest this repo only does one thing, that is, provide rust API support for sherpa-onnx.


Changed ip address for backend servers

Changed the ip address for the backend servers and the description for Back-end server pool, since the ip addresses must be in a different subnet for application gateway resource.

Hi @opentodonet, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;

@joaoma & @cherylmc - Could you review this app gateway update in the public repo and let me know if I can merge?

@joaoma, please take a look.
From: Tyson Nevil<EMAIL_ADDRESS>Sent: Thursday, March 17, 2016 4:25 PM
To: Azure/azure-content<EMAIL_ADDRESS>Cc: Cheryl McGuire<EMAIL_ADDRESS>Subject: Re: [azure-content] Changed ip address for backend servers (#5978)
@joaomahttps://github.com/joaoma & @cherylmchttps://github.com/cherylmc - Could you review this app gateway update in the public repo and let me know if I can merge?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com/Azure/azure-content/pull/5978#issuecomment-198128779

#sign-off

@opentodonet - Thanks for the update!


Minor grammatical fix in api_extra.rst

I don't see a grammatical problem. Can you point to a source saying why this expression is incorrect?

Sure:
https://www.grammarphobia.com/blog/2013/03/whether-or-not.html

Pat discusses this in her grammar and usage book Woe Is I. Here’s the passage:
“When you’re talking about a choice between alternatives, use whether: Richie didn’t know whether he should wear the blue suit or the green one. The giveaway is the presence of or between the alternatives. But when there’s a whether or not choice (Richie wondered whether or not he should wear his green checked suit), you can usually drop the or not and use either whether or if: Richie wondered if [or whether] he should wear his green checked suit. You’ll need or not, however, if your meaning is ‘regardless of whether’: Richie wanted to wear the green one, whether it had a gravy stain or not. (Or, if you prefer, whether or not it had a gravy stain.)”

https://dictionary.cambridge.org/dictionary/english/whether-or-not

used for saying that it is not important which of two possibilities is true

The "whether or not" phrasing, at least to me, means:

either that it doesn't matter which case it is, which is not right in the corrected sentence,
or that the two possibilities are logical opposites (with only one of them written down), which is also not the case.

The corrected sentence presents two alternatives (self on the left side vs. self on the right side), in which case a simple "whether" is more appropriate in my opinion.
Of course, one could argue that left and right are logical opposites, but in that sentence, both using "whether or not", and then spelling out the two alternatives, comes across as strange to me.
It would also be fine as "Depending on whether or not self is the left argument of a binary operation", but I think the other variant fits better here.

Got it, that makes sense -- thank you. Could you rebase the PR based on master? Looks like it bundles some unrelated changes.

Thanks!


fix: Bump kapeta deps

Hmm, this only bumps devDeps, so none of this should affect downstream packages. Is this the intended change/non-change?

Ah good point about it being devDependencies.
I found out bumping this was not necessary for my issue right now. I had old versions in package-lock.json in ui-web-components that was causing conflicting versions, and therefore I decided to try to bump all to latest. I have changed it to "chore" commits instead so a new version will not be released by merging it.


ci: Use sccache instead of rust-cache

Performance is about the same, but this is consistent with tket2's CI worflows.

I'm not sure that this is working.
The documentation says that you need export RUSTC_WRAPPER=/path/to/sccache (or set rustc-wrapper in config.toml).
I don't see any cache info on the action page, which I suspect means nothing is cached?

Good catch, seems I missed copying that line from
https://github.com/CQCL/tket2/blob/85ce5f9aa1233d0837d93f7666e58a318aba01ae/.github/workflows/ci.yml#L19-L20
:facepalm:

You are right on that, but even then the performance seem to be the same to rust-cache.
I'll keep an eye on it.


Significant increase in force closures after upgrade to LND 0.15

Background
Over the past 3 months I built a node with about 200 channels using 0.14.3. Initially I saw ~1-3 force closures a month, and typically the other peer confirmed that they had a major issue with their node leading to the failure. After upgrading to 0.15, I've seen on average 3 force closures per day. I've been unable to decipher the logs to get a sense of what is happening and what I should attempt to do to rectify the problem. As is, it seems my most active channels are disappearing daily.
I am not aware of any connectivity issues with my node, running hybrid mode, pubkey: 022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995.
Although Im not sure if it's relevant, I previously watched the HTLC stream with lntop, which would run for weeks in terminal. After upgrading to 0.15, lntop would lock up with many "pending" HTLCs. Now, it fails to launch at all.
Furthermore, lnd is running on a dedicated machine with bitcoind; prior to upgrading to 0.15, my cpu cores were running at an average of 0-5%; after upgrading to 0.15, most cores are at 5-10% utilization, with 1 or two cores spiking to 60-80% at any given time... No other changes have been made to the node, and I am not running rebalancing software.
Your environment

version of lnd 0.15
which operating system (uname -a on Nix) Linux WhiteBox 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
version of btcd, bitcoind, or other backend 0.23
any other relevant environment details

*** My most recent channel with c-otto force closed. I will include log entries for this channel funding outpoint below, followed by what c-otto sent me from his logs:
2022-07-16 11:55:43.342 [ERR] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): failing link: unable to accept revocation: revocation key mismatch with error: invalid revocation

2022-07-16 11:57:14.994 [ERR] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): failing link: ChannelPoint(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): unable to accept new commitment: not enough HTLC signatures with error: invalid commitment

2022-07-16 11:57:34.865 [INF] CNCT: ChannelArbitrator(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): force closing chan

*** Below are the logs sent from c-otto:
2022-07-16 17:55:32.875 [DBG] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): Failed to send 200022399 mSAT

^ Some failed HTLC, i.e. a working connection.

2022-07-16 17:55:39.614 [INF] PEER: Peer(022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995): unable to read message from peer: EOF
2022-07-16 17:55:39.614 [INF] PEER: Peer(022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995): disconnecting 022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995@<IP_ADDRESS>:44598, reason: read handler closed
2022-07-16 17:55:39.816 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): stopping
2022-07-16 17:55:39.817 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): exited

^ Disconnected

Then: reconnection

2022-07-16 17:55:39.927 [INF] SRVR: Finalizing connection to 022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995@<IP_ADDRESS>:44602, inbound=true
2022-07-16 17:55:39.934 [INF] PEER: Peer(022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995): loading ChannelPoint(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1)
2022-07-16 17:55:39.934 [DBG] CNCT: New ChainEventSubscription(id=146) for ChannelPoint(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1)
2022-07-16 17:55:39.934 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): starting
2022-07-16 17:55:39.935 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1)
2022-07-16 17:55:39.935 [INF] PEER: Peer(022a03c83e94ab037a64dd71e54f1796db185f21b1d88ceea5486a274ec257e995): Negotiated chan series queries
2022-07-16 17:55:39.935 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): HTLC manager started, bandwidth=2393706690 mSAT
2022-07-16 17:55:39.935 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): attempting to re-synchronize
2022-07-16 17:55:40.948 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): received re-establishment message from remote side
2022-07-16 17:55:40.959 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): sending 2 updates to synchronize the state
2022-07-16 17:55:40.966 [DBG] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): loaded 1 fwd pks
2022-07-16 17:55:40.966 [DBG] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): removing completed fwd pkg for height=101142
2022-07-16 17:56:44.460 [DBG] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): queueing keystone of ADD open circuit: (Chan ID=0:0:0, HTLC ID=10466197)->(Chan ID=737690:1073:1, HTLC ID=45951)

2022-07-16 17:57:44.522 [ERR] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): failing link: unable to complete dance with error: remote unresponsive

^ That's the only non-common log entry.

2022-07-16 17:57:44.522 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): exited
2022-07-16 17:57:44.525 [INF] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): stopping
2022-07-16 17:59:52.987 [INF] NTFN: Dispatching confirmed spend notification for outpoint=671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1, script=0 f4530c60eab59c020938bc93bbdb6028b758a556b8e53f42023018ae5d9cf767 at current height=745254: f68f0277aba6755ae46c98b532da40da6161444c0216ffca115fe061a99a4702[0] spending 671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1 at height=745254

Thanks for the issue! Let me know if you need further logs from my node.

I have a few of those:
2022-07-16 16:31:26.514 [ERR] HSWC: ChannelLink(671cfce1d2f30c93333e4fe472f95c859893343aa6c50dac10b4ff58b6207bdb:1): failing link: unable to complete dance with error: remote unresponsive

@mcsnubbs on a side note, lntop works for me running v0.15.0-beta. Could check your ~/.lntop/lntop.log

I had two force closes since posting this issue, both with large reputable nodes. Will include the log info. The second list of log entries has a "fwding package db has been corrupted" error that I have not yet seen.
2022-07-17 05:27:52.213 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:1)
2022-07-17 05:27:52.213 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:1): HTLC manager started, bandwidth=3075323126 mSAT
2022-07-17 05:27:52.213 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:1): attempting to re-synchronize
2022-07-17 05:27:52.213 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:1): received re-establishment message from remote side
2022-07-17 05:30:40.784 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): stopping
2022-07-17 05:30:40.784 [WRN] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): error when syncing channel states: link shutting down
2022-07-17 05:30:40.785 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): exited
2022-07-17 05:30:40.854 [INF] PEER: NodeKey(03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601) loading ChannelPoint(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2)
2022-07-17 05:30:40.854 [INF] HSWC: Removing channel link with ChannelID(4f8707322809a34b2cd464b35517f3dbc4dc22e0ff9395ae8a3724437465f0ce)
2022-07-17 05:30:40.854 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): starting
2022-07-17 05:30:40.854 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2)
2022-07-17 05:30:40.854 [INF] NTFN: New block epoch subscription
2022-07-17 05:30:40.854 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): HTLC manager started, bandwidth=4778017444 mSAT
2022-07-17 05:30:40.854 [INF] HSWC: ChannelLink(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): attempting to re-synchronize
2022-07-17 05:33:01.520 [INF] NTFN: Dispatching confirmed spend notification for outpoint=ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2, script=0 a3921047b009617866ad37ca1738d652dd123e5a5bd6ac2475eebd737421ce48 at current height=745344: f2910adaa3dd4faf75a92309020c13634610db16376ca58085b6b53b15d88606[0] spending ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2 at height=745344
2022-07-17 05:33:01.524 [INF] CNCT: Unilateral close of ChannelPoint(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2) detected
2022-07-17 05:33:01.525 [INF] CNCT: ChannelArbitrator(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): remote party has closed channel out on-chain
2022-07-17 05:33:15.614 [WRN] CHBU: Replacing disk backup for ChannelPoint(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:1) w/ newer version
2022-07-17 05:33:33.674 [INF] CNCT: ChannelArbitrator(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): still awaiting contract resolution
2022-07-17 05:33:45.683 [INF] CNCT: *contractcourt.commitSweepResolver(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): Sweeping with witness type: CommitmentToRemoteConfirmed
2022-07-17 05:33:45.683 [INF] CNCT: *contractcourt.commitSweepResolver(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): sweeping commit output
2022-07-17 05:34:44.332 [INF] NANN: Announcing channel(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2) disabled [detected]
2022-07-17 05:34:44.332 [ERR] NANN: Unable to sign update disabling channel(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): edge not found
2022-07-17 05:39:50.397 [INF] CNCT: *contractcourt.commitSweepResolver(ccf065744324378aae9593ffe022dcc4dbf31755b364d42c4ba309283207874f:2): local commitment output fully resolved by sweep tx: 997a314d838c55958f276afe3f00151453ce54922b3dccd3bfa28d8e7c811cc7
2022-07-16 21:56:36.420 [ERR] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): failing link: unable to complete dance with error: remote unresponsive
2022-07-16 21:56:36.420 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): exited
2022-07-16 21:56:36.949 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): stopping
2022-07-16 21:58:02.945 [INF] PEER: NodeKey(0262ab2e8c0f8d7062f51847eaf05900b43fe0902ba67858a4d8ec74fec68f4ba3) loading ChannelPoint(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0)
2022-07-16 21:58:02.945 [INF] HSWC: Removing channel link with ChannelID(32da91f3615a77ab6625579b80da3f46ed0fa0f19edaf2888712d6fcb25ac844)
2022-07-16 21:58:02.945 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): starting
2022-07-16 21:58:02.946 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0)
2022-07-16 21:58:02.946 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): HTLC manager started, bandwidth=3676794443 mSAT
2022-07-16 21:58:02.946 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): attempting to re-synchronize
2022-07-16 21:58:03.269 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): received re-establishment message from remote side
2022-07-16 21:58:05.661 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): sending 2 updates to synchronize the state
2022-07-16 21:58:27.683 [INF] NTFN: Dispatching confirmed spend notification for outpoint=44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0, script=0 61d99f1ebaa4a483603f65b56015dc50d294c5d2f542d985bc6a8f0ec56f1164 at current height=745303: b6fa986768f6ded26a5e465c8a0dfee1bd65a79de0cab12b20cf45f1c23a33b3[0] spending 44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0 at height=745303
2022-07-16 21:58:27.687 [INF] CNCT: Unilateral close of ChannelPoint(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0) detected
2022-07-16 21:58:27.688 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): remote party has closed channel out on-chain
2022-07-16 21:58:41.211 [ERR] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): unable to remove fwd pkg for height=12890: fwding package db has been corrupted
2022-07-16 21:58:41.211 [ERR] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): failing link: unable to resolve fwd pkgs: fwding package db has been corrupted with error: internal error
2022-07-16 21:58:41.211 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): exited
2022-07-16 21:58:58.495 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): still awaiting contract resolution
2022-07-16 21:59:06.077 [INF] CNCT: *contractcourt.commitSweepResolver(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): Sweeping with witness type: CommitmentToRemoteConfirmed
2022-07-16 21:59:06.077 [INF] CNCT: *contractcourt.commitSweepResolver(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): sweeping commit output
2022-07-16 21:59:06.529 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): a contract has been fully resolved!
2022-07-16 21:59:06.529 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): still awaiting contract resolution
2022-07-16 21:59:09.020 [INF] HSWC: Removing channel link with ChannelID(32da91f3615a77ab6625579b80da3f46ed0fa0f19edaf2888712d6fcb25ac844)
2022-07-16 21:59:09.020 [INF] HSWC: ChannelLink(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): stopping
2022-07-16 22:02:31.623 [INF] CNCT: *contractcourt.commitSweepResolver(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): local commitment output fully resolved by sweep tx: 64775e68ee277d9321f571f67f25a04313f3b0663d59cd356e61190eb69cb0ea
2022-07-16 22:02:40.153 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): a contract has been fully resolved!
2022-07-16 22:02:40.153 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): still awaiting contract resolution
2022-07-16 22:02:40.497 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): a contract has been fully resolved!
2022-07-16 22:02:40.497 [INF] CNCT: ChannelArbitrator(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): still awaiting contract resolution
2022-07-16 22:17:44.891 [INF] NANN: Announcing channel(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0) disabled [detected]
2022-07-16 22:17:44.894 [ERR] NANN: Unable to sign update disabling channel(44c85ab2fcd6128788f2da9ef1a00fed463fda809b572566ab775a61f391da32:0): edge not found

@mcsnubbs on a side note, lntop works for me running v0.15.0-beta. Could check your ~/.lntop/lntop.log

LNTOP log showing a long list of permission denied rpc errors
{"level":"error","ts":1657704389.4633987,"caller":"ui/controller.go:121","msg":"failed","logger":"controller","error":"rpc error: code = Unknown desc = edge not found","errorVerbose":"rpc error: code = Unknown desc = edge not found\ngithub.com/edouardparis/lntop/network/backend/lnd.Backend.GetChannelInfo\n\t/tmp/lntop/network/backend/lnd/lnd.go:346\ngithub.com/edouardparis/lntop/ui/models.(*Models).RefreshChannels\n\t/tmp/lntop/ui/models/models.go:63\ngithub.com/edouardparis/lntop/ui.(*controller).Listen.func1\n\t/tmp/lntop/ui/controller.go:119\ngithub.com/edouardparis/lntop/ui.(*controller).Listen\n\t/tmp/lntop/ui/controller.go:148\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/edouardparis/lntop/ui.(*controller).Listen.func1\n\t/tmp/lntop/ui/controller.go:121\ngithub.com/edouardparis/lntop/ui.(*controller).Listen\n\t/tmp/lntop/ui/controller.go:148"}
{"level":"error","ts":1657788681.9180472,"caller":"pubsub/pubsub.go:49","msg":"SubscribeInvoice returned an error","logger":"pubsub","error":"rpc error: code = Unknown desc = permission denied","stacktrace":"github.com/edouardparis/lntop/pubsub.(*PubSub).invoices.func2\n\t/tmp/lntop/pubsub/pubsub.go:49"}

Could you please edit your posts and use code block formatting (three backticks - code - three backticks)?

Done. I compacted channel db, went from 18 to 2 gb, since then my routing has largely normalized, as has my cpu utilization and ability to run lntop. Would it make sense that the large channel.db caused the issue?

The forwarding error log is a bug fixed in 0.15.1, and also the bug wouldn't be related to an upgrade to 0.15

The forwarding log error appears to occur when an on-chain spend is detected (forwarding packages get wiped) and the link is still up and tries to remove a forwarding pacakge via resolveFwdPkg. This should be fine, but what I am not sure of is why the peer with the 4f8707322809a34b2cd464b35517f3dbc4dc22e0ff9395ae8a3724437465f0ce channel would reestablish the channel if it went to force close. CC @yyforyongyu . I did make note of this a while ago and it is benign, but is probably something that should be changed

Things that could cause buggy force closes in 0.15 and prior from either you or C-Otto:

connectivity issues

tor had a ddos attack on hidden services recently, not sure if that's still ongoing

scripts that call DisconnectPeer and reconnect

scripts that call DisconnectPeer and reconnect

I do that. My node has the "add linkStopIndex to cleanly shutdown ChannelLink" fix, though.

I do that. My node has the "add linkStopIndex to cleanly shutdown ChannelLink" fix, though.

I am pretty sure you'll want the DisconnectPeer patch from here https://github.com/lightningnetwork/lnd/pull/6655 also. I think that even if one ChannelLink waits for the current one to finish, the new one might have outdated state when it is loaded.

I already have that, thanks.

tor had a ddos attack on hidden services recently, not sure if that's still ongoing

Yep it is (though of a slightly different nature): https://status.torproject.org/

Based on this comment, I think we can close this issue as most of the known force close scenarios have been resolved in 0.15.1? The one wild card still is the Tor network, as it still isn't hack at prior levels of connectivity/reliability.

So until 0.15.1 I should be regularly compacting channel.db to avoid similar issues?

No, the issues we fixed aren't related to DB compaction.

@mcsnubbs are you running with sync-freelist=true? It could be that the heavy CPU usage is coming from that which could perhaps also explain the force closes (your node not being able to respond in time). See https://github.com/lightningnetwork/lnd/issues/6737.

@mcsnubbs are you running with sync-freelist=true? It could be that the heavy CPU usage is coming from that which could perhaps also explain the force closes (your node not being able to respond in time). See #6737.

You are correct! This was my issue... both .db bloating and CPU usage returned to normal. Appreciate the assistance!

@mcsnubbs are you running with sync-freelist=true? It could be that the heavy CPU usage is coming from that which could perhaps also explain the force closes (your node not being able to respond in time). See #6737.

You are correct! This was my issue... both .db bloating and CPU usage returned to normal. Appreciate the assistance!

I had the same thing going on! Crazy db bloating after ~7 days. It looks like sync-freelist=false is helping!


use git API for downloading update-from-repo.sh

This is cool! :heart: Thanks a lot.

yeah, seems the advises how to download files from github are almost wrong, I found only one reference to api in one SO comment. everywhere else it was statet use rawgit ...

same issue here https://github.com/roundcube/roundcubemail/issues/6248
Did you already document it on SO?

OK, I'll try it and send you the link ....

Thank you.

https://stackoverflow.com/questions/49818899/official-way-to-access-download-files-from-github

Good job.


Page/outreach page

Description
Merging outreach page with master.

Quite happy with this!
It scales quite well when going smaller, but seems to be a slight spacing issue when on a wider screen my monitor is i think 21 - 23 inches (i forgot the exact dimensions) and the middle section thing is a bit funky.

Additionally, for some reason at one point i had a circle show up on the right arrow but cant seem to re-emulate it :(

Overall seems to just be minor css issues which i can nitpick, but otherwise well done!


assets folder is empty

After running the following command and go through 1000 epochs. The assets folder in the best_model folder is empty.
Command: Train the confidence model on the soccernet_v2_challenge_validation dataset (spotting_challenge_validated protocol) on the baidu_2.0 features
 python ./bin/train.py -sd ./data/splits/ -ld ./data/labels/ -fd ./data/features/baidu_2.0/ -fn baidu_2.0 -dt soccernet_v2_challenge_validat
ion -cd ./configs/soccernet_challenge_confidence -dc dense -cw 1.0 -lr 5e-5 -dwd 5e-5 -sr 0.0 -mu 0.0 -m $MODELS_DIR/spotting_challenge_validated_baidu_2.0_confidence_first_lr5e-5_dwd5e-5_sr0.0_mu0.0

folder structure of the best_model
.
├── assets
├── keras_metadata.pb
├── saved_model.pb
├── shared_args.pkl
├── validation_evaluation
│   ├── evaluation_aggregate.txt
│   └── evaluation_run.pkl
└── variables
    ├── variables.data-00000-of-00001
    └── variables.index

Also, here is our first trial result.
32/32 [==============================] - ETA: 0s - loss: 0.0167INFO:root:Validation metric (average_map_tight): 0.5696022962342601
32/32 [==============================] - 204s 7s/step - loss: 0.0167
Epoch 992/1000
32/32 [==============================] - 25s 783ms/step - loss: 0.0169
Epoch 993/1000
32/32 [==============================] - 25s 786ms/step - loss: 0.0167
Epoch 994/1000
32/32 [==============================] - 25s 778ms/step - loss: 0.0164
Epoch 995/1000
32/32 [==============================] - 25s 771ms/step - loss: 0.0167
Epoch 996/1000
32/32 [==============================] - 25s 786ms/step - loss: 0.0165
Epoch 997/1000
32/32 [==============================] - 25s 780ms/step - loss: 0.0168
Epoch 998/1000
32/32 [==============================] - 25s 780ms/step - loss: 0.0166
Epoch 999/1000
32/32 [==============================] - 25s 781ms/step - loss: 0.0164
Epoch 1000/1000
32/32 [==============================] - 53s 2s/step - loss: 0.0167 - val_loss: 0.0489
INFO:root:Validation metric (average_map_tight): 0.5709853791508082

Could you suggest how can I save the best model? we're very grateful for your continuous help!

I am sorry for asking a fundamental question in Tensorflow, and here is the guidance for Tensorflow beginners like me.
├── assets/ # The required external files, such as the initialized vocabulary file, are generally empty
├── assets.extra/ # Unnecessary files for TensorFlow graph, such as information about how to use SavedModel for users. Tensorflow does not use files in this directory.
├── saved_model.pb # The network structure of MetaGraph
├── variables # Parameter weight, including all model variables (tf.Variable objects) parameters
     ├── variables.data-00000-of-00001
     └── variables. index


Update function-call-in-argument-default (B008) to ignore arguments with immutable annotations

Extends #6781
Part of https://github.com/astral-sh/ruff/issues/3762

Summary

Test Plan

Stuck dealing with lifetimes
error[E0502]: cannot borrow `checker.diagnostics` as mutable because it is also borrowed as immutable
   --> crates/ruff/src/rules/flake8_bugbear/rules/function_call_in_argument_default.rs:148:9
    |
122 |             ArgumentDefaultVisitor::new(checker.semantic(), extend_immutable_calls.as_slice());
    |                                         ------------------ immutable borrow occurs here
...
148 |         checker.diagnostics.push(Diagnostic::new(check, range));
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ mutable borrow occurs here
149 |     }
150 | }
    | - immutable borrow might be used here, when `extend_immutable_calls` is dropped and runs the `Drop` code for type `std::vec::Vec`


Relaxnodeversion

Update to 8.12.0 and change from ~ to ^ in constraints.

Pull Request Test Coverage Report for Build 2705

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at ?%

Totals

Change from base Build 2683:
0.0%

Covered Lines:

Relevant Lines:
0

💛  - Coveralls