Messages to publish are lost when RabbitMQ connection fails

I have this simple publication code:
const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(config));

broker.on('error', (error) => {
    console.error(`Broker error: ${error.message}`);
});

setInterval(async function() {
      try {
        const publication = await broker.publish('pub', 'hello world');
        publication.on('error', (error) => {
            console.error(`Publication error: ${error.message}`);
        });
      } catch (error) {
        console.error(`Rascal config error: ${error.message}`);
      }
}, 1000);

It's running fine until I shutdown the RabbitMQ server. Then I see this error in the logs:
Broker error: Connection closed: 320 (CONNECTION-FORCED) with message "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"

The messages are not published now, obviously. There is no Publication error in the logs though, which caused some confusion for me.
When I start the RabbitMQ server, I'd expect the in-meantime messages to be published after Rascal reconnects. This doesn't happen, the messages are lost. If I understand the code correctly they're queued in poolQueue: https://github.com/guidesmiths/rascal/blob/a68834d46ccc0b476d4530a21368ca2b234348d6/lib/amqp/Vhost.js#L211 but then after reconnection the queue is overridden with an empty one in https://github.com/guidesmiths/rascal/blob/a68834d46ccc0b476d4530a21368ca2b234348d6/lib/amqp/Vhost.js#L245
My workaround is to add a local queue and pause/resume it based on the connection status:
const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(config));

const messagesToPublishQueue = async.queue((_, next) => {
	next();
}, 1);

messagesToPublishQueue.pause();

broker
    .on('error', (error) => {
        console.error(`Broker error: ${error.message}`);
    })
	.on('connect', () => {
		messagesToPublishQueue.resume();
	})
	.on('disconnect', () => {
		messagesToPublishQueue.pause();
	});

setInterval(async function() {
    messagesToPublishQueue.push(null, async () => {
      try {
        const publication = await broker.publish('pub', 'hello world');
        publication.on('error', (error) => {
            console.error(`Publication error: ${error.message}`);
        });
      } catch (error) {
        console.error(`Rascal config error: ${error.message}`);
      }
    });
}, 1000);

Hi @rogatty,
Thanks for posting. I'll have a dig into this later and make sure I understand the issue and what's going on, however I'm not sure the scenario you describe could happen using confirm channels. With a confirm channel (rascal's default). amqplib will wait for broker to acknowledge receipt of the message before yielding. The sequence of actions during publish is...

Grabs a channel from either the confirm or regular pools
Attach an error listeners to the channel
Publish the message (delegates to amqplib)
Remove the error listeners from the channel
Return the channel (may be deferred if the channel was saturated)
Emit any error reported by amqplib
Return

I don't see how amqplib would yield (without error) if it had not successfully published the message when using confirm channels.
As can be seen from the above steps, rascal does not internally queue messages pending publication, however it does queue requests to borrow channels from the channel pool. I would be reluctant to internally queue messages and attempting to republish them on disconnection. It could lead to situations where Rascal gave the impression the publish was successful, but in reality had a large stash of messages that would be lost if the application crashed / restarted. Furthermore it would be quite complicated to implement with amqplibs flow control (used to indicate when a channel is saturated, and the user should back off)

@cressie176 when saying

I'm not sure the scenario you describe could happen using confirm channels

you mean the Publication error not being logged? I'm using confirm channels and it happens 🙂 I believe it's because the first action on your list is never finished:

Grabs a channel from either the confirm or regular pools

It goes like that:

_publish calls borrowChannelFn with a callback that would emit the errors reported by amqplib
borrowChannelFn points to borrow which pushes the aforementioned callback to poolQueue
If the poolQueue isn't paused, the callback is executed and the rest of your list happens

But, when the RabbitMQ connection fails then:

handleConnectionError calls pauseChannelAllocation
pauseChannelAllocation calls confirmChannelPool.pause()
poolQueue is paused and our callback waits to be executed

When the RabbitMQ connection is back:

handleConnectionError calls init
init calls createChannelPools  which calls createChannelPool
createChannelPool overrides poolQueue which drops our callback queue

rascal does not internally queue messages pending publication, however it does queue requests to borrow channels from the channel pool

Directly, it queues requests to borrow channels. Indirectly, it also queues messages pending publication, because they're callbacks passed to borrowChannelFn.

Thanks for explaining again @rogatty, I understand the issue now. Agree it's a problem that needs fixing.

OK. Think I've fixed it. Basic premise is to never replace the channel pools. If the connection is closed, the channels will automatically be removed from the pool by the channel error/close handler. The pool will either be automatically replenished (depending on configuration) or a new channel will be created when next requested. I tested by tweaking the busy publisher example and manually closing the connection from the management ui. Will push shortly.

Published as<EMAIL_ADDRESS>

Thank you! The messages previously dropped are now published after reconnection. It works as I'd expect it to, but maybe should be mentioned in the docs so it's clear to everyone.

I would be reluctant to internally queue messages and attempting to republish them on disconnection. It could lead to situations where Rascal gave the impression the publish was successful, but in reality had a large stash of messages that would be lost if the application crashed / restarted.

This is exactly what's happening 😉

I would be reluctant to internally queue messages and attempting to republish them on disconnection. It could lead to situations where Rascal gave the impression the publish was successful, but in reality had a large stash of messages that would be lost if the application crashed / restarted.

This is exactly what's happening 😉

Not quite (unless I'm still missing something). Rascal shouldn't have called the callback, so while messages had been queued as a side effect of the event loop, Rascal shouldn't have given the impression that the publish was successful.

Messages are queued and when connection is re-established, they're successfully published. There is no publisher error event emitted.

I was trying to explain that I don't want to give people the impression that just because you call broker.publish that the message is guaranteed to be delivered until rascal has executed the callback / resolved the promise. It's still possible for the application to crash or be shutdown before a connection recovers. Under this scenario messages inflight would still be lost.

Understood. It's up to you if and how you want to document the current behaviour 🙂 If someone tries to debug what's happening they'll probably end up here and can learn from the discussion above.
Thanks again for fixing the issue so quickly!


[UI] Allow namespace creation when deploying default chart in Rancher

Is your feature request related to a problem? Please describe.
When deploying a default Epinio chart namespace creation is not possible; however, this is possible when deploying a custom Epinio chart.
Installation options when using default Epinio charts:

Installation options when using custom Epinio chart from repo:

Besides the inconsistency, this may be an issue while working with the new Epinio extension. We noticed that attempting to deploy the default Epinio chart without selecting the Epinio namespace (because it cannot be added) will end up with the application not being found in the cluster:

Describe the solution you'd like
Creating a namespace from the default chart should be possible
Describe alternatives you've considered

Additional context

it looks like the extension is expecting Epinio to be installed in epinio namespace. Epinio rancher app is using hardcoded namespace cattle-epinio-system.

There's two parts to this

Namespace is not shown when installing 102.0.3+up1.8.1 (rancher release), but is for 1.9.0 (dev epinio release)
This is because the 1.8.1 chart contains the annotation catalog.cattle.io/release-name, which tells the UI / Rancher that the chart must be installed to it's a specific namespace. Going on @enrichman comment this requirement has been dropped. So the newer chart must not contain the annotation. This is working as expected.

When installing epinio to a namespace other than epinio it's not recognised as an epinio cluster by the extension
We hadn't tracked this in an issue, so created https://github.com/epinio/ui/issues/322

Given the above, i think we can close this one


[VLIVE.TV] Add translated title(s) to metadata, infojson, output filename

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm requesting a site-specific feature
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
Worldwide
Example URLs
https://www.vlive.tv/post/1-21079894
https://www.vlive.tv/post/0-18265700
Provide a description that is worded well enough to be understood
yt-dlp currently only extracts the original title of the videos on vlive.tv. My request is to get all translated titles (if available) so they can be written to the infojson and added to the metadata if chosen (--parse-metadata). In addition, add a way to choose the English translated title as the output filename.
Thank you!
"multinationalTitles": [
    {
      "type": "VIDEO_TITLE",
      "seq": 234829,
      "locale": "en_US",
      "label": "🐻💛Seulgi's Wise Playbook⏰",
      "defaultYn": false
    },
    {
      "type": "VIDEO_TITLE",
      "seq": 234829,
      "locale": "ko_KR",
      "label": "🐻💛슬기로운 바른 생활⏰",
      "defaultYn": true
    }
  ]

"multinationalTitles": [
    {
       "type": "VIDEO_TITLE",
       "seq": 192406,
       "locale": "en_US",
       "label": "A million Tizennie are watching💚💚💚💚💚"
    },
    {
       "type": "VIDEO_TITLE",
       "seq": 192406,
       "locale": "in_ID",
       "label": "Satu juta tizennie yang menonton💚💚💚💚💚"
     },
     {
       "type": "VIDEO_TITLE",
       "seq": 192406,
       "locale": "vi_VN",
       "label": "Các bạn NCTzen đang xem chúng tớ đúng không💚💚💚💚💚"
     }
  ]

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['--remux-video', 'mkv', '--sub-langs', 'all,-live_chat', '--embed-subs', '--convert-subs', 'srt', '--embed-thumbnail', '--embed-metadata', '--embed-info-json', '--downloader', 'http:aria2c', '--concurrent-fragments', '4', '--no-playlist', '--download-archive', '/content/drive/MyDrive/VLIVE/download-archive.txt', '-vU', '--config-locations', '/content/drive/MyDrive/VLIVE/yt-dlp.conf', '--merge-output-format', 'mkv/mp4', 'https://www.vlive.tv/post/1-21079894', '-P', '/content/drive/MyDrive/VLIVE/RED VELVET', '-o', '%(release_date)s [%(channel)s] %(title)s (%(id)s).%(ext)s', '-P', 'temp:/content/drive/MyDrive/VLIVE/!temp']
[debug] | Config "/content/drive/MyDrive/VLIVE/yt-dlp.conf": ['--embed-metadata', '--parse-metadata', '%(translated_title)s:%(meta_translatedtitle)', '--parse-metadata', '%(release_date)s:%(meta_date)s', '--parse-metadata', 'webpage_url:%(meta_URL)s', '--ppa', 'EmbedSubtitle:-disposition:s:m:language:eng default']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b644025b]
[debug] Lazy loading extractors is disabled
[debug] Python 3.8.15 (CPython x86_64 64bit) - Linux-5.10.133+-x86_64-with-glibc2.27 (OpenSSL 1.1.1  11 Sep 2018, glibc 2.27)
[debug] exe versions: ffmpeg N-109368-gcc11afe502-20221206 (setts), ffprobe N-109368-gcc11afe502-20221206
[debug] Optional libraries: Cryptodome-3.16.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1730 extractors
[debug] Loading archive file '/content/drive/MyDrive/VLIVE/download-archive.txt'
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[vlive:post] Extracting URL: https://www.vlive.tv/post/1-21079894
[vlive:post] 1-21079894: Downloading post JSON metadata
[vlive] Extracting URL: http://www.vlive.tv/video/234829
[vlive] 234829: Downloading officialVideoPost JSON metadata
[vlive] 234829: Downloading inkey JSON metadata
[vlive] 234829: Downloading JSON metadata
[info] 234829: Downloading subtitles: en_US, tl_fan1, fr_FR_fan1, in_ID, in_ID_fan1, pt_BR_fan1, vi_VN, tr_TR_fan1, th_TH, ja_JP_fan1, ko_KR
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Searching for '%(meta_translatedtitle)' in '%(translated_title)s'
[MetadataParser] Could not interpret '%(translated_title)s' as '%(meta_translatedtitle)'
[debug] Searching for '(?P<meta_date>.+)' in '%(release_date)s'
[MetadataParser] Parsed meta_date from '%(release_date)s': '20210129'
[debug] Searching for '(?P<meta_URL>.+)' in '%(webpage_url)s'
[MetadataParser] Parsed meta_URL from '%(webpage_url)s': 'http://www.vlive.tv/video/234829'
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 234829: Downloading 1 format(s): avc1_1080P
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_01_31/c2f64aab-63a0-11eb-9e65-005056853b14-1612082768177_en_US_cp.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.vtt
[download] 100% of    3.01KiB in 00:00:00 at 3.80KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_02/3e2ff7ef-651d-11eb-9dc7-005056852fbb-1612246183673_tl_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.vtt
[download] 100% of    3.15KiB in 00:00:00 at 9.09KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_08/496155fa-69a8-11eb-9d6d-005056851803-1612745707147_fr_FR_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.vtt
[download] 100% of    3.42KiB in 00:00:00 at 5.78KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_05/540e5c25-67b9-11eb-93de-005056852fbb-1612533124177_in_ID_cp.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.vtt
[download] 100% of    3.20KiB in 00:00:00 at 5.85KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_15/5781dbe9-6f35-11eb-9d6d-005056851803-1613356045701_in_ID_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.vtt
[download] 100% of    3.25KiB in 00:00:00 at 5.10KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_07/982d1755-68f5-11eb-9d6d-005056851803-1612668959431_pt_BR_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.vtt
[download] 100% of    3.04KiB in 00:00:00 at 8.06KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_06/3d27957d-67e8-11eb-9925-005056853b14-1612553272101_vi_VN_cp.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.vtt
[download] 100% of    4.15KiB in 00:00:00 at 6.89KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_08/1b502d30-69d2-11eb-9925-005056853b14-1612763668722_tr_TR_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.vtt
[download] 100% of    3.32KiB in 00:00:01 at 2.18KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_07/20ce549b-6913-11eb-93de-005056852fbb-1612681644062_th_TH_cp.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.vtt
[download] 100% of    6.02KiB in 00:00:00 at 9.71KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_02_15/7de858dc-6f39-11eb-9d6d-005056851803-1613357828113_ja_JP_fan.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.vtt
[download] 100% of    3.83KiB in 00:00:00 at 10.85KiB/s
[info] Writing video subtitles to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.vtt
[debug] Invoking http downloader on "http://resources-rmcnmv.pstatic.net/globalv/c/read/v2/VOD_ALPHA/2021_01_30/6942dc6a-62a4-11eb-9dc7-005056852fbb-1611974384508_ko_KR_cp.vtt"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.vtt
[download] 100% of    3.36KiB in 00:00:00 at 6.02KiB/s
[info] Downloading video thumbnail 1 ...
[info] Writing video thumbnail 1 to: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).jpg
[SubtitlesConvertor] Converting subtitles
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.srt'
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.vtt' -f srt -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.srt'
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.vtt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.vtt (pass -k to keep)
[debug] Invoking http downloader on "http://globalv.p.naverrmc.edgesuite.net/c/read/v2/VOD_ALPHA/global_v_2021_01_29_253/1a951dd7-6239-11eb-8866-246e96398ca5.mp4?__gda__=1670434038_6ec34d2bdf9453d943ffd7ee08a071ef"
[download] Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mp4
[download] 100% of   91.32MiB in 00:00:07 at 12.39MiB/s
[VideoRemuxer] Remuxing video from mp4 to mkv; Destination: /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mp4' -map 0 -dn -ignore_unknown -c copy -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv'
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mp4 (pass -k to keep)
[EmbedSubtitle] Embedding subtitles in "/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.srt' -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.srt' -map 0 -dn -ignore_unknown -c copy -map -0:s -map 1:0 -metadata:s:s:0 language=eng -map 2:0 -metadata:s:s:1 language=tgl -map 3:0 -metadata:s:s:2 language=fra -map 4:0 -metadata:s:s:3 language=ind -map 5:0 -metadata:s:s:4 language=ind -map 6:0 -metadata:s:s:5 language=por -map 7:0 -metadata:s:s:6 language=vie -map 8:0 -metadata:s:s:7 language=tur -map 9:0 -metadata:s:s:8 language=tha -map 10:0 -metadata:s:s:9 language=jpn -map 11:0 -metadata:s:s:10 language=kor -movflags +faststart -disposition:s:m:language:eng default 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).temp.mkv'
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).fr_FR_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tr_TR_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ja_JP_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).pt_BR_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).en_US.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).ko_KR.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).vi_VN.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).in_ID.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).tl_fan1.srt (pass -k to keep)
Deleting original file /content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).th_TH.srt (pass -k to keep)
[debug] Writing info-json to: /content/drive/MyDrive/VLIVE/NCT/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).info.json.temp
[debug] ffprobe command line: ffprobe -hide_banner -show_format -show_streams -print_format json 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv'
[Metadata] Adding metadata to "/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv' -map 0 -dn -ignore_unknown -c copy -write_id3v1 1 -metadata 'title=🐻💛슬기로운 바른 생활⏰' -metadata date=20210129 -metadata purl=http://www.vlive.tv/video/234829 -metadata comment=http://www.vlive.tv/video/234829 -metadata 'artist=Red Velvet' -metadata URL=http://www.vlive.tv/video/234829 -attach '/content/drive/MyDrive/VLIVE/RED VELVET/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).info.json.temp' -metadata:s:13 mimetype=application/json -metadata:s:13 filename=info.json -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).temp.mkv'
[debug] ffprobe command line: ffprobe -hide_banner -show_format -show_streams -print_format json 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv'
[EmbedThumbnail] ffmpeg: Adding thumbnail to "/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv' -map 0 -dn -ignore_unknown -c copy -attach '/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).jpg' -metadata:s:14 mimetype=image/jpeg -metadata:s:14 filename=cover.jpg -movflags +faststart 'file:/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).temp.mkv'
[MoveFiles] Moving file "/content/drive/MyDrive/VLIVE/!temp/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv" to "/content/drive/MyDrive/VLIVE/RED VELVET/20210129 [Red Velvet] 🐻💛슬기로운 바른 생활⏰ (234829).mkv"
[debug] Adding to archive: vlive 234829

To devs: We can add a new field translations that looks like:
'translations': {
	'title': {
		'in_ID': ...,
		...
	},
	'description': ...,
}

When adding it, make sure to document this in docstring of InfoExtractor

VLIVE was shut down at the end of 2022

VLIVE was shut down at the end of 2022


1.9.1 Windows Package PyInstaller cannot read .kv import statements

From the PyInstaller documentation: it finds all import statements in your script.
However when I do the following in my .spec file:
print(a.scripts)
It only includes my main.py
(Full code: https://groups.google.com/forum/#!topic/kivy-users/7WSaBGQ-rZs)
In my case my ScreenManager when switching a screen calls new.kv which calls for new.py script. So main.py -> new.kv -> new.py.
Therefore my conclusion is that PyInstaller can't read the import statements in a .kv file.
(If I include my .py files manually to the .spec, it still doesn't work, because the .kv files are looking for the .py file)
I'm still able to successfully run my app, but that is only because I include the uncompiled .py files (through Tree() ). However I wish to make use of the PyCrypto function in PyInstaller, which doesn't work if PyInstaller doesn't compile the scripts.
Also I think including compiled .pyc/.pyo is better for the performance of my app.
Would it be possible to write a script for PyInstaller so that it can read .kv import statements or something else? And that then the import statements in the .kv file look for these compiled .pyc/.pyo files instead of the .py.

I have the same problem.

We discussed this problem on IRC. Regarding making pyinstaller automatically discover imports in kv, this is currently unlikely to happen because we'd have to make pyinstaller parse python code to find which kv files are imported. Then we'd have to make it understand kv so it could get those imports. It's a lot easier to just add those imports to hiddenimports in the spec file.
In addition, his specific issue was about failing to add his project to the python path or some such.
@2n2u please open a new issue with a clearer description of what your problem is. If it's just about imports, I'd suggest that you add them to hiddenimports.

@2n2u :
I solved it with the following file.spec:
http://stackoverflow.com/questions/35253158/kivy-1-9-1-windows-package-spec-single-exe

@NumesSanguis Not sure if I follow, I need to import inside the .kv file, I can't find on that page anything related to that.

@2n2u you can import in kv, just make sure that you also add those imports to hiddenimports, in your spec file.

@dessant
`# -- mode: python --
from kivy.deps import sdl2, glew
block_cipher = None
#'six','packaging','packaging.version', 'kivymd',
a = Analysis(['main.py'],
pathex=['D:\Dropbox\1_TESE\airflow'],
binaries=None,
datas=None,
hiddenimports=['kivymd'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
exclude_binaries=True,
name='airflow',
debug=True,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
Tree('D:\Dropbox\1_TESE\airflow\'),
*[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)],
strip=False,
upx=True,
name='airflow')
`
I guess I can't import the whole module, right?
for instance I need a label:
'kivymd.MDLabel', etc

Try the whole module, and if it does not work, list 'kivymd.MDLabel'.
If you have further support questions about pyinstaller, either visit their mailing list, or join #kivy.

@NumesSanguis @dessant Thanks for the help, the app is running, now I need to try @NumesSanguis approach to convert it to a single file :)
http://puu.sh/nxCA6/7aa9e9974d.png

@2n2u : I import my python files in Kivy like this: #import filename folder.folder.filename (filename is without .py). However nothing of this is put in the PyInstaller .spec. Just put the files in the hiddenimport = [].
Using PyInstaller for a single.exe seems to be slow, because it has to unzip at runtime everything (and I got errors trying to use the -single option). Try NSIS instead, which unpacks everything to the /temp/ folder first (standalone exe): http://stackoverflow.com/questions/35552659/nsis-after-pyinstaller-kivy-app-cant-find-database

@NumesSanguis I'm using NSIS, I'm just stuck with importing garden modules, since it always search (Windows) on C:\Users\User.kivy\garden\ instead of the app folder.


Is the msal.idtoken safe to use for authentication

Library

[ ]<EMAIL_ADDRESS>or<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[x]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>
Description
Hi
I am in the process of enabling our existing web site (Angular SPA with .Net Core API (and apps)) to make use of Azure AD through MSAL for authenticating our users.
Our solution already has built in user-management, which I expect some portion of our customers will continue using. It is a single-tenant solution. So each customer has their own installation/domain. At some customers only the API is installed, along with a windows client - instead of the Angular SPA. So each customer will also have to register our solution as an App in their Azure AD.
In our solution we manage 2.000+ userrights, that are mapped to the users through User Groups. Obviously I cannot map these in Azure as API permissions.
So my solution so far, is to use that MsalGuard on a landing page in my Angular SPA.
This gives me the msal.idtoken, once the user has authenticated with MS.
I have then made changes in the .Net Core API, and using Microsoft.AspNetCore.Authentication.AzureAD.UI so I can verify/authenticate the AzureAD tokens serverside.
From my Angular site, I then call the API to give me an accessToken, which is similar to the accesstoken we use if customers just use username/password login on our site. It holds claims to identify the user, and from which we can deduct the authorization rights this user possesses. The API endpoint for getting the Access Token requires authorization - and the ID Token is sent as Authorization Bearer header.
The first question is:
I am currently using the IDToken received from MSAL to validate the users identity in the API. I do this using the "oid" claim.
Would that be considered a correct approach? Or do I need to get an access token from MSAL? (I find that step not necessary.)
Asked on stackoverflow a week ago - and did not get fully enlightened.
https://stackoverflow.com/questions/63879052/create-my-own-accesstokens-with-ms-identity-platform
Secondly.
In the case we are now, I will make it possible for customers to enable Azure AD login on an already existing userbase.
But the existing users are obviously not mapped with Object Identifier (oid-claim) yet. And I can't see the administrators locating the object id and set it manually to connect our user-object with the Azure-account.
So, if I can't find a user with the oid of the idtoken, I am currently using the "preferred_email" claim from the id-token, and look up users in our system with the same mail registered - and then I connect the two and persist oid on our user object.
Are there better approaches here?
For new users, after AzureAD is enabled, I expect them to receive a welcome mail with a dedicated link to map the user with the Azure AD account they logon with. So we don't need the email step there.
I hope you are able to answer the above - to make me more certain that I am doing the right or wrong things. :-)
Best Regards
/Anders

I will first say that the usage you're trying to describe here is generally not fully supported by us, but it should theoretically work. It seems to me like you're on the right track and things are correct for your use case.

I am currently using the IDToken received from MSAL to validate the users identity in the API. I do this using the "oid" claim.
Would that be considered a correct approach? Or do I need to get an access token from MSAL? (I find that step not necessary.)

This is fine as long as you are using the id token explicitly for user validation rather than resource acquisition (where you would want an access token).

So, if I can't find a user with the oid of the idtoken, I am currently using the "preferred_email" claim from the id-token, and look up users in our system with the same mail registered - and then I connect the two and persist oid on our user object.
Are there better approaches here?

This also looks correct and is a perfectly valid usage.
You probably have already seen this, but I will plug the quickstart showing how to expose a custom web API in an app. It may be useful to leverage AAD more where you can--though you may also just want to stick to what you have.

Thanks for the quick reply.
I will continue on the path I am on.
It would be great to not have the step where our product generates the access token, because we could then just claim standard OAuth2 compliant.
But I can't figure out how to inject the extra information we need in our token (userid, roleid, and unitid) into the Azure token.
It comes down to a chicken or egg dilemma. I can't trust that someone "IS" the userid, and is allowed to take on the roleid, or use the unitid - without valid authentication.
So I need the ID token for that - and Azure can't tell me the mapping to user, role and unit.
So no matter how, I need the extra roundtrip to our API to get the userid, roleid, and unitid claims.
So for external developers coding against our API - it can't be just an OAuth2 implementation because I need to add information to the token after it is generated.
And then a  follow up on the scopes - this might be a good idea.
Will it make sense to make a scope called API.Access?
The purpose for this, as I see it, is that if some of the customer organization want to lock down API access even more, then by creating this scope, and saying only admins can grant it - then the individual users can not accept on their own - but need higher level access.
Our customers vary a lot in security policies, but the main objective of using Azure AD and MSAL is to get single-sign-on and MFA. And the ability for the customer organization to e.g. close a user - and know their access is revoked.
I think I still need to use my own access tokens later in the process, but could quite easily change the flow to

Require token from Azure (not sure if I can get Id + Access token for "API.Access" in one go - or need to calls)
With the Azure Access Token, I call my API, and issue our site-generated Access Token
Use our site-generated Access Token until it expires.
Then I need a refreshed AccessToken from Azure, and go to step 2.

The benefit of this approach will be

There is an explicit scope in Azure AD on the Application registration
This grant can be revoked - and then the user can no longer use our product.

The ID token cannot be revoked unless a User is closed.
Is it possible for domain admin to revoke access to scopes for other users in the domain?

I think I might experiment a bit with that.
Once again - thanks for the quick and clear reply.

Will it make sense to make a scope called API.Access?

The purpose for this, as I see it, is that if some of the customer organization want to lock down API access even more, then by creating this scope, and saying only admins can grant it - then the individual users can not accept on their own - but need higher level access.
Yes, this is the exact use case for a custom scope. AAD wants users to be able to get tokens for an arbitrary setup using scopes.

Require token from Azure (not sure if I can get Id + Access token for "API.Access" in one go - or need to calls)
With the Azure Access Token, I call my API, and issue our site-generated Access Token
Use our site-generated Access Token until it expires.
Then I need a refreshed AccessToken from Azure, and go to step 2.

This also seems correct to me for your use case. It's unfortunate that you'll have to basically do auth twice--once against AAD and once against your app--but if my understanding is correct, that may just be the nature of the situation here.

Thanks again.
And yes - I think i'm stuck with the two authorizations.
I need the token I use for data-requests to include the additional claims. But who knows - I might find a better way before I'm done.


Create empty Map for Map @Option in command methods

Currently the following will result in a null Map being passed to the method:
public Status exec(@Option(names = "-e", description = "Environmental variables in key=value format") Map<String, String> vars) {
// ...
}

From the docs:

If the field is null, picocli will instantiate it when the option or positional parameter is matched. If the type is not a concrete class, picocli will instantiate a LinkedHashMap to preserve the input ordering.

However this is isn't a field so the expectation is unclear. Ideally this would be initialized to an empty Map. Updating the documentation to account for command methods would also be useful.

@bobtiernay-okta Does this answer your question?

Yes it does thanks. Sorry, I planned to say I would help add a note to the docs. Might be able to get to it later in the week. Thanks!

That would be great, thanks!

So I'm trying to update the documentation but I'm struggling to square the existing docs:

If the field is null, picocli will instantiate it when the option or positional parameter is matched

with your comment above:

The convention in picocli is to not modify annotated fields (or parameters) that weren’t matched on the command line, unless a default value was specified.

In the case of an @Option that is on a command method parameter without a defaultValue specified, should't the behavior be to create it per the first sentence above?

I created a test to demonstrate the current behaviour and actually found a bug. :-) The test is in the issue I created: https://github.com/remkop/picocli/issues/538
The test shows the expected behaviour:

if no options are matched, method parameters without a default will have a null value, while method parameters with a default will have the default value
if options are matched, the matched value is passed to the method

In the case of fields, it's similar, but slightly more complex because fields can have an initial value in their declaration, for example:
@Option(names = "-x")
Integer x = 123;

if the option is not matched, annotated fields without a default will be set to their initial value, and annotated fields with a default will be set to the default value. Instead of "are not modified" it's more accurate to say "are reset to the initial value".
if the option is matched, annotated fields are set to the matched value. For multiple values that may be a List, and picocli will instantiate the List unless the field already has a non-null value:

@Option(names = "-y")
List<String> y = null; // picocli creates a List instance if -y is matched

@Option(names = "-z")
List<String> z = new ArrayList<>(); // picocli adds to this existing instance if -z is matched

Did my last comment answer your question?

Early warning: I’m planning to do a release in a week or so. If you can provide any doc improvements this week they will be included in the upcoming release. Of course there will be other releases if that’s too soon.

Will try to get to this tomorrow or Monday.

Hi @bobtiernay-okta, I saw the PR on #545, thank you! Is this ready to be merged and are you happy to close this ticket when #545 is merged or are there more changes in the pipeline?

I think its good enough for now. I was going to add more detail but it really didn't fit in that section. I'll try to contribute a more substantial piece next time!

Understood. I merged to PR. Do you want to keep this ticket open?


pull request for e2e test at 01/21/2017 03:26:18

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


How to set up arcgis js in my webpack config.

Hello,
Im pretty new on this webpack worlds and i need some help to set up the arcgis js api in my webpack.config,js.
Im developing an app with:
React,
Webpack dev server
Gulp
I have the following config in my webpack file:

var path = require('path');
var esri = require('esri');
module.exports = {
entry: {
login: './static/js/bundles/login.js',
dashboard: './static/js/bundles/dashboard.js,
.... //more modules here
},
output: {
path: path.join(path.join(__dirname, 'dist'), 'js'),
filename: '[name].js'
},
devServer: {
inline: true,
port: 443,
host: "<IP_ADDRESS>"
},
module: {
loaders: [
{
test: /.(js|jsx)$/,
exclude: /node_modules/,
loader: 'babel',
query: {
presets: ['react', 'es2015', 'stage-2']
}
}
]
},
externals: [
function(context, request, callback) {
if (/^dojo/.test(request) ||
/^dojox/.test(request) ||
/^dijit/.test(request) ||
/^esri/.test(request)
) {
return callback(null, "amd " + request);
}
callback();
}
]
};

What else i need to do? I have read a lot of githubs but i still dont know how to do it.
In my index,html page i have the api in this way:

Cuz i installed it following the tutorial, editing the dojo and init js files and putting the url from my server, in this case localhost:443 where wp server is being loaded.
When i start the server with npm run start i have the following error:

cannot fin module 'esri'

So how i set up the arcgis api with the config that i have using webpack step by step?
Thanks in advice for ur help!

I will put and example app that i have set up in this repo:
https://github.com/darklilium/CHQAves

remove this line, it is not valid, and does not appear to be used: var esri = require('esri');
Esri modules are not exposed as CommonJS, nor are they installed in node_modules, so you cannon statically require them.

OK.
I am a little bit confused.
Let me explain my situation.
I want to use the arcgis js api using the es6 import like i have been using for modules that i have in node_modules, something like this:
Import Map from 'esri/Map';
If i do that, having my api installed as the guideline that the API has (not using CDN) it doesnt work.
The way im loading the modules is using the legacy code that the arcgis api has (i know is not good to do it in this way:  for example: var layer = new esri.layers.FeatureLayer({the code...});
I have my repo here: https://github.com/darklilium/CHQAves thats the enviroment what im using right now.
Is there a way to use the arcgis js api doing some changes in the webpack.config.js file to be able to use the api completely? cuz if i use the legacy mode (how i have been using it) there are some functions and widgets that the api has that i cannot use (for example : BaseMapGallery);
I hope u can help me getting this done, and thanks in advice.

Did you try my above suggestion? Did that fix the 'cannot fin module 'esri'' error?

Yes.

OK. Good.
Are you still having problems, other errors?

I have the following when for example i want to do this in a jsx file to use the api:
import FeatureLayer from 'esri/layers/FeatureLayer';
Uncaught ReferenceError: WEBPACK_EXTERNAL_MODULE_484 is not defined

My webpack config is like this now:

var path = require('path');
module.exports = {
entry: {
login: './static/js/bundles/login.js',
dashboard: './static/js/bundles/dashboard.js',
interruptions: './static/js/bundles/interruptions.js',
apchq: './static/js/bundles/apchq.js',
factigisDashboard: './static/js/bundles/factigisDashboard.js',
factigisBackoffice: './static/js/bundles/factigisBackoffice.js',
factigisFrontoffice: './static/js/bundles/factigisFrontoffice.js',
factigisBackoffice2: './static/js/bundles/factigisBackoffice2.js',
apdashboard: './static/js/bundles/apdashboard.js',
apmenu: './static/js/bundles/apmenu.js',
factigisCarta: './static/js/bundles/factigisCarta.js'
},
output: {
path: path.join(path.join(__dirname, 'dist'), 'js'),
filename: '[name].js'
},
devServer: {
inline: true,
port: 443,
host: "<IP_ADDRESS>"
},
module: {
loaders: [
{
test: /.(js|jsx)$/,
exclude: /node_modules/,
loader: 'babel',
query: {
presets: ['react', 'es2015', 'stage-2']
}
}
]
},
externals: [
function(context, request, callback) {
if (/^dojo/.test(request) ||
/^dojox/.test(request) ||
/^dijit/.test(request) ||
/^esri/.test(request)
) {
console.log(request);
return callback(null, "amd " + request);
}
callback();
}
]
};

Did you try my suggestion here:
https://geonet.esri.com/message/632846-re-load-amd-modules-in-es6-way?commentID=632846&et=watches.email.thread#comment-632445
It does not supported the the repo that you link to above is a fork of this repo.

Well i clone that repo and im trying to figure out what i need to do, but there are some config i dont use and i dont understand (i dont use typescript, for example) so idk how to adapt my code for that app to that..
I have been searching for any resolution to my problem but i still dont find what im looking for.
And also idk where to ask...
Anyways , thx for ur help.

I see. Yes, if you are unfamiliar with, or do not want to use TypeScript, then it can be challenging to get started with this repo.
I have created a fork that shows the base configuration to load ArcGIS API for JavaScript (3.x) modules in webpack using es2105 and babel instead of typescript:
https://github.com/tomwayson/esri-webpack/tree/es2015
I could not get the karma test configuration to work, so I stripped it out of there for now just to show the simplest possible set up.
That does not include react, Sass, nor any of the other bells and whistles that it looks like you plan on using in your project, but hopefully you can use that as a base that you can build upon to add those.

OK Im going to try it and see how i can set up my config.
Thanks for all ur help, if i have some any other questions i will repost here :)

Sorry I missed this, been away for a while.
@darklilium:  The other thing that may be worth looking at is the new app scaffolding from Facebook:  https://facebook.github.io/react/blog/2016/07/22/create-apps-with-no-configuration.html
Should be relatively easy to start with that, "eject" out of the starter app, then add the ESRI libs as externals in your webpack config.

How should i add the esri lib as externals? Cuz I saw the example with jquery in the webpack web but i quite dont understand how they do it.
Another question, is there any difference if i installed the arcgis api lib as they said in the tutorial? i mean im not using CDN as u are, i installed the lib modifying the dojo and init.js files with the localhost:443 where i set up my webpack server in the config and then i add the script tag in the html.

Using externals should work whether you're pulling from CDN or not; you're basically just telling webpack not to bundle those files.  The relevent section in the webpack config is:
externals: [
        function(context, request, callback) {
            if (/^dojo/.test(request) ||
                /^dojox/.test(request) ||
                /^dijit/.test(request) ||
                /^esri/.test(request)
            ) {
                return callback(null, "amd " + request);
            }
            callback();
        }
    ]

i remember i add that code in my webpack config file, but after that i dont know how to , for example, call the api features as "import Map from 'esri/Map'.
I noticed in your code something that i dont use, like in the index.html u require the bundles main and vendor. Is that necessary? cuz in my config i just add the bundle directly as a compiled file  with the script tag for each html webpage im making.

It's not necessary; we're only doing that because the webpack output is set to amd.  You may need to change that if you want just use a script tag.
Also, many of the ESRI libs don't seem to be compliant with the ES6 import syntax; I'm not sure if this is a TypeScript thing only, but we had to use
import Map = require("esri/map");
to get the imports to work properly.

Ok, i have my demo repo here https://github.com/darklilium/ReactWPackArcGisJS
but im having some problem with an error, it says:
login.js:1 Uncaught ReferenceError: webpackJsonp is not defined

I'd guess that error is caused by the chunking plugin; you can take that part out of the webpack config if you're not using it.
And it may be that have to set the webpack output to AMD; I'm not sure if you'll be able to consume the ESRI libs any other way.

in the index.html u require the bundles main and vendor. Is that necessary?

Yes, that is necessary.

And it may be that have to set the webpack output to AMD; I'm not sure if you'll be able to consume the ESRI libs any other way.

That is correct. You must output to AMD and load the bundled output via a require statement in index.html. Dojo's AMD loader is going to load all the modules in the application.

im not using CDN as u are

do you have a reason to do that? If not, you should at least start by using the CDN as it will be easier to get working

i installed the lib modifying the dojo and init.js files with the localhost:443 where i set up my webpack server in the config and then i add the script tag in the html.

 If so, that should, in theory, work the same as w/ the CDN. So, again, I'd start w/ the CDN, get it working, then point to your local build of the ArcGIS API for JavaScript.
However, just to be clear, I would not follow the patterns in the "Using Bower for Custom Builds" section of that document w/ the type of webpack configuration we are using here. The modules in the bower build are meant to be built w/ the Dojo build system. It may be possible to adapt those patterns for use w/ Nordth/dojo-webpack-loader, however, I'm not aware that anyone has successfully done that yet.

Also, many of the ESRI libs don't seem to be compliant with the ES6 import syntax; I'm not sure if this is a TypeScript thing only, but we had to use import Map = require("esri/map"); to get the imports to work properly.

Yes, that is a typescript thing. I'm not sure why, but you have to use that syntax to load esri modules, but only from TypeScript. You can see that I am using the regular ES2015 imports/exports syntax w/ babel.

OK.
Yea, im using the pattern described in the download section of the api cuz here the internet has too much limitations, so i prefer to have ir locally.
I set up one example on my repo and when i tried to use ReactDom to render one component, gives me some error.
(BTW the api works, cuz w/o that i was able to load a map, now i want to add a component made with react but idk whats the problem.
login.js

import React from 'react';
import ReactDOM from 'react-dom';
import LoginApp from '../components/Login.jsx';
ReactDOM.render(, document.getElementById('myLogin'));

Login.jsx

import React from 'react';
import Map from "esri/map";
class LoginApp extends React.Component {
constructor(){
super();
}
componentWillMount(){
var map = new Map("map_content", {basemap: "streets"});
}
render(){
return (

);
}
}
export default LoginApp;

So what will be the problem?
Thanks for all ur help, in little by little understand the config that i have to have in my app.
Now i need to fix that.

I'm glad you've been able to get past the esri module loading issues. It now looks like you're getting into issues w/ how to use React w/ the ArcGIS API for JavaScript, which means I'm not likely to be much help b/c I don't use react. That said, glancing at the above code, I don't see where the dom node w/ id map_content is defined. I'm guessing you need to return that from the from the login component's render()?
Either way, now that you've got the modules loading, you should probably be able to follow the patterns that @odoe has established for using the ArcGIS API for JavaScript w/ React even if he's not using webpack to bundle the app modules, so I don't think this thread is really the best place to solve that issue.
I suggest you open an issue for this on your own repo, include the error message you're seeing, and ping the react devs from this repo and/or link to it from your GeoNet question about react so that you get the right people looking at it.

Oh i solved it.
It was missing this in the webpack.config.js
query: {
presets: ['react', 'es2015', 'stage-2']
}
Thanks for all ur help, i will follow your recommendation :)

Great, glad you worked it out!
I'm also really glad that you set that repo up as a generic esri+webpack+es2015+react configuration. I think a lot of people will find that useful.
I suggest closing this issue and opening up any further issues over there.

Will externals and libtarget amd still work with webpack 4+ and arcgisjs 3.x? We are stuck on that version for now but want to upgrade webpack

@dcworldwide I can't speak to that; you might want to look at some of the work @tomwayson has done with esri-loader.  See the readme for a link.

I'm pretty sure I got externals and libtarget working w/ webpack 4... somewhere... once.


Add troubleshooting docs

We should try looking at open issues and potential solving points (e.g. missing mksquashfs, missing pipeline configuration, singularity bind paths) and add these to the troubleshooting sections as these questions are asked quite frequently either on slack or the github issue trackers. Golden rule:
"Have a FAQ entry for it, whenever something has been asked for >= 3 times!"

e.g #103

Done: https://nf-co.re/usage/troubleshooting


updated references to the download page to instead reference latest.m…

Summary
Updated references to the download page to instead reference latest.mattermost.com and the version archive.

@ztrayner Following up on Carlos's comment, is there a particular reason why we want to reference latest.mattermost.com?

@amyblais @cpanato - The reason is because currently our marketing download page sends people to the docs for installation instructions. Then the docs currently sends people back to the marketing download page to grab the string of the server download.  It's a weird circular dependency.  By having a URL that provides the latest download we don't need to send people to an external page just to get the download URL string.
@jaydeland worked on creating the endpoint at latest.mattermost.com.  I am indifferent to the format of the URL.

@cpanato and @amyblais - this is simple an S3 routimg rule redirect to the existing releases.mattermost.com - the choice was between releases.mattermost.com/latest/<....> or latest.mattermost.com/<...> and the later seemed cleaner to me, plus we already have release.mattermost.com/server/latest as file that we didn't want to change.
If it's preffered we can look at moving it to releases.mattermost.com/latest/

I leave the decision to the PM/Release manager, I just expressed my point of view.
I think the release pipeline should update the files and push to something like: https://releases.mattermost.com/mattermost/latest/blabla-enterprise.tar.gz
because we can have desktop following the same as well
https://releases.mattermost.com/mattermost-desktop/latest/blabla-enterprise.tar.gz
and
https://releases.mattermost.com/mattermost-mmctl/latest/blabla-enterprise.tar.gz
https://releases.mattermost.com/mattermost-push-proxy/latest/blabla-enterprise.tar.gz

We used X.X.X to require the admin to decide what version they wanted on upgrade so they would ready the changelog and understand potentially breaking changes that could toast their system. i.e. We intentionally didn't want to point to latest.
I just wanted to check if our thinking changed on this?

Sorry - still testing stuff - it should work now:
jpd@jpd-Oryx-Pro:~/dev/matterbuild$ wget http://latest.mattermost.com/server-linux-team
--2020-04-03 16:44:37--  http://latest.mattermost.com/server-linux-team
Resolving latest.mattermost.com (latest.mattermost.com)... <IP_ADDRESS>
Connecting to latest.mattermost.com (latest.mattermost.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://releases.mattermost.com/5.21.0/mattermost-team-5.21.0-linux-amd64.tar.gz [following]
--2020-04-03 16:44:37--  https://releases.mattermost.com/5.21.0/mattermost-team-5.21.0-linux-amd64.tar.gz
Resolving releases.mattermost.com (releases.mattermost.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to releases.mattermost.com (releases.mattermost.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162489701 (155M) [application/x-gzip]
Saving to: ‘server-linux-team’
server-linux-team                                                                  3%[=====>                                                                                ]   5.36M  2.15MB/s               ^C

@coreyhulen - do you have an opinion on latest.mattermost.com vs releases.mattermost.com/latest?

I would lean towards releases.mattermost.com/latest, but 0/5, really no preference.

@jaydeland -- we were previously intentionally only pointing people to free E0, since it's the version people can upload a license to without needing to upgrade their server.
It could make sense to change it to show both links, but there should be a conversation about potential impacts if we do that.

@lfbrock -- so the latest URL for server should be: mattermost-enterprise-5.21.0-linux-amd64.tar.gz

Path forward:

[ ] Finalize and confirm URL structure for latest - @jaydeland @cpanato please confirm when finalized.
[ ] Update PR to only include E0 (remove Team reference) and use the finalized URL for latest - per @lfbrock 's previous post, no confirmation needed unless opinion has changed.
[ ] Remove reference to latest in upgrade.rst and instead only reference the version archive and keep the read the upgrade notes warnings already in place - @it33 @jasonblais please confirm that this resolves the X.X.X concern raised by @it33.

If any of the above steps seem incorrect, please let me know.  If your confirmation has been requested, please respond.  I'm unsure of our normal convention for collaborating on something like this and if it should take place here on the PR or in the original JIRA issue.  Seeing as we're all here, I'm going to post this here.
If we are truly opposed to using a latest endpoint, then removing all references to the marketing download page and replacing them with references to the version archive is also fine by me.  My purpose in this PR is to remove the circular reference from download-->docs-->download and relying on the download page as the canonical source of truth for the download string.  @jasonblais recommended against using the version archive and we collaborated and decided to use a latest endpoint.  I am 2/5 in favor of using a latest endpoint in the docs so users can have all of the information required for a new install on one page.  I am 0/5 on the format of the URL.  Note that a link to the version archive is included on each page in case the user wants to read the changelog for each release.
Thanks to everyone for collaborating on this.

Hi @ztrayner, huge thanks for all the good intentions and work here. Can we meet with folks on this thread along with Ken, me, Corey and Lindsay to discuss the process for changes to install guides like this?
These pages are absolutely critical to our community and company. There was a ton of testing that went into these docs, and a lot of contexts around how we do TE and EE, and how these pages integrate with the original download page:https://web.archive.org/web/20181214203140/https://mattermost.com/download/
A lot of very specific design was lost in early 2019 when we had contractors experimenting on the download page, and it's fallen out-of-sync with the doc pages--but we need to build up that understanding among the new team members so we hit the most important goals.
I spend time with Ken today talking through it and I think it's worth spending some time to get everyone on the same page.

due to the use of cloudfront with releases.mattermost.com we have no choice but to use latest.mattermost.com with this solution.
There is also a txt file being produced with the supported redirects at:
http://latest.mattermost.com
Contents of the file are:
http://latest.mattermost.com/server-linux-team
http://latest.mattermost.com/server-windows-team
http://latest.mattermost.com/mattermost-enterprise-linux
http://latest.mattermost.com/mattermost-enterprise-osx
http://latest.mattermost.com/mattermost-enterprise-windows
http://latest.mattermost.com/mattermost-desktop-amd64-deb
http://latest.mattermost.com/mattermost-desktop-i386-appimage
http://latest.mattermost.com/mattermost-desktop-i386-deb
http://latest.mattermost.com/mattermost-desktop-ia32-linux-tar
http://latest.mattermost.com/mattermost-desktop-x64-linux-tar
http://latest.mattermost.com/mattermost-desktop-x86_64-appimage
http://latest.mattermost.com/mattermost-desktop-dmg
http://latest.mattermost.com/mattermost-desktop-x64-msi
http://latest.mattermost.com/mattermost-desktop-x86-msi
http://latest.mattermost.com/mattermost-desktop-exe

Note that the latest URLs are redirect rules on the latest.mattermost.com S3 bucket that are managed by matterbuild (https://mattermost.atlassian.net/browse/IS-327)

Team Edition latest links have been added, but as @lfbrock stated we should point people to the Entreprise version to ease upgrading.
http://latest.mattermost.com/mattermost-team-linux
http://latest.mattermost.com/mattermost-team-osx
http://latest.mattermost.com/mattermost-team-windows

http://latest.mattermost.com/mattermost-enterprise-linux
http://latest.mattermost.com/mattermost-enterprise-osx
http://latest.mattermost.com/mattermost-enterprise-windows

are we planning to update these three URLs to not include the word "enterprise"?
http://latest.mattermost.com/mattermost-linux
http://latest.mattermost.com/mattermost-osx
http://latest.mattermost.com/mattermost-windows

Eric Sadur and I just had a chatted about this issue and agree the intent of this PR is to remove the circular references from the docs to the /download page. As such, the next steps to resolve this issue are as @ztrayner outlined above.
In summary:

Ensure the "latest" URLs for the "enterprise" downloads are as follow:
http://latest.mattermost.com/mattermost-linux
http://latest.mattermost.com/mattermost-osx
http://latest.mattermost.com/mattermost-windows
Update the install docs to reference the "latest" URL (#1) instead of the mattermost.com/download page
Update the upgrade docs to reference the Version Archive instead of the mattermost.com/download page

@it33 this is the next iteration for the team to improve the install / upgrade journey. Based on the conversation you and I had earlier, we are ready to move forward without meeting with Corey and Lindsay. Can you please confirm?

I'm not against this change, but I feel strongly that this is not a small iteration. It has a wider impact on install/upgrade experience beyond removing the circular reference and deserves careful consideration, including:

potential risks involved in introducing /latest -- such as the customer example I shared on Friday. Another is the risk we introduce to community installers and other automated scripts.
user flows impacted (either positively or negatively), and how these compare with other self-hosted install flows like MongoDB, Elastic, GitLab, Jira Server
other options considered for removing the circular reference

To iterate, I'm not opposed, but I do not feel comfortable making a change in the most critical user flow for the business without careful consideration. Happy to discuss.
@kolofsen As part of the iterations involving the install/upgrade experience, please include me in the conversations in addition to Sadur, so I can provide context and use cases where appropriate.

Currently, this is what the technical piece produces - any or all can be used:
https://latest.mattermost.com/mattermost-desktop-amd64-deb
https://latest.mattermost.com/mattermost-desktop-i386-appimage
https://latest.mattermost.com/mattermost-desktop-i386-deb
https://latest.mattermost.com/mattermost-desktop-ia32-linux-tar
https://latest.mattermost.com/mattermost-desktop-x64-linux-tar
https://latest.mattermost.com/mattermost-desktop-x86_64-appimage
https://latest.mattermost.com/mattermost-desktop-dmg
https://latest.mattermost.com/mattermost-desktop-x64-msi
https://latest.mattermost.com/mattermost-desktop-x86-msi
https://latest.mattermost.com/mattermost-desktop-exe
https://latest.mattermost.com/mattermost-enterprise-linux
https://latest.mattermost.com/mattermost-enterprise-osx
https://latest.mattermost.com/mattermost-enterprise-windows
https://latest.mattermost.com/mattermost-team-linux
https://latest.mattermost.com/mattermost-team-osx
https://latest.mattermost.com/mattermost-team-windows

@justinegeffen helped gather a list of pages involving a circular reference of docs —> download page —> back to same docs page. Listing below for reference:
SERVER UPGRADE GUIDEs:

https://docs.mattermost.com/administration/upgrade.html
https://docs.mattermost.com/administration/upgrading-to-3.0.html
https://docs.mattermost.com/administration/legacy-upgrade.html

SERVER INSTALL GUIDES:

https://docs.mattermost.com/install/install-ubuntu-1604.html#installing-mattermost-server
https://docs.mattermost.com/install/install-ubuntu-1804.html#configuring-mattermost-server
https://docs.mattermost.com/install/install-debian.html#installing-mattermost-server
https://docs.mattermost.com/install/install-rhel-6.html#installing-mattermost-server
https://docs.mattermost.com/install/install-rhel-7.html#installing-mattermost-server

DESKTOP APP INSTALL GUIDES:

https://docs.mattermost.com/install/desktop.html

Below are other pages we direct the reader to the download page for the latest client or server (which is fine for now in the existing journey):

https://docs.mattermost.com/overview/product.html#mattermost-editions
https://docs.mattermost.com/developer/slash-commands.html
https://docs.mattermost.com/administration/hipchat-migration-guidelines.html
https://docs.mattermost.com/getting-started/welcome_email.html
https://docs.mattermost.com/help/apps/desktop-changelog.html

Moreover, when reviewing current download a/b test metrics, we noticed it appears the circular reference of docs > download > docs does not actually often happy.
However, there was some noise in the metrics (due to a redirect in the a/b test instrumentation), so we'd want to take a closer look prior to the start of the a/b test.

@ztrayner @esadur As next steps, I'd propose we:

Look at the occurrence of people going from docs > download page > docs, so we can estimate the impact and hence the priority
With info from 1, prioritize against other initiatives Eric has in the customer journey - which includes those in the install/upgrade experience and the download page we've discussed
If the circular reference is deemed highest priority, then we should set up a way to measure the impact of this change, so we can identify negative effects to the experience, if any (the change could be a hard-coded server link, use of the /latest URL, or something else).

Let me know if there are any questions or concerns. We can discuss more on our next sync-up call on Thursday.


Polkadot - Min amount left

In Polkadot, accounts with a balance of less than 1 DOT can be considered "dust" and are at risk of being reaped or removed from the network. This is part of the network's design to maintain efficiency by reducing state bloat. Users of the Polkadot network need to be aware that maintaining a balance below this threshold could result in their account being deleted, along with any remaining balance.
To avoid this, users should ensure that their account balance does not fall below 1 DOT.
We need to provide guidance or alerts to users, it would be wise to include a clear warning about this risk, advising them to keep their balances above this minimum threshold to safeguard their accounts.
[ ] - Validate if > 1.1 is left so we don't have this risk. If they want to move all ok, but if they leave dust we should not allow it.

An account must have a balance of at least 1 DOT to be active on the Polkadot network. This is known as the Existential Deposit (ED). If your account balance ever drops below the ED, the account will be reaped (deactivated), and any remaining balance will be lost!
Fixed on https://github.com/vultisig/vultisig-ios/pull/393


Bug on selecting an item using  (page moves up)

I'm trying to add a simple dropdown menu with nb-select, e.g.:
<nb-select  formControlName="unit">
    <nb-option value="cm">Centimetre</nb-option>
    <nb-option value="m">Meter</nb-option>
</nb-select>

But when I select some item, the entire page moves up. Actually, an inline style is applied to the html, changing the "top" position for some negative value.
This problem has begun after I've upgraded from nebular 3.0.0 to the last version (3.4.2).
It is possible to reproduce this problem even in the ngx-admin demo page (http://akveo.com/ngx-admin/pages/extra-components/form-inputs).

I have the same issue with version 3.4.2

Estou com o mesmo problema, alguma solução?

I have the same issue

I have this issue too.

Let me add some info:
Current behavior:
Popup window from <nb-select> or another popup call with Material overlay is causing blank room below the footer.
Page should have vertical scroll and popup should be attached to input from second (scrolled down) screen and further.
Here is screenshot.
Steps to reproduce:

Go to demo page: http://akveo.com/ngx-admin/pages/extra-components/form-inputs.
Activate some of bottom select-boxes.
See blank stripe at the bottom of the screen.

I guess it's related to <nb-layout> and cdk-overlay.
P.S. Tons of thanks to you guys for this awesome admin panel 😍

I have the same problem. Was the project abandoned?

I have this issue too. Aren't there any solution for this?

anyone figure it out? its causing a issue with my form inputs as the input is at the bottom of the page when user clicks the drop down button the page turns white

i don't know but for now i am just using
/deep/ .cdk-global-scrollblock {
position: static !important;
}
in my component that uses the dropdown in, so far i notice nothing breaks 😍

any updates, we have same issue

Still, have this issue in nebula 3.5.0

Same issue here. Emko's suggestion above did not work for me.

I can resolve this issue with changed build option.

extractCss: from true to false

          "configurations": {
            "production": {
              "extractCss": false,

I hope this helps you.

still occurs on
"@nebular/eva-icons": "4.4.0",
"@nebular/security": "4.4.0",
"@nebular/theme": "4.6.0",

still occurs on "@nebular/theme": "4.6.0", but nb-select inside a dialog
Any update?

Same issue with the weird white footer when using a popup with nbselect component is the pull request working for this as well?
https://github.com/akveo/ngx-admin/issues/5685

I have added
.cdk-global-scrollblock {
position: static !important;
}
directly in index.html file.
Finally nb-select in modal works as expected!
Hope this helps anyone with this issue.
Nebular: 4.2.1 Angular: 8.0.0

It still happens to me in Nebular 5.0 angular 9.0.4
my index.html file
<!doctype html>

	<meta charset="utf-8">
	<title></title>

	<base href="/">

	<meta name="viewport" content="width=device-width, initial-scale=1">
	<link rel="icon" type="image/png" href="favicon.png">
	<link rel="icon" type="image/x-icon" href="favicon.ico">

	<ngx-app>Loading...</ngx-app>

@vamidi, what is the version of Nebular & Angular you are working on right now?

Nebular 5.0.0 and angular 9.0.4

i don't know but for now i am just using
/deep/ .cdk-global-scrollblock {
position: static !important;
}
in my component that uses the dropdown in, so far i notice nothing breaks 😍

Thank you so much for this!!

Angular 15.2.1
Node 18.18.2
@include nb-install-component() {
:host {
::ng-deep {
.cdk-overlay-pane {
z-index: 9999999 !important;
}
}
}
}


data: Gauthamprasath

Profile update

You have conflicts in your file, please resolve. This is usually because your fork is not up to date.
You may wish to have a look at the following Docs and video: https://linkfree.eddiehub.io/docs/how-to-guides/editing


Last update is a string instead of a datetime.

The sensor._last_update state is interpreted as a string instead of a datetime.
Possibly, it is just missing the 'device_class: timestamp' attribute.
As it is, the sensor does not get reformatted to e.g. relative time by the UI.

Can you check https://github.com/audiconnect/audi_connect_ha/pull/300 ?

It doesn't seem to solve the problem. In the entities card:

I still do not have the device_class in the attributes of the entity. I have checked that the dashboard.py contains your latest changes from #300

Yep can confirm it works in your branch

I close this case, as I guess the https://github.com/audiconnect/audi_connect_ha/pull/300 will be merged today or tomorrow @cdnninja


Export variables for non-Bash script access

Bash command scripts are sourced by the ./go script itself, so they've always had access to _GO_ROOTDIR and other such variables. With this change, these variables are exported, so that scripts written in other languages (which involves launching an interpreter in a new process) now have access to them as well.
Also fixes minor bugs with @go.printf and __create_test_script.

Coverage increased (+0.08%) to 96.288% when pulling 2777ca8b46896ecbb6e7eccc485f2ad6b4ed49e6 on core-updates into 6444f516541dbf92cc8049d2cad17616800eb379 on master.


Add the logic for EditContact.jsx

This one is to add the logic to edit a contact

set up the backend from the  Backend Repository
import useEffect and useState hooks
import useParams and useNavigate hooks
add a useEffect to load in a contact by id (you might want to useParams to get the id
add a handleEditContact function using fetch API
add a form to edit the contact data similar to the AddContact.jsx form

This should make for a fairly straight forward issue. if there are any questions please feel free to leave a comment.

Can I work on this issue ?

Hi there excited to contribute can you please assign me.

@VishalMyla I will assign you to this as you were the first to ask. @Ritish134  I will be making more issues to be worked on and it is usually first to ask gets assigned

hi @decagondev , I would love to work on this , i see you have assigned it to another student but can i still work on it and try to contribute?

Please review the PR : 5

Thanks @VishalMyla  I have taken a look at your PR and it fits the specification well and is clean. I have merged it :+1:


Is this project still alive?

Hey @ducc, first want to say, thanks for putting all this work together! Its an awesome bot, and IMO an excellent starting point for someone wanting to learn Go and also expanding a discord bot as part of their project to learn.
I feel like a lot of design choices you made make it very easy for me to wrap my head around the code base, so its been pretty easy to expand on so far.
With that in mind, I submitted a PR #3 about a week ago with no response. And it made me curious if this project was still alive. I have more features I'd like to introduce into the bot for my own purposes, such as video caching for faster play time, and trivia because I'm on discord with a bunch of The Office nerds 😂
Was curious if I should continue to submit PRs or just maintain a separate fork giving you credit?

Merged thank you :) I don't write discord bots anymore but I know this is passed around as a discordgo music bot starting point so any contributions are welcome. If I don't reply in the future please msg me spong#3338


fix: passing value null in post to type number throws type error

Closes #509
cc @nickschot

Current coverage is 88.59% (diff: 82.35%)

Merging #516 into master will decrease coverage by 0.13%

@@             master       #516   diff @@
==========================================
  Files           170        170
  Lines          1838       1842     +4
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           1631       1632     +1
- Misses          207        210     +3
  Partials          0          0

Powered by Codecov. Last update 6f4e170...8c3238e


Task2[2ptx] Food list

Food list:

[ ] Should display a list of food added by the logged-in user as in the wireframe (for teams with only 2 people - display also quantity of a given food).
[ ] Should lead to a form that allows users to add new food.


validation does not work when value is bound using expression like "currentEntity[p] & validate"

I'm submitting a bug report

Library Version:
<EMAIL_ADDRESS>
Please tell us about your environment:

Operating System:
OSX 10.x
Node Version:
6.3.0

NPM Version:
3.10.3

JSPM OR Webpack AND Version
JSPM 0.16.46

Browser:
all
Language:
all

Current behavior:
per http://aurelia.io/hub.html#/doc/article/aurelia/validation/latest/validation-basics/6, i should be able to use a binding like this
 <input type="text"   value.bind="model[field] & validate">

Binding does work but there appears to be no way to define a rule that will fire.
Expected/desired behavior:

binding doenst work, ask demonstrated in the gist: https://gist.run/?id=44ecfbe8b7693ee6975d4f91f2d9150a

What is the expected behavior?
validation should work correctly

@antonmos here's a working version: https://gist.run/?id=90812aab71ad6e131406b2a99ac9c5ea
Here are the two small changes I made to correct the issues with the usage of .ensure(...) and the data-binding:  https://gist.github.com/jdanyow/90812aab71ad6e131406b2a99ac9c5ea/revisions

@antonmos here's a working version: https://gist.run/?id=90812aab71ad6e131406b2a99ac9c5ea
Here are the two small changes I made to correct the issues with the usage of .ensure(...) and the data-binding:  https://gist.github.com/jdanyow/90812aab71ad6e131406b2a99ac9c5ea/revisions

@jdanyow thank you for the workaround!!
However, i think either documentation needs to be updated or there is a bug.  The documentation explicitly claims that  <input type="text"   value.bind="model[field] & validate"> should work.
I updated my gist to demonstrate that it doesnt when using the .on(this) syntax.

@antonmos you might have not seen the second change in my fork of your gist, this part was invalid:
.ensure("model[field]")

needs to be:
.ensure("my-email")

Once you fix that your gist will work.

I did see it and thank you. My point is that the documentation says that validation should work with dynamically selected fields and without having to use "Explicit Binding Validation".
I updated the gist to reflect this usecase.
from: http://aurelia.io/hub.html#/doc/article/aurelia/validation/latest/validation-basics/6
The binding behavior is able to identify the object and property name to validate in all sorts of binding expressions:

Automatic Binding Validation Copy
HTML
<input type="text" value.bind="firstName & validate">

<input type="text" value.bind="person.firstName & validate">

<input type="text" value.bind="person['firstName'] | upperCase & validate">

<input type="text" value.bind="currentEntity[p] & debounce & validate">

Thanks again!

Right but the docs don't say anything about using .ensure the way you've attempted:
.ensure("model[field]") and .ensure(model['${this.field}']) are invalid because they don't target a specific property.
Maybe what you're looking for is .ensure(this.field) ?

That works, if i setup the rules in the constructor instead of in the module scope and use .on(this).
This all makes sense now.   Thanks again!

Hi, @jdanyow !
I'm having trouble with your gist https://gist.run/?id=90812aab71ad6e131406b2a99ac9c5ea
When typing something -> blur -> clear -> blur > type -> blur i get validation errors stacked but as normal they shouldn't

Need fix for this asap, thanks in advance!


fixProc: simplify wrap

The previous method was slightly non-performant in that it re-allocated the string every time.
IMHO this is cleaner, and its more performant.

:+1:


Getting different color in both platform

Hi, thanks for this awesome library, I am using this in one of my react native projects. But my client wants me to get the same exact color for IOS and Android as well, but this library gives different color for the same image. what should I do?

This library is a wrapper around the Palette class on Android and UIImageColors on iOS. So because they have different implementation, they can extract different colors for the same image.
There isn't anything you can do. you could tinker with the colors extracted for some image and see which properties are closest to each other in iOS and Android.
you can check if node-vibrant works with react native since its implemented in JS and will get the same colors for both platforms.


Unable to perform file transfert from pyboard

rshell 0.0.28
Windows 10 , driver : microsoft usbser.sys
Board : Pytrack (lopy4) , Pycom MicroPython 1.18.2.r7
Each time i try to get file from lopy i'm facing an error (exception)
I can do repl, list my file with size (ls -l) but i was unable to do file transfert, each time i got, i have already try to run rshell with '-a' but without more succes
Have you any idea how i can solve this issue ?
File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1102, in recv_file_from_remote
dst_file.write(binascii.unhexlify(write_buf[0:read_size]))
binascii.Error: Non-hexadecimal digit found
Complete debug trace :
C:\Travail\temp>rshell -p com3 -d cp /flash/L76GNSS.py L76GNSS.y
Debug = True
Port = com3
Baud = 115200
User = micro
Password = python
Wait = 0
List = 0
nocolor = 0
ascii = 0
Timing = 0
Quiet = 0
BUFFER_SIZE = 512
Cmd = [cp, /flash/L76GNSS.py, L76GNSS.y]
Using buffer-size of 32
Connecting to com3 (buffer-size 32)...
Trying to connect to REPL  connected
Testing if ubinascii.unhexlify exists ... ----- About to send 304 bytes of code to the pyboard -----
def test_unhexlify():
    """Checks the micropython firmware to see if ubinascii.unhexlify exists."""
    import ubinascii
    try:
        _ = ubinascii.unhexlify
        return True
    except:
        return False
output = test_unhexlify()
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'True\r\n'
-----
Y
Retrieving root directories ... ----- About to send 222 bytes of code to the pyboard -----
def listdir(dirname):
    """Returns a list of filenames contained in the named directory."""
    import os
    return os.listdir(dirname)
output = listdir('/')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b"['flash']\r\n"
-----
/flash/
Setting time ... ----- About to send 1695 bytes of code to the pyboard -----
def set_time(rtc_time):
    rtc = None
    try:
        # Pyboard (pyboard doesn't have machine.RTC()).
        # The pyb.RTC.datetime function takes the arguments in the order:
        # (year, month, day, weekday, hour, minute, second, subseconds)
        # http://docs.micropython.org/en/latest/library/pyb.RTC.html#pyb.RTC.datetime
        import pyb
        rtc = pyb.RTC()
        rtc.datetime(rtc_time)
    except:
        try:
            import pycom
            # PyCom's machine.RTC takes its arguments in a slightly different order
            # than the official machine.RTC.
            # (year, month, day, hour, minute, second[, microsecond[, tzinfo]])
            # https://docs.pycom.io/firmwareapi/pycom/machine/rtc/#rtc-init-datetime-none-source-rtc-internal-rc
            rtc_time2 = (rtc_time[0], rtc_time[1], rtc_time[2], rtc_time[4], rtc_time[5], rtc_time[6])
            import machine
            rtc = machine.RTC()
            rtc.init(rtc_time2)
        except:
            try:
                # The machine.RTC documentation was incorrect and doesn't agree with the code, so no link
                # is presented here. The order of the arguments is the same as the pyboard.
                import machine
                rtc = machine.RTC()
                try:
                    # ESP8266 uses rtc.datetime() rather than rtc.init()
                    rtc.datetime(rtc_time)
                except:
                    # ESP32 (at least Loboris port) uses rtc.init()
                    rtc.init(rtc_time)
            except:
                pass
output = set_time((2020, 5, 20, 3, 12, 21, 45, 0))
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'None\r\n'
-----
May 20, 2020 12:21:45
Evaluating board_name ... ----- About to send 710 bytes of code to the pyboard -----
def board_name(default):
    """Returns the boards name (if available)."""
    try:
        import board
        try:
            name = board.name
        except AttributeError:
            # There was a board.py file, but it didn't have an name attribute
            # We also ignore this as an error
            name = default
    except ImportError:
        # No board.py file on the pyboard - not an error
        name = default
    except BaseException as err:
        print('Error encountered executing board.py')
        import sys
        sys.print_exception(err)
        name = default
    return repr(name)
output = board_name('pyboard')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b"'pyboard'\r\n"
-----
pyboard
Retrieving time epoch ... ----- About to send 334 bytes of code to the pyboard -----
def get_time_epoch():
    """Determines the epoch used by the MicroPython board."""
    import time
    try:
      return time.gmtime(0)
    except:
      """Assume its a pyboard, with an epoch of 2000."""
      return (2000, 1, 1, 0, 0, 0, 0, 0)
output = get_time_epoch()
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'(1970, 1, 1, 0, 0, 0, 3, 1)\r\n'
-----
Jan 01, 1970
Executing "cp /flash/L76GNSS.py L76GNSS.y"
----- About to send 474 bytes of code to the pyboard -----
def get_mode(filename):
    """Returns the mode of a file, which can be used to determine if a file
       exists, if a file is a file or a directory.
    """
    import os
    try:
        # Since this function runs remotely, it can't depend on other functions,
        # so we can't call stat_mode.
        return os.stat(filename)[0]
    except OSError:
        return 0
output = get_mode('/flash/L76GNSS.py')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'32768\r\n'
-----
----- About to send 396 bytes of code to the pyboard -----
def get_filesize(filename):
    """Returns the size of a file, in bytes."""
    import os
    try:
        # Since this function runs remotely, it can't depend on other functions,
        # so we can't call stat_mode.
        return os.stat(filename)[6]
    except OSError:
        return -1
output = get_filesize('/flash/L76GNSS.py')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'2040\r\n'
-----
----- About to send 1269 bytes of code to the pyboard -----
def send_file_to_host(src_filename, dst_file, filesize):
    """Function which runs on the pyboard. Matches up with recv_file_from_remote."""
    import sys
    import ubinascii
    try:
        with open(src_filename, 'rb') as src_file:
            bytes_remaining = filesize
            if False:
                buf_size = 32
            else:
                buf_size = 32 // 2
            while bytes_remaining > 0:
                read_size = min(bytes_remaining, buf_size)
                buf = src_file.read(read_size)
                if False:
                    sys.stdout.buffer.write(buf)
                else:
                    sys.stdout.write(ubinascii.hexlify(buf))
                bytes_remaining -= read_size
                # Wait for an ack so we don't get ahead of the remote
                while True:
                    char = sys.stdin.read(1)
                    if char:
                        if char == '\x06':
                            break
                        # This should only happen if an error occurs
                        sys.stdout.write(char)
        return True
    except:
        return False
output = send_file_to_host('/flash/L76GNSS.py', None, 2040)
if output is None:
    print("None")
else:
    print(output)

-----
Traceback (most recent call last):
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\pennamen\AppData\Local\Programs\Python\Python38\Scripts\rshell.exe\__main__.py", line 7, in <module>
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\command_line.py", line 4, in main
    rshell.main.main()
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 2966, in main
    real_main()
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 2949, in real_main
    shell.cmdloop(cmd_line)
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1752, in cmdloop
    stop = self.onecmd(line)
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1785, in onecmd
    self.onecmd_exec("".join(group))
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1796, in onecmd_exec
    return cmd.Cmd.onecmd(self, line)
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\cmd.py", line 217, in onecmd
    return func(arg)
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 2211, in do_cp
    if not cp(src_filename, dst_filename):
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 664, in cp
    return src_dev.remote(send_file_to_host, src_dev_filename, dst_file,
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1497, in remote
    xfer_func(self, *args, **kwargs)
  File "c:\users\pennamen\appdata\local\programs\python\python38\lib\site-packages\rshell\main.py", line 1102, in recv_file_from_remote
    dst_file.write(binascii.unhexlify(write_buf[0:read_size]))
binascii.Error: Non-hexadecimal digit found

I looked at thisa exact issue back in March (on a private email thread) and the problem appears to stem from a bug in the PyCom firmware. The text of my email from the analysis was:
So it looks like there is a bug of some type in the WiPy version of micropython (and not a bug in rshell).

I updated my WiPy 3.0 board to the latest firmware which reports:
Pycom MicroPython 1.20.2.rc6 [v1.11-01f49f7] on 2020-02-28; WiPy with ESP32
Pybytes Version: 1.3.1

What I observed is that when I issued the cat /flash/main.py command I get the following data coming back on the serial port:

About to open /flash/main.py
bytes_remaining = 34
23206d61696e2e7079202d2d20707574
read_size = 16
Waiting for ACK
Got ACK
bytes_remaining = 18
20796f757220636f6465206865726521
read_size = 16
Waiting for ACK
Got ACK
bytes_remaining = 2
0d0a
read_size = 2
Waiting for ACK
Got ACK
True
\n
\x04\x04
>Guru Meditation Error: Core  0 paniced (StoreProhibited). Exception was unhandled.
 It should then go back to normal (aka friendly REPL mode) and instead its crashing.