CBS: all videos failing

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log
using an assetType of Downloadable (not DRM):
https://www.cbs.com/shows/video/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_/
[debug] Command-line args: ['--verbose', 'cbs:YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.4 (CPython) - Windows-10-10.0.18363-SP0
[debug] exe versions: ffmpeg 2022-06-09-git-5d5a014199-essentials_build-www.gyan.dev
[debug] Proxy map: {}
[CBS] YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_: Downloading XML
ERROR: YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_: Failed to parse XML  (caused by
ParseError('no element found: line 1, column 0'))

same with assetType of StreamPack (not DRM):
https://paramountplus.com/movies/video/wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_
[debug] Command-line args: ['--verbose', 'cbs:wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.4 (CPython) - Windows-10-10.0.18363-SP0
[debug] exe versions: ffmpeg 2022-06-09-git-5d5a014199-essentials_build-www.gyan.dev
[debug] Proxy map: {}
[CBS] wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_: Downloading XML
ERROR: wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_: Failed to parse XML  (caused by
ParseError('no element found: line 1, column 0'))

Description
looks like the issue is that these type URLs are now returning empty responses:
http://can.cbs.com/thunder/player/videoPlayerService.php?partner=cbs&contentId=wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_
I believe this started in the last day or two, not sure if temporary or permanent issue.

As you haven't shown a verbose log, it's difficult to diagnose: clearly you aren't running the release build or a new git version, as neither recognise the CBS URL.
However the same result occurs with yt-dlp.

As you haven't shown a verbose log, it's difficult to diagnose: clearly you aren't running the release build or a new git version, as neither recognise the CBS URL.

I did show a verbose log, what are you talking about? I ran with --verbose as instructed. Also if you look closely, you'll notice I didn't provide a URL, I provided the cbs:ID

Quite so, late at night in the UK, but you did omit the backtrace, so that's my weak excuse.
Using the cbs:ID, all three versions of interest (release yt-dl and yt-dlp, yt-dl git master) fail in the same way, as in the posted logs.
Someone (OP perhaps?) will have to do some research, eg on how this works in region in a browser, to fix this.

Looks like this is all blocked (403) from UK.

does it not work with a VPN ??

Possibly, but ATM I'll leave this to people able to access it directly. The question is whether OP's failure with cbs:... is repeatable given https://github.com/ytdl-org/youtube-dl/issues/31864#issuecomment-1473757202.
We now have ParamountPlus.com/gb but if it's anything like the offerings from the same group it'll be DRMed. Difficult to tell as even their device help page is broken,

Somewhat surprisingly no yt-dlp user has reported this. Could it be an artefact of the access network, eg if not in region or if using a blocked VPN?

If can.cbs.com means Canada, then it would make sense to try Canada VPN. I retried the PHP URL with United States and Canada, still failing.

and this test link https://www.paramountplus.com/movies/video/wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_/ says it's DRM protected.

Several videos offer both DRM and non DRM streams. For example, the one you've commented on, both DASH_CENC (DRM) and StreamPack (not DRM) are offered. However in addition to the PHP error already discussed, another one has recently occured. If you make a StreamPack request, such as:
curl -v link.theplatform.com/s/dJ5BDC/media/guid/2198311517/wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_

You get an HLS URL, as expected:
https://cbsios-vh.akamaihd.net/i/temp_hd_gallery_video/CBS_Production_Outlet_VMS/video_robot/CBS_Production_Entertainment/2020/05/07/1735196227871/0_0_3436402_ful01_2588_503000.mp4.csmil/master.m3u8
but if you actually try to request the HLS, you get:
HTTP/1.0 400 Bad Request

so it appears that all StreamPack options are dead, either temporarily or permanent, with the cause being an accident or a purposeful action by CBS. So until something changes any StreamPack options are essentially DRM.

the same video at paramount + https://www.paramountplus.com/shows/video/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_/ works with yt-dlp & ytdl-patched but does not work with youtube-dl.

this is a Downloadable type, which might be the only non DRM type still available. Note that previously an HLS_CLEAR type was available as well, but I think those have been dropped, not sure.

Someone (OP perhaps?) will have to do some research, eg on how this works in region in a browser, to fix this.

I plan to work on this some more this weekend. I want to see what a web client is doing on the few non DRM options left, and also check Android client. Also I have code that is able to download and decrypt the DRM options, but I assume that is still beyond the scope of these projects.

the same video at paramount + https://www.paramountplus.com/shows/video/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_/ works with yt-dlp & ytdl-patched but does not work with youtube-dl.

also I just noticed, YT-DLP does not work with this. From my testing is shows DRM for both URL, even though I have already proved at least one URL has a non DRM option.

Does this endpoint no longer work can.cbs.com/thunder/player/videoPlayerService.php ? I get status code 200 but no output...

From the issue text:

looks like the issue is that these type URLs are now returning empty responses:
http://can.cbs.com/thunder/player/videoPlayerService.php?partner=cbs&contentId=wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_

So, yes.

How to get all the pids for all asset types now for example DASH_CENC , PRECON or HLS ? link.theplatform.com needs pid to get  m3u or mpd manifest...

@forty875 thats the whole point of this issue. Currently you cannot get that information. possibly we could discover this by using an Apple web client (Safari), or impersonating one. Alternatively, if someone knows how to MITM iOS or iPadOS that should work as well

Won't a trace of playing a test video in a desktop browser do?

@dirkf regarding web clients, if you not using Safari, or a Safari User-Agent, or another Apple specific browser (I dont know what those would be), then likely you are just going to be served DASH, either Widevine or clear. So using an Apple client, either desktop or mobile is an important point here, for any who might care about HLS formats. I didn't make time for this over the last couple of days (busy with other stuff), but I will try to look at it this week, unless someone beats me to it.

For me the original XML parse error is a somewhat separate issue from whether they are DRM'd that cropped up this week after working fine --- I had been using yt-dlp to download the subtitles and thumbnails without the content via the --skip-download parameter. Started failking for this series Tuesday for me.
Examples:
"https://www.cbs.com/shows/video/yJDiL4UCGqhPMzYeI7_uGETDWFuGRNui/"
"https://www.cbs.com/shows/video/TTobd7NjENoNESGeCpmsirueC4HOYDge/"
"https://www.cbs.com/shows/video/mjADR3rryehJKHRKqapazoYY0UX_6_F_/"
It's also retroactive with previously working videos apparently now failing, so an unresponsive endpoint seems logical to me.
Hope folks can help figure something out.

For the Colbert show, I get these interesting JS fragments in the non-JS page seen by yt-dl, after pretty-printing:
Colbert fragments
player.metaData = {
  'airdate_iso': '2023-03-29T20:35:00-07:00',
  'airdate_tv': '2023-03-29',
  'assetType': 'DASH_CENC',
  'brand': 'CBS',
  'mediaType': 'Full Episode',
  'channelName': null,
  'contentId': 'mjADR3rryehJKHRKqapazoYY0UX_6_F_',
  'contentUrl': '/shows/video/mjADR3rryehJKHRKqapazoYY0UX_6_F_/',
  'endCreditsChapterTime': 2430,
  'episodeNumber': '94',
  'ESTURLs': {
    'amazon': '',
    'iTunes': ''
  },
  'excludeOztam': null,
  'fullEpisode': true,
  'isServiceAllowed': true,
  'oztamMediaId': null,
  'pid': 'Z8rFLBAAODE5',
  'daistreamKey': null,
  'previewImageURL': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/colbert_1396_2141418_1920x1080.jpg',
  'rating': 'TV-14',
  'regionalRatings': [
    {
      'region': 'US',
      'rating': 'TV-14',
      'disclaimer': null,
      'secondaryDescriptors': null,
      'subratings': null,
      'consumerAdvice': null,
      'ratingIcon': null
    }
  ],
  'seasonNumber': '8',
  'seriesTitle': 'The Late Show with Stephen Colbert',
  'showPageURL': 'shows/the-late-show-with-stephen-colbert/video',
  'subscriptionLevel': 'FREE',
  'thumbnail': {
    'height': 1080,
    'width': 1920,
    'assetType': 'Thumbnail',
    'url': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/colbert_1396_2141418_1920x1080.jpg'
  },
  'thumbnailSheet': {
    'height': 126,
    'width': 54656,
    'assetType': 'ThumbSheet',
    'url': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/COLBERT_1396_2997DF_1920X1080_178_2CH_PRORESHQ_2141419_vr_cs_224x126_10_244.jpg'
  },
  'tvRatingFlag': true,
  'videoLength': 2439,
  'videoPageURL': '/shows/video/mjADR3rryehJKHRKqapazoYY0UX_6_F_/',
  'videoTitle': 'The Late Show - 3/29/23 (Taron Egerton, Jay Chandrasekhar)',
  'label': '3/29/23 (Taron Egerton, Jay Chandrasekhar)',
  'videoProperties': [
    'Branded',
    'Current Season',
    'Episodeless'
  ],
  'playbackEvents': {
    'endCreditChapterTimeMs': 2430000,
    'previewStartTimeMs': null,
    'previewEndTimeMs': null,
    'openCreditEndTimeMs': null,
    'openCreditStartTime': null
  }
};
player.apiMetadata = {
  'genre': 'Talk',
  'status': 'AVAILABLE',
  '_firstIngestDate': '03/30/23 02:26 EDT',
  'expired': false,
  'showPageUrl': 'shows/the-late-show-with-stephen-colbert/video',
  'cbsShowId': 61456254,
  'primaryCategory': '518039619867',
  'primaryCategoryName': 'Late Night/Late Show/Colbert/Full Episodes',
  'editDate':<PHONE_NUMBER>000,
  '_editDate': '03/30/23 02:25 EDT',
  '_lastModifiedDate': '03/30/23 02:26 EDT',
  'ingestDate':<PHONE_NUMBER>000,
  'airDate':<PHONE_NUMBER>000,
  'description': 'Stephen welcomes actor Taron Egerton and comedian Jay Chadrasekhar.',
  'shortDescription': 'Stephen welcomes actor Taron Egerton and comedian Jay Chadrasekhar.',
  'pubDate':<PHONE_NUMBER>000,
  'label': '3/29/23 (Taron Egerton, Jay Chandrasekhar)',
  'videoPageUrl': 'shows/the-late-show-with-stephen-colbert/video/mjADR3rryehJKHRKqapazoYY0UX_6_F_/the-late-show-3-29-23-taron-egerton-jay-chandrasekhar-',
  'url': 'shows/the-late-show-with-stephen-colbert/video/mjADR3rryehJKHRKqapazoYY0UX_6_F_/the-late-show-3-29-23-taron-egerton-jay-chandrasekhar-',
  'assetType': 'DASH_CENC',
  'firstIngestDate':<PHONE_NUMBER>000,
  'category': '',
  'topLevelCategory': 'Entertainment',
  'fullEpisode': true,
  'exclusive': false,
  'contentId': 'mjADR3rryehJKHRKqapazoYY0UX_6_F_',
  'title': 'The Late Show - 3/29/23 (Taron Egerton, Jay Chandrasekhar)',
  '_ingestDate': '03/30/23 02:25 EDT',
  'episodeNum': '94',
  '_pubDate': '03/30/23 02:26 EDT',
  'seasonNum': '8',
  'brand': 'CBS',
  'pid': 'Z8rFLBAAODE5',
  'seriesTitle': 'The Late Show with Stephen Colbert',
  '_airDate': '03/29/23 23:35 EDT',
  'duration': 2439,
  'lastModifiedDate':<PHONE_NUMBER>000,
  'rating': 'tv-14',
  'deviceType': 'PC',
  'thumbnail': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/colbert_1396_2141418_1920x1080.jpg',
  'amazonESTURL': '',
  'itunesESTURL': '',
  'streamingUrl': 'https://vod-gcs-cedexis.cbsaavideo.com/intl_vms/2023/03/05/2178777667682/2141560_cenc_dash/stream.mpd',
  'expirationDate':<PHONE_NUMBER>000,
  '_expirationDate': '04/06/23 03:35 EDT',
  '_pubDateISO': '2023-03-29T23:26:01-07:00',
  '_airDateISO': '2023-03-29T20:35:00-07:00',
  'playerLocUrl': 'http://can.cbs.com/thunder/player/chrome/canplayer.swf?allowScriptAccess=always&allowFullScreen=true&pid=Z8rFLBAAODE5&partner=cbs_us_free_desktop&autoPlayVid=false&owner=CBS Production Entertainment',
  'subscriptionLevel': 'FREE',
  'mediaAvailableDate': '2023-03-30T00:35:00-0700',
  'mediaAvailableDateEpoch':<PHONE_NUMBER>000,
  'closedCaptionUrl': 'https://vod-gcs-cedexis.cbsaavideo.com/intl_vms/caption-rmp/COLBERT_1396_2997_1680155824.xml',
  'isLive': false,
  'cmsAccountId': 'dJ5BDC',
  'appleWatchListShowKey': 'SHOW_61456254_The-Late-Show-with-Stephen-Colbert',
  'isProtected': true,
  'mediaType': 'Full Episode',
  'excludeNielsenTracking': false,
  'endCreditsChapterTime': '40:30',
  'thumbnailSet': [
    {
      'height': 1080,
      'width': 1920,
      'assetType': 'Thumbnail',
      'url': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/colbert_1396_2141418_1920x1080.jpg'
    }
  ],
  'downloadCountrySet': [
  ],
  'regionalRatings': [
    {
      'region': 'US',
      'rating': 'TV-14',
      'disclaimer': null,
      'secondaryDescriptors': null,
      'subratings': null,
      'consumerAdvice': null,
      'ratingIcon': null
    }
  ],
  'premiumFeatures': [
    ''
  ],
  'videoProperties': [
    'Branded',
    'Current Season',
    'Episodeless'
  ],
  'availableForProfileTypes': [
    'ADULT'
  ],
  'playbackEvents': {
    'endCreditChapterTimeMs': 2430000,
    'previewStartTimeMs': null,
    'previewEndTimeMs': null,
    'openCreditEndTimeMs': null,
    'openCreditStartTime': null
  },
  'embeddable': 'false',
  'copyright': 'CBS, Inc.',
  'addOns': [
  ],
  'isContentAccessibleInCAN': true,
  'brandSlug': 'cbs',
  'tmsprogramID': 'EP019062761435',
  'tmsseriesID': 'SH019062760000',
  'thumbnailSheetSet': [
    {
      'height': 126,
      'width': 54656,
      'assetType': 'ThumbSheet',
      'url': 'https://thumbnails.cbsig.net/CBS_Production_Entertainment_VMS/2023/03/05/2178777667682/COLBERT_1396_2997DF_1920X1080_178_2CH_PRORESHQ_2141419_vr_cs_224x126_10_244.jpg'
    }
  ],
  '__FOR_TRACKING_ONLY_MEDIA_ID':<PHONE_NUMBER>707
};
player.tms_program_id = 'EP019062761435';
// Set player.drm here/inline unless promise for client-side AJAX
if (!window.CBS.Registry.drmPromise) {
  player.drm = {
    'enabled': true,
    'sessionId': '',
    'ticket': '',
    'name': 'Widevine',
    'widevine': {
      'url': 'https://cbsi.live.ott.irdeto.com/widevine/getlicense?CrmId=cbsi&AccountId=cbsi&SubContentType=Default&ContentId=mjADR3rryehJKHRKqapazoYY0UX_6_F_',
      'header': {
        'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsImtpZCI6IjNkNjg4NGJmLWViMDktNDA1Zi1hOWZjLWU0NGE1NmY3NjZiNiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhbm9ueW1vdXNfVVMiLCJlbnQiOlt7ImJpZCI6IkFsbEFjY2Vzc01haW4iLCJlcGlkIjo3fV0sImlhdCI6MTY4MDE5NTgzNSwiZXhwIjoxNjgwMjAzMDM1LCJpc3MiOiJjYnMiLCJhaWQiOiJjYnNpIiwiaXNlIjp0cnVlLCJqdGkiOiJjZjhhZDc3NS03YTA3LTQyY2YtYjM2OC03M2NhZDM1ZDMyMWEifQ.OGbUVBCX5e1oVsDbbpglZuqUCxgjS7kWvz9f0w0lTh0'
      }
    }
  };

Looks like no Colbert for yt-dl.

No, but as the closed captioning jpeg and thumbnail are correctly published it seems I would have expected yt-dlp to move on from the XML error with --skip-download and grab those. But I'm largely out of my depth here.

I am still quite busy with other stuff (just finished a new binary to text encoding and an HTTP package this week), but I will see if I can make some time this weekend. I wanted to work on my ProtoBuf package, but I suppose this is more important. I did try to reverse engineer the app secret with no luck. Its only 64 bit, so looks like it might just be a non cryptographic hash, but I tried CRC64, FNV and MapHash with no luck. If anyone knows any other common 64 bit hashes I can try those. Thanks

OK so continuing with this URL:
https://paramountplus.com/movies/video/wQH9yE_y_Dt4ekDYm3yelhhY2KXvOra_
based on this:
https://whatismybrowser.com/guides/the-latest-user-agent/safari
I changed my User-Agent to:
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15

If I then visit the Paramount+ page above, I get this:

Sorry, this video is currently unavailable. Please try again.
Need additional help? Check out our troubleshooting tips here.
error code: 5000

So it seems HLS is no longer an option for certain videos, making those specific videos DRM only. Note the Downloadable videos are still available without DRM, for example:
https://paramountplus.com/shows/video/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_

take the Halo TV series for instance -https://www.paramountplus.com/shows/halo/ , none of the the season episodes will download, but any of the behind the scenes, special features & clips will download fine.

take the Halo TV series for instance -https://www.paramountplus.com/shows/halo/ , none of the the season episodes will download, but any of the behind the scenes, special features & clips will download fine.

@october262 yes they will download, just not with the typical tools:
> paramount -b AmBcaCActcTer6wa7QjKWWzdtKrMlHBw -a 2
GET http://link.theplatform.com/s/dJ5BDC/media/guid/2198311517/AmBcaCActcTer6wa7QjKWWzdtKrMlHBw?format=preview
GET https://www.paramountplus.com/apps-api/v3.0/androidphone/irdeto-control/anonymous-session-token.json?at=ABAAAAAAAAAAAAAAAAAAAAAAj3mOj6dirS7jbI4bsY6aO5RdLzKfKzZpmuy1mHn%2BKFQ%3D
GET http://link.theplatform.com/s/dJ5BDC/media/guid/2198311517/AmBcaCActcTer6wa7QjKWWzdtKrMlHBw?assetTypes=DASH_CENC&formats=MPEG-DASH
GET https://vod-gcs-cedexis.cbsaavideo.com/intl_vms/2022/03/16/2013439043562/1368394_cenc_dash/Halo_101_Final_PayTV_SDR_ProResHQ_422_2398_3840x2160_200_51_LoRo_28Mar2022_FRMTD_PPlus_BCAST_en-US_1262472_3_aac_128/init.m4v
POST https://cbsi.live.ott.irdeto.com/widevine/getlicense?CrmId=cbsi&AccountId=cbsi&SubContentType=Default&contentId=AmBcaCActcTer6wa7QjKWWzdtKrMlHBw
1.99%   1.15 megabyte   1.12 megabyte/s

OK I think I made some progress. If you monitor the Android app, you see a request like this:
GET https://www.paramountplus.com/apps-api/v2.0/androidphone/video/cid/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_.json?at=ABB90oFocIWpreOn1w0QqC%2FEi1lZTogArlwKOuD7ZmbjVsg7oCYtHqHGM1UFjJU8dW0%3D HTTP/2.0
cookie: CBS_COM=N0EwMjY0MDVENTU3MzJCNzJBMEQzMkIyMDQ0MjQyQUU6MTcxMTkxODg2NTMyND...

In the response is this:
"playerLocUrl": "http://can.cbs.com/thunder/player/chrome/canplayer.swf?allowScriptAccess=always&allowFullScreen=true&pid=u4uKImT6OqQR&partner=pp_us_lcp_android&autoPlayVid=false&owner=CBS Production News",

that URL is actually invalid because of the spaces, here is correct:
http://can.cbs.com/thunder/player/chrome/canplayer.swf?allowScriptAccess=always&allowFullScreen=true&pid=u4uKImT6OqQR&partner=pp_us_lcp_android&autoPlayVid=false&owner=CBS Production News
If you extract that file, you get another file canplayer~.swf. If you extract that file, the largest file should be named something like 15.82. Inside that file is some interesting stuff:
http://can.cbs.com/thunder/player/videoPlayerService.php?pid=$PID$&domain=$DOMAIN$&partner=$PARTNER$
http://can.cbs.com/thunder/player/videoPlayerService.php?pid=$PID$&domain=$DOMAIN$&partner=$PARTNER$&auth=$AUTH$

thats as far as I have gotten so far.

In the Colbert JSON quoted above:

'playerLocUrl': 'http://can.cbs.com/thunder/player/chrome/canplayer.swf?allowScriptAccess=always&allowFullScreen=true&pid=Z8rFLBAAODE5&partner=cbs_us_free_desktop&autoPlayVid=false&owner=CBS Production Entertainment',

This just gives an opaque binary file with initial bytes CWS. Apparently it's a compressed SWF which can then be edited with a suitable SWF tool imported from prehistory, or even just a hex editor, but appears to be generic.

If it helps I just used this to extract
https://github.com/mcmilk/7-Zip-zstd

OK I did some more research, I did not find any method to enumerate all assetType values:
https://github.com/4cq2/mech/blob/v1.2.8/paramount/assetType.md
So it seems for now some values are hidden, only available to people who already know they exist. Downloadable is still a valid value, so some videos are still available without DRM.

Somehow the web and Android app manage to construct a playable URL, regardless of this (possibly requiring WV/FP). How does that happen? What do playable URLs look like?

Somehow the web and Android app manage to construct a playable URL, regardless of this (possibly requiring WV/FP). How does that happen? What do playable URLs look like?

Not really sure what you are asking here. The web client always chooses DRM, even if non DRM is available. In fact, it will prevent you from playing at all if you disable DRM browser plugin.
Regarding the Android client, as mentioned in the research document in my previous comment, the Android client only returns DRM options, even if non DRM options are available.
So regarding "playable URLs", it doesnt really matter what methods the official clients use to general playable URLs (at least as far as YouTube-DL/YT-DLP are concerned), as all playable URLs generated by official clients are DRM enabled. Again to be clear, non DRM options are still available, but they are more limited than ever, and it appears no method is currently available to even enumerate what non DRM options might be available. I didnt any research into the auth query string key, so its possible that might be the fix. If I had to guess, I would say its the same as the LS_Session value, but I didn't check that. My previous research document was nearly exhaustive, so I am about mentally checked out on this matter.

Thanks: you answered the question anyway!
Some other platforms have legacy implementations that can't be updated but have to continue playing content: perhaps these limited non-DRM options are provided for such a reason.

one note - currently I think the only non DRM assetTypes value is Downloadable. I did find some examples with mediaType of Full Episode:
https://paramountplus.com/shows/video/YxlqOUdP1zZaIs7FGXCaS1dJi7gGzxG_
but none with mediaType of Movie, so its possible that all Movie are now DRM only. I am not sure of anything I just said, so if someone can prove me wrong, I will be happy to hear it.

For this video:
https://www.cbsnews.com/video/sunday-morning-full-episode-4-2-2023/
We just need to parse index.html to find the URL to master.m3u8 file.  Snippet:
Sunday Morning Full Episode 4/2","timestamp":1680440400000,"duration":3760,"durationLabel":"01:02:40","label":null,"images":{"sd":"https://assets1.cbsnewsstatic.com/hub/i/r/2023/04/02/a457a56d-498d-420b-9ba1-d91fab2ee07b/thumbnail/640x360/6bd395cc9a6be06149ad04014c1d9296/smjanepauley040223-1848688-640x360.jpg","hd":"https://assets3.cbsnewsstatic.com/hub/i/r/2023/04/02/a457a56d-498d-420b-9ba1-d91fab2ee07b/thumbnail/1280x720/58ec9371cb82766448dfed42a78b2182/smjanepauley040223-1848688-640x360.jpg"},"previewUrl":"https://splice.amlg.io/api/v2/video/A0u8tIz70VHrE_UwZAjPeMOAa54TH4T5/preview/","video":"https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/master.m3u8","video2":"https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/master.m3u8","format":"application/x-mpegURL","url":"https://www.cbsnews.com/video/sunday-morning-full-episode-4-2-2023/
If we fetch https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/master.m3u8 - it contains metadata on all the video and audio formats.  Snippet:
#EXT-X-STREAM-INF:BANDWIDTH=634827,AVERAGE-BANDWIDTH=516066,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=29.970,VIDEO-RANGE=SDR,AUDIO="audio_aac",CLOSED-CAPTIONS=NONE
0402_SUNMO_FULL_1_1848685_375/stream.m3u8
That SUNMO_FULL.../stream.m3u8 is the video playlist:
Video/audio links (VLC can play) are:
AAC Audio:  https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_en-US_1848685_aac_128/stream.m3u8
640x360:  https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_1848685_375/stream.m3u8
768x432:  https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_1848685_750/stream.m3u8
960x540:  https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_1848685_1500/stream.m3u8
1280x720: https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_1848685_2100/stream.m3u8
HLS 1920x1080:  https://prod.vodvideo.cbsnews.com/cbsnews/vr/hls/2023/03/22/2185212995641/1848685_hls/0402_SUNMO_FULL_1_1848685_3000/stream.m3u8

@Tetracerus is now blocked, they wont be able to comment further. Please pay attention, this is about CBS and Paramount+, NOT CBS News. the difference is important. If you have further on topic comments, please contact me privately to be unblocked.


Discount Codes model

Unfortunately there has been another merge request done before this which caused conflict. Please pull the changes from upstream and fix the conflict.
If you leave your code at the end of the file, leave last line blank, so the code is according to PEP8

Issues fixed, discount types added.

@TomasHo did you commit correct version? it seems you have removed the class at all, commit 6ea846526d is removing whole class DiscountCodes(models.Model)

Sorry, should be ok now.

@TomasHo sure thanks, merging the code into upstream.
For the future at the end of the file, leave last line blank, so the code is according to PEP8


Add Restart Command

Add a Restart Command for Process Manager 2.
I do not have Process Manager 2 installed, so I would like you to check that it does restart

@Fredthedoggy Is this still needed? Who can review it do you think?


Robert Longson: Watch percentagescalc.com

Robert Longson requests the watch of the watch_keyword percentagescalc\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

percentagescalc\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Makyen in Charcoal HQ


TextureAtlas fail - Error: Image height must be greater than 0.

Godot version
v3.4.stable.official [206ba70f4]
System information
Windows 7 64bits GLES3
Issue description
When importing some textures as TextureAtlas, it fails.
Gives all those errors in console (since the Image height one is the first, I thought it was appropiate to put it in the issue's title) :
ERROR: Image height must be greater than 0.
at: (core/image.cpp:1743)

ERROR: Condition "data.size() == 0" is true.
at: (core/image.cpp:2543)

WARNING: png_image_write_to_memory: invalid argument
at: check_error (drivers/png/png_driver_common.cpp:56)

ERROR: Condition "compressed_size <= png_size_estimate" is true. Returned: FAILED
at: image_to_png (drivers/png/png_driver_common.cpp:186)

ERROR: Can't convert image to PNG.
at: (drivers/png/resource_saver_png.cpp:54)

ERROR: Invalid image.
at: (scene/resources/texture.cpp:190)

ERROR: Can't find file 'res://atlas/atlasFAIL.png'.
at: (editor/editor_file_system.cpp:1668)

Minimal reproduction project included.
Note: I already thought at first that it could be something that went wrong when saving those images, but I already tried to export them in different softwares (aseprite, krita, photoshop) to be sure, in case there was any hidden problem in the file or something. But even with those exported files from different programs, I get the exactly same result. The image seems to be perfect.
In fact, I was able to replicate it by making more images that have the same sizes (both for the image size, and the inner rect (that contains just the alpha of the image).
After the confirmation on reddit by some users, I thought that it was worthy to open an issue.
Here is a guideline for recreating more images that fails:

Extending the inner box just one pixel (image at the bottom), makes it work properly. But the top image fails.
Steps to reproduce

Open the project
Open the folder "images_fail" or "images_stack"
Re-import all the images of any of those folders as a TextureAtlas
See how it fails*

*NOTE:
"images_fail" folder contains only just the images with the specific dimensions that seems to be causing the problem.
"images_stack" generates an AtlasTexture, but isn't working properly (images are stacked). Image:

Minimal reproduction project
AtlasIssue.zip

I've been testing with different sizes that also fails (cropping the image, changing the inner rect of the sprite, etc).
So, in case it helps to identify the bug or to test it, I upload it here:
doomed_cursed_slide_image_of_the_demon.zip
Its separated in 2 main folders, one containing a set that works, and the other has some different sets with diferent sizes that triggers the bug.

The issue is in this method:
https://github.com/godotengine/godot/blob/b0e93711b36bae3d2bb3a7e9a4a88faf055499fb/editor/editor_atlas_packer.h#L72
It's being called in here. It seems it's not always working properly forp_cell_resolution > 1.  For p_cell_resolution = 1 all test cases above seems to work fine. The p_cell_resolution parameter is used to perform packing on a p_cell_resolution times smaller bitmaps (making the whole packing process quicker). But it seems like some logic is incorrect for some cases when dealing with shrinked bitmaps (p_cell_resolution > 1). Haven't pinpoint the exact issue(s) with the current logic yet.


Create Balances RPC

The account data (and with it the accounts storage) has changed a few times in the last year which has lead to some confusion for readers (e.g. wallets, exchanges) that did not react to the metadata changes quickly.
It would be useful to have a balances RPC endpoint that returns the balance of a user account independent of the underlying storage (and that changes less frequently).
We basically want to expose similar functions to free_balance via RPC.
Suggested RPC endpoints:

free_balance
reserved_balance
account_data

Is it possible to make those RPC a subscription in order to support subscribe balance changes?

Hello, I see no one is assigned to this. I would like to hack on this issue, is there any etiquette I must know? Shall I just begin writing tests and code?

Cool that you want to contribute!
I think it's fine to just start hacking and open a PR. I would suggest opening a draft PR early to get feedback on the general design so you are less likely to paint yourself into a corner.

Feel free to tag me in the PR. Maybe @shawntabrizi and @xlc would also be willing to pitch in.
If you want to go the subscription route (which will make this a bit more complex AFAICT) maybe this PR might help as a reference: https://github.com/paritytech/substrate/pull/5732


chore(pre-bundle-new-url): minor tweak

#613  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @hi-ogawa and the rest of your teammates on  Graphite


Opprette dct:relation

#52
"Range: rdfs:Resource
This property refers to a related resource. "

Gjelder versjon 1.1


Oracle-CX-PY3 Image for a new Oracle Integration

Status
Ready/In Progress/In Hold (Reason for hold)
Related Issues
related: link to the issue
Description
A docker container for a new Oracle Integration that I am currently coding.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: aymansm:x: ayman-mYou have signed the CLA already but the status is still pending? Let us recheck it.

Hi @ayman-m,
Can you provide info about the integration you are planning to build?

I am closing this. Please use demisto/genericsql image which contains the oracle driver and python package.


Don't stop checking for updates after one has been installed.

fixes #44685
Not sure of what else I need to do for this pr...
Couldn't see any tests but I assume either the pr build will fail or hopefully someone will let me know.

any thoughts on this @sandy081, @joaomoreno ??

Also this doesn't for macOS, Squirrel won't allow to download two updates.

Also this

Closing due to inactivity.


Undo implementation

Hello,
I'm getting unexpected results from my undo implementation. This is how I'm doing it:
1. Initialize (global) empty array of strokes:
let strokes = []
2. When strokerecorded is fired, push stroke to array:
atrament.addEventListener('strokerecorded', function(obj) {
    if (!atrament.recordPaused) {
        strokes.push(obj.stroke);
    }
});

3. When the undo button is pressed, do the following:
function undoStroke() {
    strokes.pop();
    atrament.clear();
    atrament.recordPaused = true;
    for (let stroke of strokes) {
        // set drawing options
        atrament.mode = stroke.mode;
        atrament.weight = stroke.weight;
        atrament.smoothing = stroke.smoothing;
        atrament.color = stroke.color;
        atrament.adaptiveStroke = stroke.adaptiveStroke;

        // don't want to modify original data
        const points = stroke.points.slice();

        const firstPoint = points.shift();
        // beginStroke moves the "pen" to the given position and starts the path
        atrament.beginStroke(firstPoint.x, firstPoint.y);

        let prevPoint = firstPoint;
        while (points.length > 0) {
            const point = points.shift();

            // the `draw` method accepts the current real coordinates
            // (i. e. actual cursor position), and the previous processed (filtered)
            // position. It returns an object with the current processed position.
            const { x, y } = atrament.draw(point.x, point.y, prevPoint.x, prevPoint.y);

            // the processed position is the one where the line is actually drawn to
            // so we have to store it and pass it to `draw` in the next step
            prevPoint = { x, y };
        }

        // endStroke closes the path
        atrament.endStroke(prevPoint.x, prevPoint.y);
    }
    atrament.recordPaused = false;
}

But this is what I get:

Notice how after I press undo, the last line disappears, as expected, but the other two lines are extended a little bit.
Can you help me with this?
I'm on Linux. This happens on both Firefox 72.0.1 and Google Chrome 89.0.4389.90.
Thank you for the library!

@nidoro thank you for submitting the issue! Apologies for only getting to this now, I haven't been able to dedicate time to this lib for a while. I'll test your use case, try to reproduce the bug and fix it in the next few weeks.

Thanks for this example @nidoro . I modified it slightly to add support for fills, and fixed a few things that I assume have changed since your post (Point.point.x instead of point.x, ...). Not battle tested or anything but might save someone else a bit of trouble.
        this.sketchpad.addEventListener('strokerecorded', (obj) => {
            if (!this.sketchpad.recordPaused) {
                obj.stroke.type = "stroke";
                this.strokes.push(obj.stroke);
            }
        });

        this.sketchpad.addEventListener('fillstart', ({ x, y }) => {
            var obj = {};
            obj.type = "fill";
            obj.fillColor = this.sketchpad.color;
            obj.x = x;
            obj.y = y;
            this.strokes.push(obj);
        });

undo(){
            this.strokes.pop();
            this.sketchpad.clear();
            this.sketchpad.recordPaused = true;
            for (let stroke of this.strokes) {
                if (stroke.type == "stroke"){
                    // set drawing options
                    this.sketchpad.mode = stroke.mode;
                    this.sketchpad.weight = stroke.weight;
                    this.sketchpad.smoothing = stroke.smoothing;
                    this.sketchpad.color = stroke.color;
                    this.sketchpad.adaptiveStroke = stroke.adaptiveStroke;

                    // don't want to modify original data
                    const points = stroke.points.slice();

                    const firstPoint = points.shift().point;
                    // beginStroke moves the "pen" to the given position and starts the path
                    this.sketchpad.beginStroke(firstPoint.x, firstPoint.y);

                    let prevPoint = firstPoint;
                    while (points.length > 0) {
                        const point = points.shift();

                        // the `draw` method accepts the current real coordinates
                        // (i. e. actual cursor position), and the previous processed (filtered)
                        // position. It returns an object with the current processed position.
                        const { x, y } = this.sketchpad.draw(point.point.x, point.point.y, prevPoint.x, prevPoint.y);

                        // the processed position is the one where the line is actually drawn to
                        // so we have to store it and pass it to `draw` in the next step
                        prevPoint = { x, y };
                    }

                    // endStroke closes the path
                    this.sketchpad.endStroke(prevPoint.x, prevPoint.y);
                } else {
                    //setTimeout(() => {console.log("filling");this.sketchpad._floodFill(stroke.x, stroke.y, stroke.color); }, 100);
                    this.sketchpad.color = stroke.fillColor;
                    const startColor = Array.from(this.sketchpad.context.getImageData(stroke.x, stroke.y, 1, 1).data);
                    this.sketchpad._floodFill(stroke.x, stroke.y, startColor);
                }

            }
            this.sketchpad.recordPaused = false;
        }

@nidoro I may be nearly 3 years late, but I believe this bug has now been fixed :) it looks like the stroke segments recorded when beginStroke and endStroke are called were redundant. Specifically the segment recorded during endStroke caused the re-drawn stroke to be slightly longer than the original.
I am now working on a big update which will be released as v4 in the next few weeks. This fix will be part of it.
Thank you again for the bug report and apologies for being very, very late!

Hi @jakubfiala, no need for apologies, that's the nature of open source. I'm currently not using atrament in any of my projects, but it's good to know that, if I ever need it, this will be fixed. Thank a lot!

@feored This works pretty well but there is one things that might trip some people up.
The redraw loop will set the brush style for each undo and redo which is correct but then it will leave the brush style in that state when you next go to draw again.
Maybe this is a feature not a bug but you will most likely want to restore the original brush styles to what they were when you started redrawing the history.
Here is my implementation...
 // Store original brush settings
      const original = {
        mode: annotations.mode,
        weight: annotations.weight,
        smoothing: annotations.smoothing,
        color: annotations.color,
        adaptiveStroke: annotations.adaptiveStroke,
      };

      annotations.clear();
      annotations.recordPaused = true;

      const history = historyMap.get(pageNumber) || [];

      // Replay all strokes up to the target index
      for (let i = 0; i <= targetIndex; i++) {
        const stroke = history[i];
        if (!stroke?.segments?.length) continue;

        // Set stroke properties
        annotations.mode = stroke.mode;
        annotations.weight = stroke.weight;
        annotations.smoothing = stroke.smoothing;
        annotations.color = stroke.color;
        annotations.adaptiveStroke = stroke.adaptiveStroke;

        // Create a copy of segments to avoid modifying original data
        const segments = [...stroke.segments];
        const firstPoint = segments.shift().point;
        annotations.beginStroke(firstPoint.x, firstPoint.y);

        let prevPoint = firstPoint;
        while (segments.length > 0) {
          const point = segments.shift().point;
          const { x, y } = annotations.draw(
            point.x,
            point.y,
            prevPoint.x,
            prevPoint.y
          );
          prevPoint = { x, y };
        }

        annotations.endStroke(prevPoint.x, prevPoint.y);
      }

      annotations.recordPaused = false;

      // Restore original brush settings
      // timeout because it breaks without?!
      setTimeout(() => {
        annotations.mode = original.mode;
      }, 100);
      annotations.weight = original.weight;
      annotations.smoothing = original.smoothing;
      annotations.color = original.color;
      annotations.adaptiveStroke = original.adaptiveStroke;
      ```

Interestingly if the original mode was eraser and you undo a brush stroke it won't work properly unless the restoring of the mode is in a timeout, kind of weird, any ideas @jakubfiala ?


Conduct UAT of eCR data store

UAT Plan

[x] Schedule UAT session w/ Allene
[x] Conduct UAT session w/ Allene
[ ] Check in on async UAT over the week of August 7-11
[ ] Schedule UAT session w/ Jordan
[ ] Conduct UAT session w/ Jordan
[ ] Schedule UAT session w/ PJ
[ ] Conduct UAT session w/ PJ

Messages Jordan on LAC Teams about time to meet on Hep A datamart. Waiting to hear back.


When can I go back to school?

                          `

I'm sorry there is no fix for that in version 2020. Kindly refer to commits made in 2019 to trace the error. Thanks


Keywords in originals get propagated in export

*** Neat piece of software - thank you for developing and sharing it ***
If an original in macOS photo app has keywords (set before importing by some other program) and these keywords were later removed within photos, they still appear in the exported files.
used command line:
osxphotos export --update --touch-file --exiftool --ignore-date-modified --album-keyword --cleanup --filename {created.date}_{created.hour}.{created.min}.{created.sec} --directory {created.year}/{created.mm} .
Possible reason:

Export obtains the original stored in the App Library
macOS photos does not change the original (in fact if you export the original using Photos you get the original keywords)
exiftool appends the keywords
keywords appear in multiple places "Subject", "Tags List", "Hierarchical Subject" and "Keywords"

Possible solution:

Remove pre-existing keywords in the file (ExifTool can do this with proper options)
even better add a flag to control if pre-existing keywords should be removed or not

I'll look into this.  --exiftool writes keywords to the following fields:

IPTC:Keywords
XMP:TagsList
XMP:Subject

The documentation should be explicit about what it will do and a flag to control this would be useful. I'm in favor of never deleting metadata unless explicitly told to do so.  Thus maybe a --overwrite-exif flag to overwrite existing EXIF data and a documented default of appending it.

In my testing, when run with --exiftool keywords in the original file's metadata do get overwritten. Thus duplicates are coming from somewhere else (maybe album name, person name, etc).  When setting keywords via the --exiftool  or --keyword-template options, it's the equivalent of doing this:
exiftool -ITPC:Keyword=keyword1 -IPTC:Keyword=keyword2... which does overwrite the keywords already in the file.
That said, you can merge them already using the exiftool template:
osxphotos export ~/Desktop/export -V --exiftool --keyword-template "{keyword}" --keyword-template "{exiftool:IPTC:Keywords}" --keyword-template "{exiftool:XMP:TagsList}"
This uses the template system to create exif keywords from the Photos keywords and the existing keywords in the file.

Given the behavior is already to overwrite the existing metadata, maybe an option for --merge-exifdata that would cause osxphotos to read all exifdata on the original file then merge it before writing it back to the file.

This is interesting. I checked and yes, the keywords are generally overwritten also for me. I have pinpointed one of the files that have the issue and probably it is an issue with the original metadata.
If I try to export a json sidecar for the json file it shows the expected metadata and the jpg file shows the metadata of the original file. This is good.
If I try to apply the metadata with exiftool -json=picture.jpg.json picture.jpg then I get an error
Error: [minor] MakerNotes offsets may be incorrect (fix or ignore?) - <IMAGE_FILENAME>
0 image files updated
1 files weren't updated due to errors
Adding either the exiftool flag -m (ignore minor error) or -F (fix) helps here.
Here a new hypothesis on the problem: exiftool triggers an error that is somehow silently ignored in spite of error handling code in the exiftool wrapper and does not update the metadata for the file.
If this is the issue then the solution would probably be one or more of:

understand why the error is silently ignored and report it
allowing to set additional exiftool flags on the command line of osxphotos (e.g., the -m or -F), since fixing the original file in photos might be hard

Are you able to share the image and the associated json file? If so, you can email to<EMAIL_ADDRESS>I'd like to see if it's a problem with the jpeg image or the output of osxphotos.
I'll play around with -m and -F to see if it makes sense to add those as standard options.  That makes me a little nervous as it could potentially cover a bug in osxphotos but I see the point that it would be best to make sure the --exiftool works even if there's a problem in the original photo.

Hi @RhetTbull,
Took me a bit to answer as I was running a few more tests on the whole library. I have exported the entire library with json sidecar files and then with an own script called exiftool -json=picture.jpg.json picture.jpg for each file.
Have discovered the following:

I have a number of files with corrupted metadata that exif with standard options (i.e. no -m, no -F) can manage, it triggers a warning that is ignored by osxphotos -> suggestion capture warnings and report them in the summary at the end of the export
There were a handful of files (ca. 0.01% of the library) with corrupted metadata that triggered errors with exiftool:
Error: [minor] MakerNotes pointer references previous MakerNotes directory
Error: [minor] MakerNotes offsets may be incorrect (fix or ignore?)
Error: [minor] Different 'rdf:about' attributes not handled
I all cases calling with -m allows proceeding. Calling -F fixes the files so that successive calls of exiftool do not need -m -> suggestion capture errors and report them at the end of the export, possibly also offer the option to pass -m and/or -F
The test also highlighted a number of originals with the wrong extension (e.g., jpg file with png extension) in the library. Photo.app works fine with these files and osxphotos exports them consistently with the wrong extension (*.png). However, exiftool triggers an error and does not update the file. Calling osxphotos export -exiftool . does not capture the error and reports success. In this case neither -m nor -F fix the problem, the only way to fix is to change the extension -> Same suggestion as point 2. It might also be worth thinking if osxphoto should fix the extension of the exported file in these cases.

Point 3 also offers a way to reproduce the issue that osxphotos does not report errors from exiftool

take any jpg,
[optional] add any tags with exiftool
change extension to png
import the picture.png in Mac Photos
[optional] change tags with Mac Photos
export with osxphotos export -exiftool .

Based on this analysis, it is probably 1 bug and 3 new features:

Bug: osxphotos does not report exiftool errors and warnings
New feature 1: add an option to pass additional parameters to exiftools
New feature 2: detect and correct originals with wrong extensions
New feature 3: offer an option to retain tags in the originals

Hope it helps
PS all the pictures of point 2 above have family members in them and I do not feel comfortable sharing. Nothing personal.

I opened new issues for #1, #3 and the exiftool warnings/errors.  The code for the latter is almost done.
After reading the exiftool documentation I do not want to add -m or -F as defaults but these could be added via a new --exiftool-flags option.  I've also not been able to test -m since I couldn't replicate a file that caused minor warnings.
I'm very leery of implementing #2 as I'd have to have a guaranteed way of detecting this and then figuring out the correct extension. file can detect this but a system call for every exported image would slow things down terribly.  Interestingly Photos does not detect this --it correctly shows the image but the database shows that a PNG image with .JPEG extension has UTI of "public.jpeg" so I can't rely on what Photos says the image is.
I could add a flag that does this only if set, e.g. --fix-filetypes but would need to think about how to safely implement this as getting the extension for a given filetype isn't easy given the plethora of image filetypes.  Currently, I use a Mac system call to get the file extension given the UTI in the database but again, in your case, that's wrong.  This also complicates the --update process as I need to keep track of multiple filenames (original and fixed).  I'll have to think on this one some more.

Thank you point 2, fully agree with what you wrote, thought the same.
One possible idea is to do so only when using -exiftool and rely on the error of -exiftool. This would cost 1 extra call to exiftool for the files that are impacted.
A more advanced version of this idea might rise a warning when osxphotos changes an extension and include a flag to opt-out (or to opt-in) the automatic extension correction.
Just my two cents here - I see this as a workaround for incorrect behaviour of Apple Photos.

This might help as it detects more types than imghdr:
https://github.com/h2non/filetype.py
Also, exiftool can do this:
exiftool '-filename<$filename.$filetype'
But adding this to the code would take some work due to where exiftool gets called in the flow.
Will keep thinking about this.

Also, it would be possible to use my photoscript library along with osxphotos to write a script that finds all the incorrect file extensions in Photos and re-imports those photos with correct extension while re-applying all metadata in Photos.  You'd lose edit history though.

Thank you, I have already cleaned up the library!

@finestream release v0.38.7 will now report all exiftool errors and will report exiftool warnings when run with --verbose.  Additionally, if you use the --report option, the resulting CSV file will have two new columns: exiftool_warnings and exiftool_errors with the text of the warning or error for each file exported.

@finestream New feature 1: add an option to pass additional parameters to exiftools:  added in v0.38.9

New feature 3: offer an option to retain tags in the originals

Implemented in v0.38.18 with --exiftool-merge-keywords and --exiftool-merge-persons


re-evaluate language parsing

Integration sources introspection is done using a mix of technique:

inspecting nodes for mark-up languages
regexp for everything else

While 1. may be good enough, that is not certainly the case for 2. as we have a number of complains such as the fact that the parser does not properly handle commented out code. Using regexp make it also quite complex to keep-up with the evolution of Camel's DSL as example, we can't understand the new endpoint DSL.
Because of the limitation of the current approach we should re-evaluate it and switch to something like ANTLR4 to generate a proper model definition.
As side note I wonder if we should keep the result of this introspection as side resource of Integration  for troubleshooting.

@lburgazzoli  should we leverage the capabilities offered by Camel JBang?

maybe not exactly camel-jbang per se but something that is used behind the scenes by camel-jbang to inspect routes.

maybe not exactly camel-jbang per se but something that is used behind the scenes by camel-jbang to inspect routes.

Thanks, but how would you see this in relation to #4024? I mean, if we manage to leverage the project generated by Camel JBang, shouldn't we be fine? or is there anything else we still need to parse by any chance? I may miss this last point.


App is only ignored from resolution if it has a dependecies property in the app.json

If I understand correctly. Having a "dependencies" property in the app.json is not mandatory.
If the property is missing the following block will not be executed.
if ($appJson.psobject.Properties.name -contains "dependencies") {
  if ($appJson.psobject.Properties.name -contains "id" -and $lvl -eq 0) {
      $ignoredDependencies += $appJson.Id
  }
  ...
}

This will result in the app being downloaded even if it shouldn't.
Which might result in an error in the CI/CD Pipeline.

LGTM


Adds stackPercentages prop to allow for multi bar progress:

Hello!
This is my first real pull request, so I apologize if I've done anything amateur!
This adds the ability to stack multiple percentages into a single bar. Perhaps you've got different resources loading at different rates, or you'd like to use this component as a pie chart. This allows you to pass in an array of integers as stackPercentages and see the pieces of the whole.
If this is too far out of the purview of this component, I understand.
Thanks for reviewing! :smile:
Commit summary:

Refactors some state changes to create object, then apply to state
Checks for prop stackPercentages during construction and
componentWillRecieveProps to handle animation
Adds default to getProgressStyle for percentage as
this.state.percentage
Adds param stackOffsetPercent to handle multibar percentages
Adds transform styles to return value of getProgressStyle
Adds stackPercenagesPaths to render method for multibar stack

No worries about the response time; we're all busy. Thanks for taking a quick review. I'll utilize the changes in my fork and take ideas from your sandbox post. Thanks!


this only works on windows?

Hi @mardahl
Thanks for doing this. do you have any idea what it would take to make this work on Mac ( and linux ) too?
I see some of the data lives in the windows 'registry', the coolest solution would be if one could backup and restore these profiles across OS boundaries ... eventually ...

Thanks TheBigBear.
Unfortunately, I am without a MacOS device to test/develop on.
But yeah, a conversion would be nice, so some sort of roaming could take place - albeit it might take some time.


IV-4843 Make Enable Managed Login output cleaner

Tested on RL 10.4.0 and RL 10.5.0 instance.
The STDERR> [ERROR] write /dev/stdout: broken pipe is removed form the output

LGTM


Cannot allocate color 235

When I start Vim 8.0.1453 on Ubuntu 18.04 LTS, I get the following error:
Line  101:
E254: Cannot allocate color 235.
This relates to the following line in my .vimrc:
highlight LineNr ctermfg=245 ctermbg=235
My terminal uses the following settings:
echo $TERM
xterm-256color
Any suggestions how this can be fixed?

What does Vim answer to
:verbose set term? t_Co? termguicolors?

? The expected values are xterm-256color and 256 for the first two respectively, the default for the third one (if compiled-in) is notermguicolors, but Vim will also tell you in which script (if any) they were last set. Not telling you for one or more of them means that it was (or they were) set either as a startup default or by manual action.
See also the black on pale-yellow list of patches near the top of http://ftp.vim.org/pub/vim/patches/8.0/ — some of the 334 changes (to date) after 8.0.1453 might be fixes for your problem, but none of them strikes me bang in the head as the obvious one.
Compiling one's own Vim (to get the latest fixes) is not very hard, especially on Linux; see http://vim.wikia.com/wiki/Getting_the_Vim_source_with_Mercurial and http://users.skynet.be/antoine.mechelynck/vim/compunix.htm
Best regards,
Tony.

Thanks for your response, Tony.
The mentioned snippet returns the following:
  term=xterm-256color
  t_Co=256
        Last set in ~/.vimrc
notermguicolors

Do you have a recommendation how to proceed from here?

Do you have a recommendation how to proceed from here?

As E254 matters only when you are using termguicolors Vim or GVim, it looks strange to get the error against notermguicolors Vim.
@AvexMedia , will you try and do :hi LineNr immediately after you get E254, and report back the result here?  That might give us a clue to diagnosing the issue.

:hi LineNr returns the following @nuko8:
LineNr         xxx term=underline ctermfg=245 ctermbg=235 guifg=#3E4853 guibg=#111314

Thanks.  Then, please try:
$ vim --clean +'hi LineNr ctermfg=245 ctermbg=235'

Do you still have an E254 error with this?  What does :hi LineNr say this time?

I am a bit stumped, because according to what I know, E254 should only occur for color names and not color indexes.  You don't have by any chance used the index 235 as guifg or guibg attribute? Are you sure, the corresponding line in your .vimrc is correct? Are there other highlighting definitions?
BTW: Can you please set verbose setting to one and see if this returns some more output?

@AvexMedia
If you don't have E254 with the command I asked you to run, then that strongly indicates that there's guifg=235 or guibg=235 (Neither ctermfg=235 nor ctermbg=235,  since both are irrelevant to E254 at any rate.) at Line 101 of one of the Vim scripts that constitute the whole of your Vim settings (Not necessarily .vimrc only).  If that's the case, please follow @chrisbra 's comment.  That's exactly the same as mine for the case where you don't have E254 with the mentioned command.
Honestly, I thought you would report back a result showing that LineNr had either guifg=235 or guibg=235, but that turned out not the case.  Probably, in your settings, gui{fg|bg}of LineNr is set to 235 at Line 101 of a file unknown to us at some point of startup, and then  overwritten to #3E4853 and #111314, respectively, somewhere afterwards.  The :verbose command would help you spot the place causing the trouble.  If it fails you, you might need to examine every relevant file at worst.

No more info. I am closing this as unreproducible.


How can i make it vertical (top to bottom) rather than horizontal?

How can i alter to make it vertical i.e. top to bottom rather than horizontal chart? I'm starting from one ancestor and have all children below.
I'm using this example i.e. Ancestors & Defendants.

My response in this issue explains how to do it: https://github.com/justincy/d3-pedigree-examples/issues/10

thanks for quick response @justincy

@justincy can you or someone please provide an example/link for top to bottom pedigree? I actually got the time today to work on that module again but it doesn't seem so simple as its stated in your comment and the third note in the readme. I tried converting this example to vertical pedigree following your guide and finally I got top to bottom structure as per my requirements but still the links between the nodes aren't perfect.
I think it will be good if we can add a new issue to create a top-bottom example when possible.

Good idea. https://github.com/justincy/d3-pedigree-examples/issues/17

Thanks, I really appreciate.

Any news on this

Nope. I haven't done it yet.


⚠ GitLab Server has degraded performance

In 636eef9, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11594 ms

Resolved: GitLab Server performance has improved in fd71e88.


Adding ExtensionsRunner parameter to getExtensionRestHandlers

Description
Provides the ExtensionRunner to getExtensionsRestHandlers. This allows us to pass in the same instance of the extension runner to all registered rest actions.
Companion PR : https://github.com/opensearch-project/anomaly-detection/pull/835
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #530 (e3aa1f5) into main (9bbc9dd) will increase coverage by 0.03%.
The diff coverage is 50.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main     #530      +/-   ##
============================================
+ Coverage     66.98%   67.02%   +0.03%
  Complexity      191      191
============================================
  Files            34       34
  Lines           836      837       +1
  Branches         26       26
============================================
+ Hits            560      561       +1
  Misses          264      264
  Partials         12       12

Impacted Files
Coverage Δ

.../main/java/org/opensearch/sdk/ActionExtension.java
57.14% <ø> (ø)

...rch/sdk/sample/helloworld/HelloWorldExtension.java
66.66% <ø> (ø)

...main/java/org/opensearch/sdk/ExtensionsRunner.java
70.34% <50.00%> (+0.17%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Jensen Shannon Distance

The Jensen-Shannon Distance (square root of the Jensen-Shannon Divergence) is a true metric distance with good statistical properties and biological interpretation.
It's a kind of must in Biology, here just some examples:
https://www.nature.com/articles/nature09944
http://compbio.mit.edu/cummeRbund/
https://liorpachter.wordpress.com/tag/jensen-shannon-divergence/
There are some attempts to do it in python, but they seem all (pretty) dead, maybe there is no pressure for their fields:
https://github.com/scipy/scipy/pull/3213
https://github.com/scikit-learn/scikit-learn/pull/4191#issuecomment-345925427
and here another https://github.com/biocore/qiime/issues/280 for qiime, but again there are no news about it since 2015.
Scikit-bio got a shannon entropy already, skbio.diversity.alpha.shannon but I don't know how far is from the Jensen Shannon Distance.
So I was wondering whether scikit-bio can take any advantage to implement it.

I can look into this and see what I can do


Extend the duration of raffles

About the PR

Made raffles longer
Why / Balance

Raffles are way too short. They are easily missed.
An alternative might be to announce ghost roles. That would be a longer PR.
Technical details

Media

[ ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

Why are you removing razorium in this pr.


Truvari, STRs and Expansion Hunter - Query

HI @ACEnglish - thanks for the brilliant tool and the TR benchmarking work!
I have a query regarding the benchmarking files, Truvari and outputs from Expansion Hunter, specifically looking at STRs. We've sequenced HG002 and produced an alignment file using the Illumina DRAGEN platform. I've then run this through Expansion Hunter and I wanted to use Truvari and the TR benchmarking files to get some precision/recall metrics for this.
I've run as follows (I had to put --pctseq 0 and remove other parameters), after splitting multiallelic calls with bcftools norm:
2024-04-24 14:50:43,875 [INFO] Truvari v4.2.2
2024-04-24 14:50:43,875 [INFO] Command /home/v.so257591/.conda/envs/truvari/bin/truvari bench --pctseq 0 -b ../HG002_
GRCh38_TandemRepeats_v1.0.1.vcf.gz -c original_sample_split.vcf.gz --includebed ../HG002_GRCh38_TandemRepeats_v1.0.1_
Tier1.bed.gz -o Original_Bench_Results/ -f /data/resources/human/references/GRCh38_masked/GRCh38_full_analysis_set_pl
us_decoy_hla_masked.fa
2024-04-24 14:50:43,875 [INFO] Params:
{
    "base": "/mnt/HomeApps/Home/v.so257591/STR_Validation/HG002_GRCh38_TandemRepeats_v1.0.1.vcf.gz",
    "comp": "/mnt/HomeApps/Home/v.so257591/STR_Validation/Version4/original_sample_split.vcf.gz",
    "output": "Original_Bench_Results/",
    "includebed": "/mnt/HomeApps/Home/v.so257591/STR_Validation/HG002_GRCh38_TandemRepeats_v1.0.1_Tier1.bed.gz",
    "extend": 0,
    "debug": false,
    "reference": "/data/resources/human/references/GRCh38_masked/GRCh38_full_analysis_set_plus_decoy_hla_masked.fa",
    "refdist": 500,
    "pctseq": 0.0,
    "minhaplen": 50,
    "pctsize": 0.7,
    "pctovl": 0.0,
    "typeignore": false,
    "chunksize": 1000,
    "bSample": "HG002",
    "cSample": "211028_A00748_0168_AHFYNFDSX2_20M05405-1",
    "dup_to_ins": false,
    "sizemin": 50,
    "sizefilt": 30,
    "sizemax": 50000,
    "passonly": false,
    "no_ref": false,
    "pick": "single",
    "check_monref": true,
    "check_multi": true
}
2024-04-24 14:51:34,743 [INFO] Including 1638106 bed regions
2024-04-24 14:52:07,773 [INFO] Found 9 chromosomes with overlapping regions
2024-04-24 14:53:06,778 [INFO] Zipped 1196641 variants Counter({'base': 1196606, 'comp': 35})
2024-04-24 14:53:06,778 [INFO] 11344 chunks of 1196641 variants Counter({'__filtered': 1180470, 'base': 16149, 'comp'
: 22})
2024-04-24 14:53:08,386 [INFO] Stats: {
    "TP-base": 0,
    "TP-comp": 0,
    "FP": 12,
    "FN": 16149,
    "precision": 0.0,
    "recall": 0.0,
    "f1": null,
    "base cnt": 16149,
    "comp cnt": 12,
    "TP-comp_TP-gt": 0,
    "TP-comp_FP-gt": 0,
    "TP-base_TP-gt": 0,
    "TP-base_FP-gt": 0,
    "gt_concordance": 0,
    "gt_matrix": {}
}
2024-04-24 14:53:08,386 [INFO] Finished bench

As you can see, I'm getting TP as 0, but from eyeballing my VCF file from Expansion Hunter and from the benchmark VCF, I'd expect at least some TPs. Here's an example:
#My Expansion hunter VCF
chr2    190880872       .       C       <STR8>  .       PASS    END=190880920;REF=16;RL=48;RU=GCA;VARID=GLS;REPID=GLS
        GT:SO:REPCN:REPCI:ADSP:ADFL:ADIR:LC     1/0:SPANNING/SPANNING:8/14:8-8/14-14:15/16:11/14:0/0:33.8096
chr2    190880872       .       C       <STR14> .       PASS    END=190880920;REF=16;RL=48;RU=GCA;VARID=GLS;REPID=GLS
        GT:SO:REPCN:REPCI:ADSP:ADFL:ADIR:LC     0/1:SPANNING/SPANNING:8/14:8-8/14-14:15/16:11/14:0/0:33.8096

#Benchmark VCF
chr2    190880872       .       CGCAGCA C       60      PASS    AN=2;AC=1;NS=86;AF=0.168605;MAF=0.168605;AC_Het=27;AC_Hom=2;AC_Hemi=0;HWE=0.446931;ExcHet=0.260788   GT:BPDP:FT:AD   0/1:1,1,1,1:PASS:1,1
chr2    190880872       .       CGCAGCAGCAGCAGCAGCAGCAGCA       C       60      PASS    SVTYPE=DEL;SVLEN=24;AN=2;AC=1;NS=86;AF=0.244186;MAF=0.244186;AC_Het=34;AC_Hom=8;AC_Hemi=0;HWE=0.769395;ExcHet=0.386269       GT:BPDP:FT:AD   1/0:1,1,1,1:PASS:1,1

#Benchmark BED
chr2    190880842       190880919       Tier1   TP_TP_TP        13      0.8795  24      6

Is this a formatting issue as I've called STR but the benchmark VCF doesn't? I took a look and there are no SVTYPE=STR in the benchmark VCF, so I'm wondering if I'm even using it correctly?!! Is it just that I need to run truvari refine as well?
Any thoughts/advice are welcome! I've attached my VCF for reference (changed to txt just for upload purposes).
Thanks
Sophie
original_sample_split.txt

Hello,
Thank you for including the VCF with your ticket. It makes it so much easier for me to work on.
The VCF's format is compliant to VCF standards. However, the e.g. <STR20> type annotations are not handled by truvari. Truvari assumes that variants are resolved whereas this VCF has CNV-like information. However, we can make this VCF truvari compatible by running a simple script that sets the SVTYPE/SVLEN.
converter.py
import pysam

vcf = pysam.VariantFile("original_sample_split.vcf")
# Add the new header line to the VCF. SVTYPE is already in the VCF, though it is not used
header = vcf.header.copy()
header.add_line('##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Alternate length - Reference length">')
output = pysam.VariantFile("/dev/stdout", 'w', header=header)

for entry in vcf:
    if entry.alts is None:
        ## Reference monozygotic seems to happen sometimes. We can't alter those
        output.write(entry)
        continue

    # parse the ref/alt alleles
    alt = entry.alts[0]
    alt_copy_number = int(alt[len("<STR"):-1])
    ref_copy_number = entry.info["REF"]

    # The reference and alternate sequences *should* just be some number of motifs
    motif = entry.info['RU']
    motif_len = len(motif)
    ref_len = ref_copy_number * motif_len
    alt_len = alt_copy_number * motif_len

    # We can set the length as the ALT - REF, and when length is < 0 it is a DEL, > 0 an INS
    svlen = alt_len - ref_len

    if svlen > 0:
        svtype = "INS"
    else:
        svtype = "DEL"

    # output
    entry.translate(header)
    entry.info['SVLEN'] = svlen
    entry.info['SVTYPE'] = svtype
    output.write(entry)

I made some assumptions here, so please check that this is correct behavior before committing to this script. The main point is that this style of VCF is not handled by Truvari and TR expansions/contractions can be represented as insertions/deletions just fine.
Now we just convert the VCF
python converter.py | bgzip > typed_trs.vcf.gz
tabix typed_trs.vcf.gz

For speed, we're just going to subset the benchmark to only the regions analyzed by the caller.
This is going to inflate the recall. The correct way to do this would be to intersect ExpansionHunter's bed file of
regions it analyzed with the benchmark's regions to create the subset.regions.bed
bedtools intersect -u -a GIABTR.HG002.benchmark.regions.bed.gz -b typed_trs.vcf.gz > subset.regions.bed

Finally, we run truvari with the recommended parameters from the benchmark's README. Note that we also added  --pctseq  0 since these are not sequence resolved calls. This also means the variants cannot be run through truvari refine
truvari bench -b GIABTR.HG002.benchmark.vcf.gz --includebed subset.regions.bed \
    -s 5 --pick ac --pctseq 0 -c typed_trs.vcf.gz -o bench_result

Results:
{
    "TP-base": 18,
    "TP-comp": 17,
    "FP": 7,
    "FN": 4,
    "precision": 0.7083333333333334,
    "recall": 0.8181818181818182,
    "f1": 0.7593052109181142,
    "base cnt": 22,
    "comp cnt": 24,
    "TP-comp_TP-gt": 14,
    "TP-comp_FP-gt": 3,
    "TP-base_TP-gt": 14,
    "TP-base_FP-gt": 4,
    "gt_concordance": 0.8235294117647058,
    "gt_matrix": {
        "(1, 0)": {
            "(0, 1)": 3,
            "(1, 0)": 2,
            "(1, 1)": 1
        },
        "(0, 1)": {
            "(0, 1)": 3,
            "(1, 0)": 1,
            "(1, 1)": 2
        },
        "(1, 1)": {
            "(1, 1)": 5
        },
        "(1, None)": {
            "(1, 1)": 1
        }
    }
}

Thanks so much for the detailed reply @ACEnglish! All makes sense!

Hi, I'm currently also looking at STRs from ExpansionHunter (DRAGEN version and the standalone). I am a bit confused about why it's necessary to intersect the ExpansionHunter BED file with the GIAB TR BED file. Is this not explicitely handled by truvari refine? Basically I would expect the following two approaches generating the same results:
1:
truvari bench -b GIABTR.HG002.benchmark.vcf.gz --includebed GIABTR.HG002.benchmark.regions.bed.gz \
    -s 5 --pick ac --pctseq 0 -c typed_trs.vcf.gz -o bench_result
truvari refine --use-original-vcfs --regions ExpansionHunter_Catalog.bed bench_result/

2:
bedtools intersect -u -a GIABTR.HG002.benchmark.regions.bed.gz -b  ExpansionHunter_Catalog.bed > subset.regions.bed
truvari bench -b GIABTR.HG002.benchmark.vcf.gz --includebed subset.regions.bed \
    -s 5 --pick ac --pctseq 0 -c typed_trs.vcf.gz -o bench_result
truvari refine --regions subset_regions.bed bench_result/

Is my understanding correct?
Best,
Raphael

Yes, refine does as you've described. The subsetting I was describing was as it relates to the bench output.


Burp parser aggregates findings on type

https://github.com/DefectDojo/django-DefectDojo/blob/ca6628dd3d295426fecf08c1c82190564e49ad67/dojo/tools/burp/parser.py#L42
https://github.com/DefectDojo/django-DefectDojo/blob/ca6628dd3d295426fecf08c1c82190564e49ad67/dojo/tools/burp/parser.py#L142
Burp parser aggregates all findings per type alone.
This means all Cross-site scripting (reflected) (for instance) fall under the same finding, regardless of endpoints, parameters or even severity.
I might be missing context on other people triage approach, but I find this confusing for mine as I'm unable to triage specific issues but rather an entire class of them.
Am I missing something?
Would a PR to change this be acceptable or too big of impact? Does it make sense to create a separate parser for this, if the latter? Or maybe make this dedupe_key configurable in settings.py?

What is your suggestion for updating the behavior? Are you thinking to aggregate by a combo of type and severity? I don't have much experience with the Burp parser, but the general intention of handling dynamic findings in dojo is to aggregate as many endpoints to a single finding as possible

However, this statement is probably the root of my confusion: why is this so?

The primary reason is value proposition. If a tool is generating 1000 results, and you import into Defectdojo, and get 100 aggregated findings, then DefectDojo is doing a lot of the organizing you would have to do otherwise

SAST findings are not aggregated by file path, why aggregate DAST ones per endpoint?

This is due to a historical architectural decision. Endpoints in DAST findings are stored in another database table, such that many endpoints can be associated with a single finding. File path and line number are single fields on a given finding, so there is really no possibility in aggregating those natively.

Also, as the hash code fields do not consider endpoints for "uniqueness", nor description, it means that:

1 report with SQLi on endpoint A is submitted
later, 1 report with SQLi on endpoints A and B is submitted
The second report is considered duplicate of first. But the first does not mention endpoint B, so it is missed.

And if it would include endpoints in the hash code fields, the opposite would happen:

the second report would create a new finding that mentions an issue that already exists in dojo

This is where reimport will shine! However there are two use cases for controlling the behavior you (specifically) may or may not desire. The two cases are whether endpoints are used in the deduplication algorithm for a given tool.

Endpoints are in the deduplication algorithm (the default - largely due to backwards compatibility)

On each reimport, the list of endpoints must match verbatim to be considered the same finding.

Endpoints are not included in the dedupe algorithm (my preference as I exclusively use reimport)

Using your example of the SQLi finding with endpoint A, a reimport of the same finding with endpoint B will result in the original finding have both endpoint A and B. An third reimport that omits endpoint A would leave the same finding, but would mitigate endpoint A. This allows for a true point in time view of what endpoints are mitigated on a given finding

The nuance here is that are very valid use cases for each approach. I believe it comes down to using import vs reimport. My personal adage is that import is great for when you are getting audited, and reimport is great for when you are working with developers to mitigate issues. Reusing your SQLi example again:

Import: Would likely create a lot of SQLi findings that would be difficult to manage, but I could know exactly what endpoints were vulnerable, and when they were vulnerable on a single finding without having to do much thinking
Reimport: Little to no duplicates, and each finding operates as a "running document" of work to be done. If the finding is open, then at least one endpoint is vulnerable still. The potential downside is that an old finding could be reopened if new endpoints for a given vulnerability are rediscovered.

Each of those examples are wildly different use cases but serve as very valuable tools to have. The point I am trying to make is that DefectDojo is not really a one size fits all, but with the right tuning can be the best tool you have!
This may have been TMI, but I hope it's helpful!

Definitely not TMI but wanted info, appreciated the time taken.
These differences between import and reimport were the kind of differences I was looking for when I kicked off ]this thread in slack](https://owasp.slack.com/archives/C2P5BA8MN/p1730962412460509).
I think this should be documented (if it is I failed to find it) because it's a huge technical difference for handling imports, if import will lose details and reimport won't (for these cases which are not edge cases)

DefectDojo is doing a lot of the organizing you would have to do otherwise

My main issue with it is that it's not "organizing". I would expect default behavior from parsers to be 1 dojo finding per 1 report finding (or as close as possible to it) and, then, I can create finding groups as I please. I could group by type as it is being done, but I could also group by endpoint or other field instead.
If they are already aggregated by the parser itself, I cannot un-aggregate them. If they are not aggregated by the parser, I can aggregate them with groups. To be flexible, I'd expect parsers to do less, not more, if that makes sense...

but with the right tuning can be the best tool you have

Problem here is that I don't see any tuning I can do for my case, where each endpoint has findings handled differently and by different developers/teams, apart from splitting the report myself before uploading into different engagements or tests...
Regardless I still agree it is the best tool I have for finding 😄


VM can be added more than one time to the same nic

In [27]: len(client.kvm.info(vm_uuid)['Net'])
Out[27]: 4

In [28]: client.kvm.add_nic(vm_uuid, 'vlan', id=t1)

In [29]: len(client.kvm.info(vm_uuid)['Net'])
Out[29]: 5

In [30]: client.kvm.add_nic(vm_uuid, 'vlan', id=t1)

In [31]: len(client.kvm.info(vm_uuid)['Net'])
Out[31]: 6

Version
1.1.0-alpha @Revision: 867d58e68dd938f90f2190c2a48cbf300489f1a0

Verified on master @Revision: f3dd77f8dd098b4c93ae77706d84acf488ce4283


(maint) Remove old platforms

There are old platforms that have reached the end of their lives. We no
longer want to deal with them.

OK fine :+1:


[USD] update to 22.08

Library name: usd
New version number: 22.08
Other information that may be useful (release notes, etc...)
https://github.com/PixarAnimationStudios/USD/releases/tag/v22.08
This request is from https://github.com/PixarAnimationStudios/USD/pull/2041.

Hi @Cheney-W the PR looks great, thanks so much for tackling this :)
We are now in final QA on 22.11. The work in your PR https://github.com/microsoft/vcpkg/pull/26941), for 22.08 is the same as what will be needed for 22.11, just the hashes are different of course. Would it make sense to wait until the release of 22.11 [next week] and update USD on vcpkg all the way to 22.11, or would it make sense to land 22.08 now, and follow up shortly with the 22.11 version update?

If the waiting time is not more than a month, I think it make sense to wait until the release of 22.11.

Ok, thanks! I'll update this thread asap with release data.

@meshula
I saw there is a v22.11-rc1 tag in PixarAnimationStudios/USD, can I update to this version?

The real release is almost ready, we're just finishing up now! I'll update this thread.

Okay v22.11 is tagged and released! https://github.com/PixarAnimationStudios/USD/releases/tag/v22.11
Thanks for your patience!

Hi @Cheney-W! I was just looking at https://github.com/PixarAnimationStudios/USD/pull/2041/files ~ it was waiting on the update to 22.08, as it seemed like a good checkpoint, in terms of pointing people to vcpkg to get a recent USD build. I was wondering what the status might be of updating USD in vcpkg?

Update is blocked by https://github.com/PixarAnimationStudios/USD/issues/1600 now.

Oh! I didn't know that. I'll start tracking these issues together.

Fixed by https://github.com/microsoft/vcpkg/pull/29688.


Automatically apply GPC preferences when Fides.js is initialized

Automatically apply GPC preferences when Fides.js is initialized.

Send preference downstream (can mock API call for now)
Preference is reflected in Fides cookie

A. If no experience applies for the user + location, we cannot send GPC-override preference downstream as there are no notices to attach preference to.

Makes sense. However this is one of the reasons why I believe we need some "global default notices", which I've flagged as a separate follow-up with @mfbrown so we can design that too.

B. If some kind of configuration option is invalid, we cannot obtain details to fetch experience / location, and also cannot send GPC-override preference downstream.

Yeah, if there's an error, we'll have to just fail.

C. If consent overlay is disabled (we have this as a global config option), do we still attempt to retrieve experience / user location, and send GPC-override preference downstream? Probably not, because the disabled state represents the entire ecosystem of the overlay, so we shouldn't attempt any side-effects.

Disagree (politely!) on this one. The GPC signal should be honoured even if there is no overlay experience at all - in fact, what complicates GPC a bit is that it should work even if only a privacy center experience is configured...
So I think overall the GPC logic should be handled outside of the component-specific overlay bits. In other words, in the Fides.init() method and not part of initOverlay or the <Overlay> component, etc.

D. Along the same vein, if consent overlay is disabled, should we generate a device id (for new users) and save the cookie?

Yes, we should save the cookie if we save the GPC preferences due to GPC. That does complicate the lifecycle of the cookie a bit... I think the cleanest logic is likely:

Always getOrMakeFidesCookie() in the Fides.init setup
Do not immediately save the cookie to the browser, though, just keep it in memory
Save the cookie as soon as a preference is saved (or, in the case of the privacy center, as soon as a new consent request is started which might require identity verification, etc.)

That's essentially "lazy-saving" the cookie, which is a pattern that makes sense to me here.

So I think overall the GPC logic should be handled outside of the component-specific overlay bits. In other words, in the Fides.init() method and not part of initOverlay or the  component, etc.

I'm planning to fetch experiences / geolocation within initOverlay, which would be a dependency for GPC. This signals to me that I'll need to extract this logic out of initOverlay to live in Fides.init() instead.

Do not immediately save the cookie to the browser, though, just keep it in memory
To keep it in memory such that the PC can access it, too, you think it's OK to place it on the window.Fides obj that's created as part of Fides.init()?


add button in row and link the label with data row so that when click on button the values of column with respect to particular row gets displayed in ag-grid

i don't understand this


Reviewing pull requests with git difftool

I would love to be able to list outstanding pull requests, pick one to review using git difftool (which in my case fires up a gui diff tool).

Listing pull requests (specifically open ones) would be awesome. I know I can add the pulls namespace to origin fetch, but the problem is that has all pull requests, not just open ones. On a small repo that's fine, but I work on a repo with 17k+ pull requests. I had to stop doing that because it made autocomplete take 10+ seconds.

If someone does implement a script, link to it here. ;) I'm not ready to do it myself, but I totally want this feature.

Here's a windows bat script that I made for myself PrDifftool.bat @geoffreywiseman


Helidon JPA - GraalVM Native - env variables not read  (Issue 1/2)

Environment Details

Helidon Version: 3.1
Helidon MP
JDK version:17.05
OS: Linux
Docker version (if applicable): N/A

Problem Description
If Helidon + JPA + Hibernate + Jackson + Oracle DB driver is used with GraalVM Native
Passing env variables to modify the microprofile-config.properties does not work.
The goal is to set the connection parameters at runtime for sec purpose.
The problem is probably a lot more generic than with JPA. But I reproduce it repetitively with it.
Steps to reproduce
Just use the sample generated by Helidon starter.
You need a running database
In your microprofile-config.properties, you can leave it empty if you want too.
-------
# Microprofile server properties
server.port=8080
server.host=<IP_ADDRESS>

# Change the following to true to enable the optional MicroProfile Metrics REST.request metrics
metrics.rest-request.enabled=false

# Application properties. This is the default greeting
app.greeting=Hello

# Datasource properties
javax.sql.DataSource.ds1.dataSourceClassName=oracle.jdbc.pool.OracleDataSource
javax.sql.DataSource.ds1.dataSource.url=##JDBC_URL##(invalid value on purpose)
javax.sql.DataSource.ds1.dataSource.user=##DB_USER##
javax.sql.DataSource.ds1.dataSource.password=##DB_PASSWORD##
--------

test 1: with JIT (OK):
mvn package -DskipTests
cd target
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_URL="(jdbc:oracle:thin:@(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1521)(host=xxxxx.adb.eu-frankfurt-1.oraclecloud.com))(connect_data=(service_name=xxxxxxx.adb.oraclecloud.com))(security=(ssl_server_dn_match=yes))))"
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_USER=admin
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_PASSWORD=xxxxxx
java -jar helidon.jar
-> OK

test 2: with Native (BUG):
mvn package -Pnative-image -Dnative.image.buildStatic -DskipTests
cd target
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_URL="(jdbc:oracle:thin:@(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1521)(host=xxxxx.adb.eu-frankfurt-1.oraclecloud.com))(connect_data=(service_name=xxxxxxx.adb.oraclecloud.com))(security=(ssl_server_dn_match=yes))))"
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_USER=admin
export JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_PASSWORD=xxxxxx
./helidon
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Caused by: java.sql.SQLException: Invalid Oracle URL specified
	at oracle.jdbc.datasource.impl.OracleDataSource.getPhysicalConnection(OracleDataSource.java:566)
	at oracle.jdbc.datasource.impl.OracleDataSource.getConnection(OracleDataSource.java:347)
	at oracle.jdbc.datasource.impl.OracleDataSource.getConnectionInternal(OracleDataSource.java:1996)
	at oracle.jdbc.datasource.impl.OracleDataSource.getConnection(OracleDataSource.java:322)
	at oracle.jdbc.datasource.impl.OracleDataSource.getConnection(OracleDataSource.java:283)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
	... 72 more

test 3:
Hardcode the values in microprofile-config.properties
Recompile with GraalVM Native.
It works....

To be clear: this doesn't concern JPA at all, right? It just concerns configuration and data sources?

Yes, I think so too. There is no relation between this behavior and JPA.

So the data source integration code currently reads configuration to know what data sources there are, so it can figure out what injection points to satisfy in your CDI-based application.  When you build a native image, the configuration is read at build time, because the mechanism used to create data source implementations to inject needs to be done at build time.  At build time, in your example above, you do not supply an environment variable, so META-INF/microprofile-config.properties is not overridden (as designed).
If you run:
JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_URL="(jdbc:oracle:thin:@(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1521)(host=xxxxx.adb.eu-frankfurt-1.oraclecloud.com))(connect_data=(service_name=xxxxxxx.adb.oraclecloud.com))(security=(ssl_server_dn_match=yes))))" JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_USER=admin JAVAX_SQL_DATASOURCE_DS1_DATASOURCE_PASSWORD=xxxxxx mvn package -Pnative-image -Dnative.image.buildStatic -DskipTests

…I expect you will see that the environment variable value is used.  Do you observe this?

Thanks. In my case, I build a docker container and pass the DB connection details as kubernetes secret (= env variables)
If I understand due to that build-time hardcoding, Using META-INF/microprofile-config.properties is not possible anymore in Native mode to pass connection details at run-time.
Any idea if there is a way to pass user/password/DB_connection without hardcoding them ? The way with META-INF/microprofile-config.properties is the only one I knew...
Thanks.

(First, META-INF/microprofile-config.properties is one of an infinite number of possible MicroProfile Config-compatible configuration sources and is usually reserved for last-ditch defaults. Environment variables are probably a better choice of MicroProfile Config-compatible configuration source.)
The short answer to your question is: no, at least as of this writing.
If you want to build using one configuration and run with another, you probably shouldn't use native image anyway in this scenario. As mentioned above, the choice of what data sources exist (regardless of how they are configured) must be done at build time.  Currently the configuration for them also happens at build time in native-image scenarios, because the CDI container is validated before being "frozen", and is "thawed" only at native-image run time (i.e. with the data source integration code the way it is, configuration has already been read).
Changing this so that some configuration items (which ones?) are read at build time, and some (which ones? what's the litmus test?) are applied at run time while preserving the validation phase of the container could be done but won't happen anytime soon. You may feel free to file an issue if you want this to be worked on.
Finally, if you are not writing a stateless serverless function (I'm guessing this is the case since you're connecting to a database, so there's state involved) your performance overall will also probably be better if you eschew native image and just go with regular old Java. jlink and application class data sharing can be your friends here if for some reason size is at an absolute premium.

I believe this was fixed with #8771. Closing. If you feel this should be reopened, please feel free to comment.


Feature request: support queries that cross retention policies

Use case: I want to have a database with the following retention policies at the given data granularities:

10sec bins for 1 hour
5min bins for 1 day
1 hour bins for 1 week

etc. Assume the continuous queries that implement these ensure that they are disjoint; e.g. there are no 5-minute bins for the past hour; just starting 1 hour ago.
Now, I would like to issue a single query for the past three days. This crosses three retention policies, and there is currently no way to do this; instead I must issue a separate query to each retention policy.
Not 100% sure what the response should look like; maybe each record should contain its retention policy? No, it seems more that the client doesn't care about the retention policy of the data it's querying; even the bin size seems like something the client should have to care about when making the query, though it will have to deal with them in the response.
I have very little experience with InfluxDB at the moment (still evaluating for my application), but based on what I have seen, I would expect something like SELECT count FROM messages.* WHERE time > now() - 3d, with a response containing three series, named messages.10s, messages.5m, and messages.1h or something. Again, wild speculation; not sure what the best interface would be here. The homegrown timeseries database we currently use returns the bin boundaries with each record, but it also automatically rolls newer, small bins into larger ones so that all the bins in the result are the same size. I'm not sure I like that approach (but I am glad to have the option with e.g. GROUP BY time(5m)).
Thanks for your consideration (and gentle correction if need be).

+1 - this would be cool.
Since the query format is:
select value from <db>.<rp>.<measurement> ...
Perhaps the query notation could be:
select value from <db>.*.<measurement> ...
What would also be nice is to be able to use the same series, but apply a 'downsampling policy' so that, for example, the latest 2 weeks of data is at raw/high granularity, 2weeks-1month is downsampled to hourly, 6m-1y to daily, etc. So your queries would remain the same.

Yeah, that sounds like exactly what I'm looking for too. That approach would definitely require the returned points to contain their bin width, since you'd now have one series with points representing different granularities of data.

+1 holy grail

closed as dupe of #2625


Add Support for Vue, React, Angular frameworks

Is your feature request related to a problem? Please describe.

Currently, as a react, Angular, or Vue developer, there is no way for me to modify precompiled CSS or HTML markup from the Edge Devtools. Currently, only runtime HTML and CSS are supported, with the exception of external scss/sass.
Describe the solution you'd like:

In the same way I can click, open, and apply changes a pre-compiled SCSS stylesheet and see it updating in the browser, I would like to be able to click and open a react, vue, and/or angular component precompiled within dev tools, make changes, and have the changes interpreted in the browser and apply the changes to the source files.
Describe alternatives you've considered:

I've tried using framework specific browser extensions within Edge, but this is not supported for many reasons.
I've enabled source maps in local environments for scss and vue components, but the dev tools do not interpret vue components.
Additional context:

This is probably a very complex task, but would dramatically simplify developer workflows, given how the industry has evolved to adopt these frameworks for many web-based projects, whether that's a website or a SAAS application. This would probably help increase adoption of this tool, and be useful for developers building things beyond static html webpages.
It can also be useful for migratory projects, where one or more frameworks may be used within a single UI. Many web projects go through transitional phases where multiple UI frameworks exist as a means to deprecate one over the other, or the first UI framework is being implemented piecemeal into a project.
This is something that's missing from devtools in general, but since you're able to allow VSCode to read source files in a local project, there should be no reason to not support HTML markup and SCSS changes for precompiled React, Vue, or Angular projects.

Hello David, we're working on Sourcemaps support for CSS, and should have that in the extension to try out soon. We understand your issue, and the main problem is to have good sourcemaps generated by different frameworks to allow the developer tools to get access to the sources.

Could you possibly provide at least rough ETA for this feature? I am running angular 13 monorepo with multiple projects and I can debug just fine all of the files but whenever I try to edit anything within my scss files i get errors such as Error while mirroring css content to document. Could not mirror css changes to document. No workspace mapping was found for 'http://localhost:4200/XXXXX.component.scss'.
I tried using Vscode-edge-devtools: Source Map Path Overrides and/or Path Mapping but it doesn't seem to do the job for multiple projects under one repo

This should work, if you have set up your sourcemaps the right way. @mliao95 @hkal can you help?

Also @Mkurkow there is a scenario where this is tracked with a demo app that has the right sourcemaps setup:
https://github.com/microsoft/vscode-edge-devtools/issues/965

Based on what Chris said, this should now work. So closing the issue.

I'm not seeing this work at all in vue, I have a single file component with scoped css, as well as inline styles in the template, and i'm not seeing either change when I change the style in devtools?


A bug in macos

In MacOS,I found out that PyScreeze used the screencapture  command line to take a screenshot, but the size of the picture taken by screencapture is 2880x1800. And PyScreeze return a coordinate with respect to 2880x1800.But pyautogui.moveTo() regard it as a a coordinate with respect to 1440x900.

if you have a retina everything is multiplied by 2

I ran into the same issue. For now, I divide the coordinates by 2 each time I call LocateCenterOnScreen().


Set GOPATH if not already set

Hello Kenny,
I just installed golang. Had to set GOPATH in exec-env to fix errors like below:
$ go get github.com/elazarl/goproxy
package github.com/elazarl/goproxy: cannot download, $GOPATH not set. For more details see: go help gopath

I'm not sure if this is the right thing to do (new to golang). I just followed conventions I previously used for other plugins (each version has it's packages in it's own install directory). Please do let me know if there is a better way to do this.

Hello, I am using asdf-vm 7.0 with latest asdf-golang
Why is my GOPATH being overridden? I am a golang newb so more than likely this is all user error.
$ echo $GOPATH
/home/tom/go

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tom/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/tom/.asdf/installs/golang/1.12/packages"
...

I can get goproxy but not my own package (im trying to follow this tutorial) - it does not look like my set GOPATH is being used to find the package:
$ go get github.com/elazarl/goproxy
$ go run main.go
main.go:3:8: cannot find package "structs/computer" in any of:
        /home/tom/.asdf/installs/golang/1.12/go/src/structs/computer (from $GOROOT)
        /home/tom/.asdf/installs/golang/1.12/packages/src/structs/computer (from $GOPATH)


[Bug Report] Offline player doesn't receive virtual key when it's given to them

Hello! I gave a player a virtual key when they were offline and when they joined the server they didn't receive it.
Minecraft Server Version: 1.18.2
Plugin Version: 1.10.3

Update to v1.10.4
https://github.com/Crazy-Crew/Crazy-Crates/releases/tag/v1.10.4


Check support at GitHub Enterprise

URL: https://git.daimler.com/ The endpoints for the API should be the same.


Allow to filter on time oldest and newest event to show

this is also an important item to filter, e.g. to restrict view to one OS boot.

duplicate


Add bower and grunt-cli install to homepage?

It can be confusing to run the command from the homepage when you don't have bower or grunt-cli installed.

Looks like it now says:

You will need Node.js, Yeoman, Grunt, and Bower to run these commands.