Some inconsitencies regarding dv/dvcam/dvcpro SD files

It seems that (at least) the last version of mediainfo reports erroneously dv25 DVCAM 420.
For example 2019.4 reports as DVCPRO with 4.2.2 colorspace which at 25Mbit/s
as follows:
General
Complete name                            : C:\bmx\RS.MXF
Format                                   : MXF
Commercial name                          : DVCPRO
Format version                           : 1.2
Format profile                           : OP-1a
Format settings                          : Closed / Complete
File size                                : 4.43 GiB
Duration                                 : 19 min 39 s
Overall bit rate                         : 32.3 Mb/s
Recorded date                            : 1970-01-01 00:00:00.000
Encoded date                             : 2019-05-09 14:28:16.000
Writing application                      : AVID TRMG 3.01
Video
ID                                       : 2
Format                                   : DV
Commercial name                          : DVCPRO
Format settings, wrapping mode           : Frame
Codec ID                                 : 0D01030102020201-0401020202010200
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 24.4 Mb/s
Encoded bit rate                         : 28.8 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 2.357
Time code of first frame                 : 00:00:00:00
Time code source                         : Subcode time code
Stream size                              : 3.36 GiB (76%)
Encoded stream size                      : 3.95 GiB (89%)
Color range                              : Limited
Transfer characteristics                 : BT.709
ChromaSubsampling_Original               : 4:2:0
Audio #1
ID                                       : 2-0
Format                                   : PCM
Muxing mode                              : DV
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Encoded bit rate                         : 0 b/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 216 MiB (5%)
Encoded stream size                      : 0.00 Byte (0%)
Audio #2
ID                                       : 3
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 16 bits
Stream size                              : 108 MiB (2%)
Locked                                   : Yes
Audio #3
ID                                       : 4
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 16 bits
Stream size                              : 108 MiB (2%)
Locked                                   : Yes
Audio #4
ID                                       : 5
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 16 bits
Stream size                              : 108 MiB (2%)
Locked                                   : Yes
Audio #5
ID                                       : 6
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300
Duration                                 : 19 min 39 s
Bit rate mode                            : Constant
Bit rate                                 : 768 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 16 bits
Stream size                              : 108 MiB (2%)
Locked                                   : Yes
Other #1
ID                                       : 1-Material
Type                                     : Time code
Format                                   : MXF TC
Time code of first frame                 : 01:00:00:00
Time code settings                       : Material Package
Time code, striped                       : Yes
Other #2
ID                                       : 1-Source
Type                                     : Time code
Format                                   : MXF TC
Time code of first frame                 : 01:00:00:00
Time code settings                       : Source Package
Time code, striped                       : Yes
Other #3
Type                                     : Time code
Format                                   : SMPTE TC
Muxing mode                              : SDTI
Time code of first frame                 : 01:00:00:00
While ffmpeg (linux) reports as:
[root@st test]# ffmpeg -i RS.MXF
ffmpeg version 2.2.1 Copyright (c) 2000-2014 the FFmpeg developers
built on Apr 13 2014 13:00:18 with gcc 4.4.6 (GCC) 20120305 (Red Hat 4.4.6-4)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --enable-runtime-cpudetect --enable-gpl --enable-version3 --enable-postproc --enable-avfilter --enable-pthreads --enable-x11grab --enable-vdpau --disable-avisynth --enable-frei0r --enable-libopencv --enable-libdc1394 --enable-libgsm --enable-libmp3lame --enable-libnut --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --disable-stripping
libavutil      52. 66.100 / 52. 66.100
libavcodec     55. 52.102 / 55. 52.102
libavformat    55. 33.100 / 55. 33.100
libavdevice    55. 10.100 / 55. 10.100
libavfilter     4.  2.100 /  4.  2.100
libswscale      2.  5.102 /  2.  5.102
libswresample   0. 18.100 /  0. 18.100
libpostproc    52.  3.100 / 52.  3.100
Guessed Channel Layout for  Input Stream #0.1 : mono
Guessed Channel Layout for  Input Stream #0.2 : mono
Guessed Channel Layout for  Input Stream #0.3 : mono
Guessed Channel Layout for  Input Stream #0.4 : mono
Input #0, mxf, from 'RS.MXF':
Metadata:
uid             : 3b13996f-f744-9941-b225-de58406fe6bc
generation_uid  : 5f1a1f46-4d1c-fc41-a08d-f951d207ef49
company_name    : AVID
product_name    : TRMG
product_version : 3.01
product_uid     : 00000000-0000-0000-0000-000000000000
modification_date: 2019-05-09 14:28:16
timecode        : 01:00:00:00
Duration: 00:19:39.52, start: 0.000000, bitrate: 32256 kb/s
Stream #0:0: Video: dvvideo, yuv420p, 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream #0:1: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Stream #0:2: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Stream #0:3: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Stream #0:4: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Moreover if I extract the dv-DIF with old Snell Wilcox MXF unwrap wizard from that MXF mediainfo 2019.4 reports now as 4.2.0 and still erroneous as DVCPRO
General
Complete name                            : C:\bmx\RS.dif
Format                                   : DV
Commercial name                          : DVCPRO
File size                                : 144 MiB
Duration                                 : 41 s 800 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 28.8 Mb/s
Recorded date                            : 1970-01-01 00:00:00.000
Video
Format                                   : DV
Commercial name                          : DVCPRO
Duration                                 : 41 s 800 ms
Bit rate mode                            : Constant
Bit rate                                 : 24.4 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 2.357
Time code of first frame                 : 00:00:00:00
Time code source                         : Subcode time code
Stream size                              : 122 MiB (85%)
Audio
ID                                       : 0
Format                                   : PCM
Duration                                 : 41 s 800 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Encoded bit rate                         : 0 b/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 7.65 MiB (5%)
Avid MediaComposer which has generated the file reports as DV-25
Please check this issue .
Thank You.

Please provide a sample file (small, I only need 1 frame).

I will pm you a link with the file I've tested. As for 1 frame file I can send you tomorrow morning (I have to generate it in Avid)
Thank You.

Chroma subsampling issue:
Your MXF header is wrong, you can see 2 pieces of metadata in the MediaInfo report:
Chroma subsampling : 4:2:2
ChromaSubsampling_Original : 4:2:0

That means that there is a mismatch between container info (4:2:2) and essence real content (4:2:0).
You need to report a bug to your encoding tool vendor with this info about MXF wrong metadata:
000156E  Horizontal colour subsampling - 2 (0x2) (8 bytes)
000156E   Header (4 bytes)
000156E    Code:                                13058 (0x3302)
0001570    Length:                              4 (0x0004)
0001572   Data:                                 2 (0x00000002)
0001576  Vertical colour subsampling - 1 (0x1) (8 bytes)
0001576   Header (4 bytes)
0001576    Code:                                13064 (0x3308)
0001578    Length:                              4 (0x0004)
000157A   Data:                                 1 (0x00000001)

MediaInfo behavior is the expected one (it shows that your file is not conform and it shows MXF value)

DVCPRO issue:
From Wikipedia or adamwilt, and for PAL:

base DV is 4:2:0 and unlocked audio
DVCPRO is 4:1:1 and locked audio
Your file is... 4:20 and locked audio! So "officially" none of them.

00036F7   audio_source (5 bytes)
00036F7    Pack Type:                           80 (0x50)
00036F8    LF - Locked mode:                    Yes

As NTSC base DV is 4:1:1 so no difference at the color space part with NTSC DVCPRO, my arbitrary choice with such stream is to prioritize the locked audio and consider any stream with locked audio as DVCPRO, as this is the most "expensive" feature (locked audio has a cost when in hardware).
@dericed maybe a comment on this DV inconsistency?

On 7/1/2019 10:32 PM, Jérôme Martinez wrote:

Chroma subsampling issue:
Your MXF header is wrong, you can see 2 pieces of metadata in the
MediaInfo report:
|Chroma subsampling : 4:2:2 ChromaSubsampling_Original : 4:2:0 |
That means that there is a mismatch between container info (4:2:2) and
essence real content (4:2:0).
You need to report a bug to your encoding tool vendor with this info
about MXF wrong metadata:
|000156E Horizontal colour subsampling - 2 (0x2) (8 bytes) 000156E
Header (4 bytes) 000156E Code: 13058 (0x3302) 0001570 Length: 4
(0x0004) 0001572 Data: 2 (0x00000002) 0001576 Vertical colour
subsampling - 1 (0x1) (8 bytes) 0001576 Header (4 bytes) 0001576 Code:
13064 (0x3308) 0001578 Length: 4 (0x0004) 000157A Data: 1 (0x00000001) |
MediaInfo behavior is the expected one (it shows that your file is not
conform and it shows MXF value)

DVCPRO issue:
From Wikipedia https://en.wikipedia.org/wiki/DV#DVCPRO or adamwilt
https://www.adamwilt.com/DV-tech.html#Details, and for PAL:

base DV is 4:2:0 and unlocked audio
DVCPRO is 4:1:1 and locked audio
Your file is... 4:20 and locked audio! So "officially" none of them.

|00036F7 audio_source (5 bytes) 00036F7 Pack Type: 80 (0x50) 00036F8
LF - Locked mode: Yes |
As NTSC base DV is 4:1:1 so no difference at the color space part with
NTSC DVCPRO, my arbitrary choice with such stream is to prioritize the
locked audio and consider any stream with locked audio as DVCPRO, as
this is the most "expensive" feature (locked audio has a cost when in
hardware https://www.adamwilt.com/DV-FAQ-tech.html#LockedAudio).
@dericed https://github.com/dericed maybe a comment on this DV
inconsistency?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MediaArea/MediaInfo/issues/365?email_source=notifications&email_token=AB4MQYIKR3WC6MFHZTIDZXDP5JLUNA5CNFSM4H4UTMQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7DYQY#issuecomment-507395139,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB4MQYN56KF76XEZGK3VE6TP5JLUNANCNFSM4H4UTMQA.

My files are PAL ones.
PAL (4.2.0) DV locked audio means DVCAM (not DVCPRO).
As for inconsistencies between container and essence since just one of
colorspace  is the real one I assume mediainfo shall report essence.
Older versions of mediainfo versions reports 4.2.0 (so essence) .
Or maybe better (more exactly) mediainfo should report both colorspaces
in General (mxf side) container one and in Video side essence one.
Rreporting 4.2.2 on a 25Mbit/s DV codec that simply does not exists.
420/411 is 25Mbit/s     422 is DVCPRO50 so 50Mbit/s
I will check with newer versions of Avid if the essence vs container
problem persists.
Thank You for quick verification response.

Later edit. I just noticed that mediainfo reports indeed both chroma subsampling.
Since chroma subsampling is container related maybe it would be better to be displayed in mxf part of description not necessary in essence side.  So you're right. If you decide t

PAL (4.2.0) DV 48KHz/16bit locked audio means DVCAM (not DVCPRO).

Got it, I was not handling DVCAM, so just ideogrammatically selecting between 2 paths (DV or DVCAM) so I'll add this case for PAL.
Now wondering how I can do the difference between DVCAM and DVCPRO for NTSC...

Rreporting 4.2.2 on a 25Mbit/s DV codec that simply does not exists.

MediaInfo is a metadata reporter, and your file says that it exists. This is a bug in the file.
This is exactly the purpose of MediaInfo for some people: checking that file is coherent (yours is not coherent).

in General (mxf side) container one and in Video side essence one.

General section is info not linked to any specific track.
In the case of color subsampling, the piece of metadata is specifically linked to the video track with a specific ID, so it is in the video dedicated part.
Due to historical reasons (when implementing the handling of buggy files, with study about how to implement the feature at lower cost), the choice was to provide both pieces of information with the container metadata first; there is no universal rule about which metadata is used (container or essence) when there is a mismatch between container and essence (for frame rate it is usually the container metadata but not always, for HDR it is usually the essence metadata but not always and so on).

| Now wondering how I can do the difference between DVCAM and DVCPRO for NTSC...
Well NTSC is another world...
Thank You. We can close the issue if you like.
Or we can wait until tomorrow to check with a new version of Avid and post the results here.

\Here  is a file generated with the last version of Avid Mediacomposer:
General
Complete name                       \em\inhouse\test_new_avid.mxf
Format                                   : MXF
Commercial name                          : DV
Format version                           : 1.3
Format profile                           : OP-1a
Format settings                          : Closed / Complete
File size                                : 2.15 MiB
Duration                                 : 40 ms
Overall bit rate                         : 451 Mb/s
Package name                             : Source Package
Encoded date                             : 2019-07-02 09:51:27.968
Writing application                      : Avid Technology, Inc. Avid
MediaProcessor Plug-In <IP_ADDRESS>05.1
Writing library                          : MXF::SDK (4.6.47) on Win64
<IP_ADDRESS>05.1
Video
ID                                       : 512
Format                                   : DV
Format settings, wrapping mode           : Frame
Codec ID                                 : 0D01030102020201-0401020202010200
Duration                                 : 40 ms
Bit rate mode                            : Constant
Bit rate                                 : 24.4 Mb/s
Encoded bit rate                         : 28.8 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 2.357
Stream size                              : 119 KiB (5%)
Encoded stream size                      : 141 KiB (6%)
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Audio #1
ID                                       : 768
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 40 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 24 bits
Stream size                              : 5.63 KiB (0%)
Locked                                   : Yes
Audio #2
ID                                       : 1024
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 40 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 25.000 FPS (1920 SPF)
Bit depth                                : 24 bits
Stream size                              : 5.63 KiB (0%)
Locked                                   : Yes
Other #1
ID                                       : 1-Material
Type                                     : Time code
Format                                   : MXF TC
Time code of first frame                 : 01:00:00:00
Time code settings                       : Material Package
Time code, striped                       : Yes
Other #2
ID                                       : 0-Source
Type                                     : Time code
Format                                   : MXF TC
Time code of first frame                 : 01:00:00:00
Time code settings                       : Source Package
Time code, striped                       : Yes
Other #3
Type                                     : Time code
Format                                   : SMPTE TC
Muxing mode                              : SDTI
Time code of first frame                 : 01:00:00:00
It seems okay as Chroma subsampling  but wrong
as
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
But it doesnt matter. Case closed.
Thank You.

It seems okay as Chroma subsampling but wrong as
Color primaries : BT.709
Transfer characteristics : BT.709

Same kind of issue, it comes from MXF header (there is no such metadata in DV, at least not read by MediaInfo), so MediaInfo shows what it finds in the file.

commercial name issue fixed in development snapshots (mini 20190702) (direct link Windows, direct link macOS).


Provide a way to lazy load AnalyticsBrowser without pulling in all the other dependencies

It appears that at the moment, simply importing import { AnalyticsBrowser } from '@segment/analytics-next'; causes various dependencies get pulled into the main bundle. I can md5 and other things I would not expect.

It is not a dramatic difference, but it would be still nice to address it:
with @segment/analytics-next: 555 993 bytes (69% ununsed)
without @segment/analytics-next: 445 108 bytes (66% ununsed)

I would imagine that there should be an entry point such as:
import { AnalyticsBrowser } from '@segment/analytics-next/lazy';

Where the entry would be along the lines of:
const analytics = {
  identify: () => {},
  load: () => {
    // lazy load Segment and replace the instance
  },
  reset: () => {},
  track: () => {},
};

I don't quite follow. All you need to do is move:
export class AnalyticsBrowser extends AnalyticsBuffered {
  private _resolveLoadStart: (
    settings: AnalyticsBrowserSettings,
    options: InitOptions
  ) => void

  constructor() {
    const { promise: loadStart, resolve: resolveLoadStart } =
      createDeferred<Parameters<AnalyticsBrowser['load']>>()

    super((buffer) =>
      loadStart.then(([settings, options]) =>
        loadAnalytics(settings, options, buffer)
      )
    )

    this._resolveLoadStart = (settings, options) =>
      resolveLoadStart([settings, options])
  }

  /**
   * Fully initialize an analytics instance, including:
   *
   * * Fetching settings from the segment CDN (by default).
   * * Fetching all remote destinations configured by the user (if applicable).
   * * Flushing buffered analytics events.
   * * Loading all middleware.
   *
   * Note:  This method should only be called *once* in your application.
   *
   * @example
   * ```ts
   * export const analytics = new AnalyticsBrowser()
   * analytics.load({ writeKey: 'foo' })
   * ```
   */
  load(
    settings: AnalyticsBrowserSettings,
    options: InitOptions = {}
  ): AnalyticsBrowser {
    this._resolveLoadStart(settings, options)
    return this
  }

  /**
   * Instantiates an object exposing Analytics methods.
   *
   * @example
   * ```ts
   * const ajs = AnalyticsBrowser.load({ writeKey: '<YOUR_WRITE_KEY>' })
   *
   * ajs.track("foo")
   * ...
   * ```
   */
  static load(
    settings: AnalyticsBrowserSettings,
    options: InitOptions = {}
  ): AnalyticsBrowser {
    return new AnalyticsBrowser().load(settings, options)
  }

  static standalone(
    writeKey: string,
    options?: InitOptions
  ): Promise<Analytics> {
    return AnalyticsBrowser.load({ writeKey }, options).then((res) => res[0])
  }
}

to a separate file with a dedicated entry point, and then lazy load loadAnalytics function.
This will allow to initialize core logic without pulling in all dependencies into the main bundle.

Thanks @gajus, I misread what you were asking. We will track this!


Fix refresh method

This PR fixes a crash when calling the refresh method on the Client to refresh the access/refresh tokens. Closes #72
It also fixes the url that is used to obtain a new access token. Since previously it would prefix the path with the baseUri and result in an invalid url like this: https://api.coinbase.com/v2/https://api.coinbase.com/oauth/token?access_token=...

Fixed it, good work!
I´m still getting the Error "request is missing a required parameter, includes an unsupported parameter value, or is otherwise malformed." but that seems like a coinbase issue

Again, why has this not been merged yet?

Will this be merged soon? It seems to have been in limbo for quite a while. Who has write access to this library?

when will this be merged?

Hello,
May I ask if it would be possible to have a merge ? Indeed, when installing the coinbase library with npm (Version 2.0.8), I always have the same error, the code lines are not modified as in your commit.
Thank you in advance.


Problem when setting a JSON Object and accessing it

Today, I faced a strange problem while migrating my project to Polymer 1.0. I have a JSON object which has sub values. After my Polymer component is created and ready to use, I change my JSON object as:
this.school.name = "New School";

After I change the value as shown, the value inside my template is not updated! There is nothing in the  below:
<template>
     <span>{{school.name}}</span>
</template>

I tried everything and finally realise that setting JSON object value in a different way solves my problem:
this.school["name"] = "New School";

I think there is a problem in the serialization process of Polymer for JSON objects. I am not sure if somebody faces the same problem but it needs to be fixed. There are a lot of JSON objects declarations in my project with "." notation and because of this problem, I needed to change them all.

You need to use the set method to notify the data-binding about the changed sub-property (see docs for more information):
This should work:
this.set('school.name','New School');

I've tried this one and it worked, thank you. But I still have a question in my mind. How is it possible to effect DOM by using two different notation while setting a sub-property of a JSON object? Both are giving me the same result when I print them out, but the first one cannot change DOM:
this.school.name = "New School";

this.school["name"] = "New School";

Both assignments to a subproperty now work in Polymer 2: http://jsbin.com/nebazadaze/edit?html,console,output


SQS ExpiredToken: The security token included in the request is expired status code: 403

I have a long-running worker process running on EC2 that consumes items from an SQS queue.  After a time (8-12 hours, I reckon) I begin getting expired security token errors.  I would expect the aws lib to handle the refresh of credentials automatically but this seems not to be the case.
I have tried initializing the SQS object in the following ways:
// Method 1
q := sqs.New(&aws.Config{
	Region: region,
})

// Method 2
q := sqs.New(&aws.Config{
	Credentials: aws.DefaultChainCredentials,
	Region: region,
})

// Method 3
q: = sqs.New(nil)

I experience the errors with the first two of these methods, but I have not yet seen the issue with the third.  Granted, the third method is being utilized on a different EC2 instance, but I am restricted from using it in the current context due to the need to specify the region when running tests and doing local development outside of EC2.
Any help would be greatly appreciated.  Thanks!

@squirkle how are you providing configuration to your process? Are you using IAM roles?

IAM, correct.  The process gets the credentials and runs successfully for a while before failing with the expired token error.

@Isegal--am I correct in assuming that expired credentials should be automatically refreshed by the aws-sdk or is this the responsibility of the calling program?

Correct @squirkl, the request should be retried if the service returned a response with "ExpiredTokenException" error code. Is this exception code you are receiving?
Looking at the expired creds request retry test case, it looks like there might be a bug around the AfterRetry request handler. The test case was doing a bit more mocking that it should, and may of been hiding the bug.

After removing the extra code from the expired creds request test it doesn't look like the issue you're seeing is related to the AfterRetry request handler after all.
Are you seeing the same request being retried multiple times with expired credentials, without refreshing them or is the request not attempting to retry at all, and just returning an error?

I think I've identified the problem, and in am in the process of verifying the fix.
The SDK only considered a single error code when determining if credentials should be refreshed, but there are two other codes that were not being checked.

Hi @squirkle A fix for this bug has been pushed and you should no longer be encountering any more expired token errors.  Please let us know if you run into any further problem.

@jasdel Awesome!  Thanks for the quick response!

@jasdel Unfortunately this issue is still occurring.

Thanks for the feedback @squirkle are you seeing the same error message or a different one?

@jasdel It is the same error but I believe I may not have deployed the fix properly.  I'm going to re-close.  If I see the issue again I will let you know.  Thanks!

Great thanks for the update!


blank

@Slapbox sorry for not getting back to you 😬 . But it looks like you were able to figure out the issue.
Closing this issue. Don't hesitate to let me know if you have any further questions.


Dell U2520D volume control is not working

Before opening the issue, have you...?

[X] Searched for existing issues
[X] Read through the Monitor-Troubleshooting Wiki
[X] Updated MonitorControl to the latest version (if applicable)

Describe the issue
Volume control isnt working while seeing the volume variation via native macos volume panel.i am sure DDC is working well.
The brightness control is fine.
Mac->U2520D->external speaker with USB and 3.5mm audio output
Expected behavior
Adjusting volume.
Anything else?
No response
Environment Information (please complete the following information)
- macOS version:11.4 Big Sur
- Mac model:Macbook Pro (15-inch,mid 2014)
- MonitorControl version:4.1.0 version 7034
- Monitor(s):DELL U2520D
- Monitor Cable(s)/Connection(s):Mac->DP2.1->DELL U2520D
- Apple Silicon/M1 (yes or no):no

I'm seeing the same issue with my Dell U2718Q. Brightness control works perfectly, but audio volume will not adjust.

Also, my monitor doesn't have integrated speakers so I'm using the audio-out jack on it to external speakers. It seems like other folks with Dell and Samsung monitors with external speakers are also having trouble (#1042). I wonder if the issue might be related to external vs internal speakers?

Doesn't work for my Benq PD2705U neither.

Same here with DELL U2722DE

I've the same issue with a DELL U2723QE. Brightness works fine, audio doesn't.

Many displays support only brightness control via DDC and no audio support. My display (Lenovo Creator Extreme) also has no volume controls via DDC. So I guess I'll join the list. ;)

I've the same issue with a Dell S2721H. Brightness works fine, audio doesn't. Connected through HDMI.

I connected via type c

Audio control via HDMI will not work as HDMI on Mac blocks DDC. Otherwise these are possibly display firmware issues.
I'll move this thread over to discussions so everybody can chime in with their display.


Fastlane 2.225.0 fails to launch with error about sysrandom

New Issue Checklist

[X] Updated fastlane to the latest version
[X] I read the Contribution Guidelines
[X] I read docs.fastlane.tools
[X] I searched for existing GitHub issues

Issue Description

I'm running Fastlane 2.225.0 installed from Homebrew on macOS Sequoia. I think the Ruby 3.3.6 update broke Fastlane, because it works on a computer that's still on 3.3.5. When I launch Fastlane, I get an error about sysrandom gem, even if it's just running fastlane --version in a folder with no Fastfile.
Command executed
fastlane --version
Complete output when running fastlane, including the stack trace and command used

% fastlane --version
/opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1417:in `rescue in block in activate_dependencies': Could not find 'sysrandom' (~> 1.0) among 154 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=/Users/buildkite/.local/share/fastlane/3.3.0:/opt/homebrew/Cellar/fastlane/2.225.0/libexec' at: /opt/homebrew/Cellar/fastlane/2.225.0/libexec/specifications/fastlane-sirp-1.0.0.gemspec, execute `gem env` for more information
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1414:in `block in activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `each'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1385:in `activate'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1421:in `block in activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `each'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1385:in `activate'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:280:in `block in activate_bin_path'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:279:in `synchronize'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:279:in `activate_bin_path'
        from /opt/homebrew/Cellar/fastlane/2.225.0/libexec/bin/fastlane:25:in `'
/opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/dependency.rb:303:in `to_specs': Could not find 'sysrandom' (~> 1.0) - did find: [sysrandom-1.0.5] (Gem::MissingSpecVersionError)
Checked in 'GEM_PATH=/Users/buildkite/.local/share/fastlane/3.3.0:/opt/homebrew/Cellar/fastlane/2.225.0/libexec' , execute `gem env` for more information
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1415:in `block in activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `each'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1385:in `activate'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1421:in `block in activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `each'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1403:in `activate_dependencies'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems/specification.rb:1385:in `activate'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:280:in `block in activate_bin_path'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:279:in `synchronize'
        from /opt/homebrew/Cellar/ruby/3.3.6/lib/ruby/3.3.0/rubygems.rb:279:in `activate_bin_path'
        from /opt/homebrew/Cellar/fastlane/2.225.0/libexec/bin/fastlane:25:in `'

Environment
fastlane env fails the same as --version. Here's what sw_vers says:
ProductName:            macOS
ProductVersion:         15.1
BuildVersion:           24B83

Hello,
You're encountering an issue where Fastlane is not working after updating Ruby to version 3.3.6 on macOS Sequoia, and it’s throwing an error related to the sysrandom gem. Since Fastlane works fine on a machine with Ruby 3.3.5, the issue seems to be specifically related to the new Ruby version.
Here’s a step-by-step checklist of things you can try to resolve this issue:

Check Ruby Version

Verify the exact Ruby version you're running with:
ruby -v
Ensure it is Ruby 3.3.6 (or whatever version you suspect is causing the issue).
2. Reinstall sysrandom Gem
The error might be related to a gem conflict or installation issue, especially with the updated Ruby version. Try reinstalling the sysrandom gem:
gem uninstall sysrandom
gem install sysrandom
3. Reinstall Fastlane
Since Fastlane is often tightly coupled with Ruby and its gems, try reinstalling Fastlane to ensure all dependencies are correct:brew uninstall fastlane
brew install fastlane
Alternatively, if you installed Fastlane via other means (like gem install fastlane), try reinstalling using that method.
4. Update Homebrew and Dependencies
Ensure that both Homebrew and all installed dependencies are up-to-date:brew update
brew upgrade
brew doctor
After updating Homebrew, you can try reinstalling Fastlane again.
5. Check for Known Issues on GitHub
Since you're already familiar with searching GitHub issues, check the Fastlane GitHub Issues for any reported problems related to Ruby 3.3.6 and sysrandom. You might find others with the same issue, along with possible solutions or workarounds.
6. Try a Different Ruby Version (Temporarily)
If the issue is definitely tied to Ruby 3.3.6, you could temporarily downgrade your Ruby version to 3.3.5 (which is working for you) using a version manager like rbenv or rvm:
Install rbenv (if not already installed):brew install rbenv

nstall Ruby 3.3.5:
rbenv install 3.3.5
rbenv global 3.3.5
After switching Ruby versions, reinstall Fastlane:
gem install fastlane
7. Check for macOS Compatibility
Given that you're on macOS Sequoia (macOS 15.1), ensure that your Fastlane version (2.225.0) and Ruby are fully compatible with the macOS version you're using. Sometimes, newer macOS versions introduce compatibility issues with certain tools, especially after system updates.
8. Run fastlane env in Debug Mode
Run fastlane env again with debugging to capture more detailed logs. This might help pinpoint where the issue is occurring: FASTLANE_DEBUG=1 fastlane env
Look for anything unusual in the output, especially related to the sysrandom gem or Ruby environment.
9. Verify Dependencies
Sometimes, system dependencies like OpenSSL or other libraries can cause issues with gem installations. Make sure all required system libraries are correctly installed:brew install openssl
hope that helps!

I worked around the problem by switching the project over to using asdf to install Ruby 3.3.5 and fastlane.

any updates?

I ran into this issue as well after updating to Fastlane 2.226.0 on macOS 15.2. Resolved it by uninstalling Fastlane using brew uninstall fastlane and then re-installing using brew install fastlane.

In my case, I fixed it, uninstalling ruby, reinstalling it with brew and then installing Fastlane

This fixed it for me. Downgrading to a specific version (2.224.0 of fastlane) with brew has become unnecessarily complex.
This version is the most recent working version. After this my eas build (expo) worked again.
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/4209dee/Formula/fastlane.rb

brew tap-new $USER/fastlane
brew tap homebrew/core --force
brew extract --version=2.224.0 fastlane $USER/fastlane
brew install<EMAIL_ADDRESS>


Adding new doc for TeamsNetworkRoamingPolicy

Adding new docs for Get, New, Set and Remove TeamsNetworkRoamingPolicy

Docs Build status updates of commit cdb540c:
:warning: Validation status: warnings

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit c310d18:
:warning: Validation status: warnings

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:warning:Warning
View (skype-ps)
Details

skype/skype-ps/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md

Line 0, Column 0: [Warning-author-not-found] Invalid value for author: 'tristan-msft' is not a valid GitHub ID.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 521567c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 79a75a3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@TristanChen-msft, If you have a Microsoft alias, you can link your accounts in GitHub so that a blue "Microsoft submitter" label appears to any PR's or issue you submit. Guidance on it here: https://docs.opensource.microsoft.com/tools/github/index.html
Here is the site you can use to see the mappings between GitHub and Microsoft aliases: https://repos.opensource.microsoft.com/people?q=TristanChen-msft

Docs Build status updates of commit 73f52d4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 213bb7f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 7db8c85:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit fa66566:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/New-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Remove-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsNetworkRoamingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Support resource arrays in usage file

Terraform resources can be created using for_each or count. This creates an array of resources. We could support setting usage values for all elements of the array by using a wildcard like this:
resource_usage:
  aws_cloudwatch_log_group.loggroup[*]:
    ...

I think expected functionality would be that the usage value should apply individually to each element of the array. But an alternative is that the usage value is the aggregated using across all elements of the array.
Relevant code: https://github.com/infracost/infracost/blob/master/internal/providers/terraform/parser.go#L101-L105
Unit tests can be added to cover the new functionality.
Also add a new integration test to cover this. This can be added in a file called parser_test.go (separate from parser_internal_test.go) within a package called parser_test.

@alikhajeh1 hey, I want to work on it

Released today in v0.8.4 - thanks @ekifel!


Running out of memory during build_event_and_measurement_dfs and polars dataset

I am running out of memory when running the build_dataset script with parquet input sources. Here is where I run out of memory:
https://github.com/mmcdermott/EventStreamGPT/blob/5f20eaaf2b7b1896205a17d0e75b302d1a932a03/EventStream/data/dataset_base.py#L266-L282
With this being one of the likely culprits:
https://github.com/mmcdermott/EventStreamGPT/blob/5f20eaaf2b7b1896205a17d0e75b302d1a932a03/EventStream/data/dataset_polars.py#L388-L392
Any suggestions on how this might be optimized?

@juancq how much RAM does the running machine have? And I presume this is with limitations to the number of active threads, yes?

@mmcdermott the machine I'm using has 31GB of usable memory. I have two input sources. Input source 1 is a parquet of size 3.7GB (compressed) and input source 2 is a parquet of size 2.1 GB (compressed). I am using only five fields from input source 1 and five fields from input source 2. The total number of subjects is 9 million (median number of events per subject is 3).
I can run the build_dataset script by limiting polars active threads and using about a third of the data and only input source 1. But with the two input sources and all the data (with or without streaming), it keeps crashing during build_event_and_measurement_dfs.
The problem could be entirely Polars (with various current issues documenting memory issues).
If you run https://github.com/mmcdermott/EventStreamGPT/blob/dev/sample_data/generate_synthetic_data.ipynb and change the number of subjects to 1 million or more, you might be able to replicate the issues I'm having.
As a solution, I'm leaning towards running build_dataset separately on chunks of data, and then implementing a lazy pytorch dataloader pretraining a model that separately loads the cached DL representations from the separate build_dataset runs.

Thanks for the additional information. I've had what sounds like similar
issues on some larger datasets and larger machines. Sharding the patients
is ultimately likely the way to fix the problem most reliably.
That said, in the interest of debugging, can you load up your two input
files in full in memory, or do they take up too much RAM to load all at
once? How much space do they take on disk if you save them in uncompressed
format? Even with streaming, Polaris still too aggressively tried to load
everything into memory, which may not be feasible on a 31GB machine.
On Tue, Aug 15, 2023, 4:11 AM Juan Quiroz Aguilera @.***>
wrote:

@mmcdermott https://github.com/mmcdermott the machine I'm using has
31GB of usable memory. I have two input sources. Input source 1 is a
parquet of size 3.7GB (compressed) and input source 2 is a parquet of size
2.1 GB (compressed). I am using only five fields from input source 1 and
five fields from input source 2. The total number of subjects is 9 million
(median number of events per subject is 3).
I can run the build_dataset script by limiting polars active threads and
using about a third of the data and only input source 1. But with the two
input sources and all the data (with or without streaming), it keeps
crashing during build_event_and_measurement_dfs.
The problem could be entirely Polars (with various current issues
documenting memory issues).
If you run
https://github.com/mmcdermott/EventStreamGPT/blob/dev/sample_data/generate_synthetic_data.ipynb
and change the number of subjects to 1 million or more, you might be able
to replicate the issues I'm having.
As a solution, I'm leaning towards running build_dataset separately on
chunks of data, and then implementing a lazy pytorch dataloader pretraining
a model that separately loads the cached DL representations from the
separate build_dataset runs.
—
Reply to this email directly, view it on GitHub
https://github.com/mmcdermott/EventStreamGPT/issues/54#issuecomment-1678568986,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADS5X56GESB6IPGTAREPALXVMVK5ANCNFSM6AAAAAA3MRXQHI
.
You are receiving this because you were mentioned.Message ID:
@.***>

@mmcdermott very good debugging idea!
I cannot load (using read_parquet) either of the input source parquets by themselves. I run out of memory, so this is all on polars. This is without restricting which columns I load.
If I load (using read_parquet) only the columns defined in the config file, I successfully load (1) the two input sources mentioned above and (2) all of the subjects input source into memory, using only about 56% of available memory. Memory usage is even lower (about 40%) if I use scan_parquet and then collect only the columns defined in the config file.

Ok, so that suggests to me that there is likely some inefficient optimization of the polars queries going on, in which polars is trying to read in the full files prior to filtering down to only the requisite columns maybe? Or that some join is somehow blowing up memory even though it really shouldn't have that sort of effect.
I encountered a similar issue recently in a different branch that required some more explicit management of the polars state to optimize, so I can try to apply similar logic here. Is your data publicly available @juancq, or private? It would help to have a realistic test case that is of a modest but non-trivial size so I can appropriate see how various changes impact the memory footprint. No pressure if it is not shareable, of course, just figured I'd check to make it easier to debug.

@mmcdermott The dataset is private, so I cannot share it. But I am making a modification of your synthetic data generation script that will be a good proxy for the data I have.

No worries, @juancq -- feel free to not worry about the synthetic data generation example. Given the issue here is performance working with larger datasets or on smaller machines should give me enough to work with as a general rule, I just thought things would be more precise with the exact set-up you're working with.

@mmcdermott Here is a script that roughly simulates the dataset that I have. The columns generated in this script are the ones that I define in my config for building the dataset.
sample_data/generate_synthetic_hospital_admin_data.py

My actual dataset has about 100 other columns that I don't use (I didn't bother including 100 dummy string columns in the data generation script).

This is great @juancq , thank you! Do you think it would be possible to make this script play double duty and add another block for generating an emergency file to simulate your issue with shared names across files from https://github.com/mmcdermott/EventStreamGPT/issues/53? As I commented in that issue, when I try to replicate it on my current synthetic data set-up, I don't have a problem with duplicate names, actually. Regardless I'll try to improve performance with this example for this issue proper. Thanks again!

@juancq Is this still an issue? I'd like to close it if not so.


Using TensorFlow backend with tensorflow-directml-plugin

Using TensorFlow backend
Epoch 1/3
Traceback (most recent call last):
File "E:\custom_train_step_in_tensorflow.py", line 131, in
model.fit(x, y, epochs=3)
File "C:\Python_310\lib\site-packages\keras_core\src\utils\traceback_utils.py", line 123, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Python_310\lib\site-packages\tensorflow\python\eager\execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
Detected at node 'StatefulPartitionedCall' defined at (most recent call last):
File "E:\custom_train_step_in_tensorflow.py", line 131, in
model.fit(x, y, epochs=3)
File "C:\Python_310\lib\site-packages\keras_core\src\utils\traceback_utils.py", line 118, in error_handler
return fn(*args, **kwargs)
File "C:\Python_310\lib\site-packages\keras_core\src\backend\tensorflow\trainer.py", line 307, in fit
logs = self.train_function(iterator)
File "C:\Python_310\lib\site-packages\keras_core\src\backend\tensorflow\trainer.py", line 112, in one_step_on_iterator
outputs = self.distribute_strategy.run(
Node: 'StatefulPartitionedCall'
Could not find compiler for platform DML: NOT_FOUND: could not find registered compiler for platform DML -- was support for that platform linked in?
[[{{node StatefulPartitionedCall}}]] [Op:__inference_one_step_on_iterator_697]
The original code is at https://github.com/keras-team/keras-io/blob/master/guides/keras_core/custom_train_step_in_tensorflow.py.

Could not find compiler for platform DML: NOT_FOUND: could not find registered compiler for platform DML -- was support for that platform linked in?

This error typically means that your TF runtime is not working. You can try to run any piece of TF code (in graph mode) to check.

Could not find compiler for platform DML: NOT_FOUND: could not find registered compiler for platform DML -- was support for that platform linked in?

This error typically means that your TF runtime is not working. You can try to run any piece of TF code (in graph mode) to check.

The problem is self.jit_compile = jit_compile on line 71 in keras-core\keras_core\trainers\trainer.py. Because the computing device of tensorflow-directml-plugin has been set to 'GPU', but the general strategy adopted by keras requires self.jit_compile =True, which makes the computing device of xla seem to be unable to find. The logic of the code here should be modified to self.jit_compile = False. The suggested code is:
'''python
if jit_compile and backend.backend() == "tensorflow":
from keras_core.utils.module_utils import tensorflow as tf
         if tf.config.list_physical_devices("GPU"):
             self.jit_compile = False
         else:
             self.jit_compile = jit_compile
     else:
         self.jit_compile = jit_compile

'''

Thanks for the suggestion. It makes sense that you couldn't use XLA with DirectML.
But list_physical_devices doesn't seem like a reliable way to detect DirectML, no? It will be non-empty if there's any GPU on your machine.

Thanks for the suggestion. It makes sense that you couldn't use XLA with DirectML.
But list_physical_devices doesn't seem like a reliable way to detect DirectML, no? It will be non-empty if there's any GPU on your machine.

Thank you for your reply. I mostly use it on Windows, but TensorFlow (>1.11.0) recently ended CUDA support. For acceleration, I used TensorFlow based on DirectML-Plugin. When running keras-core, there is a problem that the XLA device cannot be found. Unfortunately, I can't give the code to detect the presence of DirectML, all I can do is describe the line of code where the problem might be.


Shape Keys applied to wrong target in Blender 2.8

Exporting a GLB out of Blender applies the blend shape attributes to the wrong morph target.
In my example, the "Glee Eyes" target is being applied to the "Super Open" target. We have tried removing the "Super Open" target completely and re exporting and it just gets applied elsewhere.
Blend and GLB file: https://drive.google.com/drive/folders/14YZx1q0INs-SvKSFIdRv8dy_2QTbapA0?usp=sharing

Confirmed. What I suspect is that the problem come from inactive shapekeys ( Super Open and Frown).
Can you please confirm that everything is OK if you delete these 2 shapekeys (delete only 1 is not enough) ?

Am checking right now, thankyou for such a swift reply to this!

It seems to move the behaviour to the Left Wink, updated file in original link

Confirmed that something is really weird. Will check


fix(rpc): handle nil value in preapply result

Seems that sometimes POST /blocks/head/helpers/preapply/operations returns [%{"contents" => preapplied_operations}] with preapplied_operations == nil, this PR handles this case.

Sure what do you think of {:error, :nil_preapply}?


Allow code to be installed in Globals

Currently the installation creates 2 new SymbolDictionaries in SystemUser's symbol list: #GemConnectForPostgres and #GemConnectForPostgres_Tests . It would be nice to have the option to install all the code into Globals instead of creating new SymbolDictionaries.

Added a new install script install.into.globals.topaz to install code into Globals.


EXCEPTION CAUGHT BY WIDGETS LIBRARY

 ══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
I/flutter ( 8169): The following assertion was thrown building:
I/flutter ( 8169): Assertion failed
I/flutter ( 8169):
I/flutter ( 8169): Either the assertion indicates an error in the framework itself, or we should provide substantially
I/flutter ( 8169): more information in this error message to help you determine and fix the underlying cause.
I/flutter ( 8169): In either case, please report this assertion by filing a bug on GitHub:
I/flutter ( 8169):   https://github.com/flutter/flutter/issues/new?template=BUG.md
I/flutter ( 8169):
I/flutter ( 8169): When the exception was thrown, this was the stack:
I/flutter ( 8169): #0      new MultiChildRenderObjectWidget.<anonymous closure> (package:flutter/src/widgets/framework.dart:1691:11)
I/flutter ( 8169): #1      new MultiChildRenderObjectWidget (package:flutter/src/widgets/framework.dart:1696:8)
I/flutter ( 8169): #2      new Flex (package:flutter/src/widgets/basic.dart:3707:8)
I/flutter ( 8169): #3      new Column (package:flutter/src/widgets/basic.dart:4234:8)
I/flutter ( 8169): #4      ContactState.build.<anonymous closure> (package:flutterdemo/contacts/contact_sider_list.dart:72:26)
I/flutter ( 8169): #5      SliverChildBuilderDelegate.build (package:flutter/src/widgets/sliver.dart:442:15)
I/flutter ( 8169): #6      SliverMultiBoxAdaptorElement._build.<anonymous closure> (package:flutter/src/widgets/sliver.dart:1153:67)
I/flutter ( 8169): #7      _HashMap.putIfAbsent (dart:collection-patch/collection_patch.dart:137:29)
I/flutter ( 8169): #8      SliverMultiBoxAdaptorElement._build (package:flutter/src/widgets/sliver.dart:1153:26)
I/flutter ( 8169): #9      SliverMultiBoxAdaptorElement.createChild.<anonymous closure> (package:flutter/src/widgets/sliver.dart:1166:55)
I/flutter ( 8169): #10     BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2320:19)
I/flutter ( 8169): #11     SliverMultiBoxAdaptorElement.createChild (package:flutter/src/widgets/sliver.dart:1159:11)
I/flutter ( 8169): #12     RenderSliverMultiBoxAdaptor._createOrObtainChild.<anonymous closure> (package:flutter/src/rendering/sliver_multi_box_adaptor.dart:354:23)
I/flutter ( 8169): #13     RenderObject.invokeLayoutCallback.<anonymous closure> (package:flutter/src/rendering/object.dart:1715:58)
I/flutter ( 8169): #14     PipelineOwner._enableMutationsToDirtySubtrees (package:flutter/src/rendering/object.dart:796:15)
I/flutter ( 8169): #15     RenderObject.invokeLayoutCallback (package:flutter/src/rendering/object.dart:1715:13)
I/flutter ( 8169): #16     RenderSliverMultiBoxAdaptor._createOrObtainChild (package:flutter/src/rendering/sliver_multi_box_adaptor.dart:343:5)
I/flutter ( 8169): #17     RenderSliverMultiBoxAdaptor.insertAndLayoutChild (package:flutter/src/rendering/sliver_multi_box_adaptor.dart:489:5)
I/flutter ( 8169): #18     RenderSliverList.performLayout.advance (package:flutter/src/rendering/sliver_list.dart:191:19)
I/flutter ( 8169): #19     RenderSliverList.performLayout (package:flutter/src/rendering/sliver_list.dart:234:19)
I/flutter ( 8169): #20     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #21     RenderSliverPadding.performLayout (package:flutter/src/rendering/sliver_padding.dart:181:11)
I/flutter ( 8169): #22     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #23     RenderViewportBase.layoutChildSequence (package:flutter/src/rendering/viewport.dart:406:13)
I/flutter ( 8169): #24     RenderViewport._attemptLayout (package:flutter/src/rendering/viewport.dart:1334:12)
I/flutter ( 8169): #25     RenderViewport.performLayout (package:flutter/src/rendering/viewport.dart:1252:20)
I/flutter ( 8169): #26     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #27     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #28     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #29     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #30     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #31     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #32     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #33     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #34     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #35     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #36     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #37     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #38     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #39     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #40     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #41     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #42     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #43     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #44     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #45     RenderStack.performLayout (package:flutter/src/rendering/stack.dart:510:15)
I/flutter ( 8169): #46     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #47     MultiChildLayoutDelegate.layoutChild (package:flutter/src/rendering/custom_layout.dart:142:11)
I/flutter ( 8169): #48     _ScaffoldLayout.performLayout (package:flutter/src/material/scaffold.dart:443:7)
I/flutter ( 8169): #49     MultiChildLayoutDelegate._callPerformLayout (package:flutter/src/rendering/custom_layout.dart:212:7)
I/flutter ( 8169): #50     RenderCustomMultiChildLayoutBox.performLayout (package:flutter/src/rendering/custom_layout.dart:356:14)
I/flutter ( 8169): #51     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #52     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #53     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #54     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #55     _RenderCustomClip.performLayout (package:flutter/src/rendering/proxy_box.dart:1214:11)
I/flutter ( 8169): #56     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #57     MultiChildLayoutDelegate.layoutChild (package:flutter/src/rendering/custom_layout.dart:142:11)
I/flutter ( 8169): #58     _ScaffoldLayout.performLayout (package:flutter/src/material/scaffold.dart:443:7)
I/flutter ( 8169): #59     MultiChildLayoutDelegate._callPerformLayout (package:flutter/src/rendering/custom_layout.dart:212:7)
I/flutter ( 8169): #60     RenderCustomMultiChildLayoutBox.performLayout (package:flutter/src/rendering/custom_layout.dart:356:14)
I/flutter ( 8169): #61     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #62     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #63     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #64     _RenderProxyBox&RenderBox&RenderObjectWithChildMixin&RenderProxyBoxMixin.performLayout (package:flutter/src/rendering/proxy_box.dart:105:13)
I/flutter ( 8169): #65     _RenderCustomClip.performLayout (package:flutter/src/rendering/proxy_box.dart:1214:11)
I/flutter ( 8169): #66     RenderObject.layout (package:flutter/src/rendering/object.dart:1619:7)
I/flutter ( 8169): #67     MultiChildLayoutDelegate.layoutChild (package:flutter/src/rendering/custom_layout.dart:142:11)
I/flutter ( 8169): #68     _ScaffoldLayout.performLayout (package:flutter/src/material/scaffold.dart:443:7)
I/flutter ( 8169): #69     MultiChildLayoutDelegate._callPerformLayout (package:flutter/src/rendering/custom_layout.dart:212:7)
I/flutter ( 8169): #70     RenderCustomMultiChildLayoutBox.performLayout (package:flutter/src/rendering/custom_layout.dart:356:14)
I/flutter ( 8169): #71     RenderObject._layoutWithoutResize (package:flutter/src/rendering/object.dart:1496:7)
I/flutter ( 8169): #72     PipelineOwner.flushLayout (package:flutter/src/rendering/object.dart:765:18)
I/flutter ( 8169): #73     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding&PaintingBinding&SemanticsBinding&RendererBinding.drawFrame (package:flutter/src/rendering/binding.dart:346:19)
I/flutter ( 8169): #74     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding&PaintingBinding&SemanticsBinding&RendererBinding&WidgetsBinding.drawFrame (package:flutter/src/widgets/binding.dart:701:13)
I/flutter ( 8169): #75     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding&PaintingBinding&SemanticsBinding&RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:285:5)
I/flutter ( 8169): #76     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1016:15)
I/flutter ( 8169): #77     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:958:9)
I/flutter ( 8169): #78     _WidgetsFlutterBinding&BindingBase&GestureBinding&ServicesBinding&SchedulerBinding._handleDrawFrame (package:flutter/src/scheduler/binding.dart:874:5)
I/flutter ( 8169): #82     _invoke (dart:ui/hooks.dart:236:10)
I/flutter ( 8169): #83     _drawFrame (dart:ui/hooks.dart:194:3)
I/flutter ( 8169): (elided 3 frames from package dart:async)
I/flutter ( 8169): ════════════════════════════════════════════════════════════════════════════════════════════════════
I/flutter ( 8169): Another exception was thrown: Assertion failed

Please give this a more meaningful title, I have no clue what happened here
It says Assertion failed: . Is there anything after that? What caused this? Can you post your code? Namely ContactState.build. Much more detail is needed here.

@mmzuo Is it still relevant?

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact descriptions what happens, logs and the output of ' flutter doctor -v'  please.
All system setups can be slightly different so its always better to open new issues and reference related issues.


fixed issue 4292 by updating latex drawer

Summary
Fixes #4292
Details and comments
Fixed the issue by adding the check for initial_state input for classical bits (was present only for qubits). Now the classical and quantum inputs consistently show up depending on the value of the initial_state flag as suggested by @1ucian0 in the original issue.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[windows] Avoid %r for quoting module-cache-path in Windows.

%r returns a representation of the object that is valid Python syntax.
For numbers and strings this representation is very compatible with
Unix shells, but for Windows, the quoting performed by Python will not
be compatible. For example the Windows path separator \ will be
escaped as \\, which is not necessary for Windows and might make
some tests that try to match path fail.
Python 3.3 has shlex.quote, which was previously available as
pipes.quote in earlier Python versions. pipes.quote was indeed
used in several points of the lit.cfg file. For Windows, one need to
do their own quoting.
This change introduces shell_quote which uses shlex.quote or
pipes.quote in Unix depending on the Python version, and provides an
implementation of shell_quote for Windows. It replaces every usage
of pipes.quotes for shell_quote, and modifies the value of
mcp_opt to use shell_quote and not %r.
This should fix the test Driver\working-directory.swift in the
Windows Visual Studio 2017 builder.

@swift-ci please smoke test

I might replace every usage of %r at some point, but for now, let's introduce only one change at the time.

Seems that re.replace doesn't work with the windows builders?

Sorry. I run it in several of the test directories, but not all of them. It should have been s.replace(…) not re.replace(…).
@swift-ci please smoke test

@swift-ci please test Windows platform

The problem now is that because the \ are not longer escaped. There are usages of Lit's SubstituteCaptures (https://github.com/apple/llvm-project/blob/1a6659f0e7add3f1c1f5352fbf5b9811bac1adfc/llvm/utils/lit/lit/TestingConfig.py#L167) that use string interpolation with the non-escaped values. Because those strings end up as replacement strings of a regular expression, they have to be escaped for the replacement string (easy to do with .replace('\\', '\\\\')).
But then the working-directory.swift test starts to fail for some reason (FML). It seems that the matching is performed against some output of the Swift driver, which outputs the jobs command lines escaped, but probably escaped for Unix. I have to circle back here. The fix for one particular test might be easier than this, even if this should be good in the long run.
I don't know, however, if the card castle of escaping and unescaping can ever be solved.
PD: I still don't understand why it works in Windows VS2019.

So I think I understand why it works in the VS 2019 nodes: they have Python 3.7.5 installed (comes from Visual Studio from reading the logs), while the VS 2017 nodes have 3.8.5 (installed with Chocolatey).
For os.path.realpath the documentation says "Changed in version 3.8: Symbolic links and junctions are now resolved on Windows.". When LLVM Lit starts it does a realpath on the lit configuration path which gives T:\swift\test-windows-x86_64 in Python 3.7.5, but C:\jenkins\workspace\oss-swift-windows-x86_64\build\swift\test-windows-x86_64 in Python 3.8.5.
And then enter PathSanitizingFileCheck into the game: it provides both SOURCE_DIR and BUILD_DIR, but BUILD_DIR value comes from CMake, which doesn't do any realpath shenanigans, so when Python 3.8.5 uses paths like C:\jenkins\workspace\oss-swift-windows-x86_64\build\swift\test-windows-x86_64 as the build directory, PathSanitizingFileCheck still uses T:\swift as prefix, and do not matches both.
To put the cherry on top, if one uses realpath for the substitutions of BUILD_DIR, PathSanitizingFileCheck stops working because those paths where always using forward slashes, and realpath changes everything to forward slashes.
Since this problem will start appearing when the VS 2019 nodes update to Python 3.8.5, I don't think the solution is going back to 3.7.5 in the VS 2017 nodes, but the solution is not going to be pretty.

@swift-ci please test Windows platform

@swift-ci please smoke test

@swift-ci please test Windows platform

@swift-ci please smoke test

os.normpath should convert the path to the canonical spelling for the platform - that is / will be converted to \ which might be what needs to be applied here?

Do not merge yet. The Windows CI failed badly. I have to look and figure out what's happening in that machine (it is quite hard to get things right because I have to copy code back and forth between machines).

@swift-ci please test Windows platform

Removed the part that adds carets. Seems that Lit already does some massaging of the arguments, and the quoting I introduced was misbehaving with the one Lit implements. I tested having a space in my Python path, like it seems to happen in the build machine.
@swift-ci please test Windows platform

@swift-ci please smoke test

@swift-ci please smoke test Linux platform

@swift-ci please test Linux platform

Build failed
Swift Test Linux Platform
Git Sha - 8470a6624f08283719af2cbc6fdc10a2163d4888

@swift-ci please test Linux platform


Are writekey and onlykeyed output options implemented?

Hi,
I tried running AlphaImpute2 with the following code:
AlphaImpute2 -genotypes data/Genotypes_all.txt
-pedigree data/ped_certain.txt
-out outputs/ai2_SW_geno -writekey genotypes -onlykeyed
-cycles 10 -maxthreads 12
I was hoping to get back only the imputation for those individuals that we have sequenced and we know their positions in the PLINK file we're using, but the -writekey and -onlykeyed flags are ignored and the run outputs all the individuals it can find in the pedigree and dummy individuals in alphanumeric order, instead of the order in the genotype file. I was looking at the AlphaImpute2 script and I noticed that these flags are never invoked. Is this feature not implemented as it says in the manual? I appreciate any information.
Kind regards,
Sergio

@sergiogonmoll thanks for opening this issue. Indeed, the main developers rushed the public push to GitHub ahead of their departure from Roslin. Hence, some parts of the suite are not well tested. Could you create a minimal reproducible example so that we can look into the issue - it should not be too hard to fix this and we would be happy to work with you on this;)


Unable to build productionRelease

It looks like lint isn't happy with passing a raw number to res.getColor/getDimension
› ./gradlew assembleProductionRelease
:preBuild UP-TO-DATE
:preProductionReleaseBuild UP-TO-DATE
:checkProductionReleaseManifest
:preInternalDebugBuild UP-TO-DATE
:preInternalReleaseBuild UP-TO-DATE
:preProductionDebugBuild UP-TO-DATE
:prepareComAndroidSupportAppcompatV72301Library UP-TO-DATE
:prepareComAndroidSupportDesign2301Library UP-TO-DATE
:prepareComAndroidSupportRecyclerviewV72301Library UP-TO-DATE
:prepareComAndroidSupportSupportV42301Library UP-TO-DATE
:prepareComF2prateekRxPreferencesRxPreferences100Library UP-TO-DATE
:prepareComJakewhartonRxbindingRxbinding020Library UP-TO-DATE
:prepareComJakewhartonThreetenabpThreetenabp102Library UP-TO-DATE
:prepareComJakewhartonTimberTimber401Library UP-TO-DATE
:prepareComSquareupLeakcanaryLeakcanaryAndroidNoOp131Library UP-TO-DATE
:prepareIoReactivexRxandroid101Library UP-TO-DATE
:prepareProductionReleaseDependencies
:compileProductionReleaseAidl UP-TO-DATE
:compileProductionReleaseRenderscript UP-TO-DATE
:generateProductionReleaseBuildConfig
:generateProductionReleaseAssets UP-TO-DATE
:mergeProductionReleaseAssets UP-TO-DATE
:generateProductionReleaseResValues UP-TO-DATE
:generateProductionReleaseResources UP-TO-DATE
:mergeProductionReleaseResources UP-TO-DATE
:processProductionReleaseManifest UP-TO-DATE
:processProductionReleaseResources UP-TO-DATE
:generateProductionReleaseSources
:processProductionReleaseJavaRes UP-TO-DATE
:compileProductionReleaseJavaWithJavac
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
:compileRetrolambdaProductionRelease
:compileProductionReleaseNdk UP-TO-DATE
:compileProductionReleaseSources
:lintVitalProductionRelease
/Users/mattm/Code/u2020/build/generated/source/apt/production/release/com/jakewharton/u2020/ui/MainActivity$$ViewBinder.java:19: Error: Expected resource of type color [ResourceType]
    target.statusBarColor = res.getColor(2131361859);
                                         ~~~~~~~~~~
/Users/mattm/Code/u2020/build/generated/source/apt/production/release/com/jakewharton/u2020/ui/trending/TrendingView$$ViewBinder.java:41: Error: Expected resource of type dimen [ResourceType]
    target.dividerPaddingStart = res.getDimension(2131099747);
                                                  ~~~~~~~~~~

   Explanation for issues of type "ResourceType":
   Ensures that resource id's passed to APIs are of the right type; for
   example, calling Resources.getColor(R.string.name) is wrong.

2 errors, 0 warnings
:lintVitalProductionRelease FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':lintVitalProductionRelease'.
> Lint found fatal errors while assembling a release target.

  To proceed, either fix the issues identified by lint, or modify your build script as follows:
  ...
  android {
      lintOptions {
          checkReleaseBuilds false
          // Or, if you prefer, you can continue to check for errors in release builds,
          // but continue the build even when errors are found:
          abortOnError false
      }
  }
  ...

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 15.452 secs

Does a clean build fix it?

This will be fixed in Butter Knife 8.

:+1: fwiw, clean build does not fix it.

BTW we suppress this in lint.xml internally:
<lint>
  <!-- TODO: Re-enable when butterknife is updated -->
  <issue id="ResourceType" severity="Warning"/>
</lint>

This was suppressed on master for now.


UpgradeToAgs4.6

@TheKeithStewart These are the changes I mentioned.

@JordeyWijnbergen This looks good. The only reason that I haven't merged this yet is because I'm having an issue with my computer that is preventing me from being able to try it out and test it in an application. I should have that worked out shortly and then we should be able to merge in this PR and publish an update to NPM.
Thanks for your hard work on this!!!


Accept stdin as input

This is a great project! I've been using it while writing Nim lately, but I'd like to use it within Helix as my default formatter. Unfortunately, Helix needs the formatter to be able to accept the original program as input over stdin, and emit it via stdout (which nimpretty_t already does).
Is this something that would be able to be looked at?

Hello @polyrain,
Glad to hear it's already helping. I see the problem and gonna look into it. However, the earliest I can make time is next week, please bear with me. And thanks for the report 🙂

@polyrain the latest release should allow you to format input from std. Let me if things work for you 🙂


mapping - update QGIS file to use updated data model

Mostly done but needs to be checked.

Will require PK BT additions but looks ok for now


Fix non 1.11 api ref on k8s.io/docs/reference/

Addresses #9313 with the old api ref urls.
Before: https://kubernetes.io/docs/reference/
After: https://deploy-preview-9343--kubernetes-io-master-staging.netlify.com/docs/reference/

/lgtm
/approve

/assign @tengqm

/lgtm

/approve


Add TensorFlow Datasets data-loading notebook

@alexbw, before merging, let's decide whether to do the reshape in the TF pipeline (which requires import tensorflow as tf), or to keep it in NumPy (which removes the TF import).

Also, tested this only on CPU (i.e. commented out the pip install JAX whl and added jaxlib instead). @mattjj said that typically there's an error when you try to import both jax and tensorflow. Not using the whl may have been why I avoided it but I'm not sure.

Wooo this is awesome!

The stepsize should be 10x smaller, but I'll merge as is and make a small local change.

I prematurely merged because I got so excited, I have now reverted.
Re: your "REVIEWER" comment: I like option #2, where the reshape is explicit but no tf import is required.
Also, I get an error running here (which is how we recommend people run the colab):
https://colab.sandbox.google.com/github/rsepassi/jax/blob/tfds/notebooks/neural_network_with_tfds_data.ipynb
Error:
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "tensorflow/compiler/xla/xla_data.proto":
  tensorflow/compiler/xla/xla_data.proto: A file with this name is already in the pool.


CutMix for Image Captioning

Hi, I really like your paper. But I have a question.
I understand how CutMix is applied to the image classification task, but I'm confused about how do you implement CutMix for Image Captioning task
Given λ that sampled from the uniform distribution (0, 1), how do you merge 2 captions?
It will be great if you could provide an example like :
with λ = 0.2, y1 = "A man wearing a blue jeans standing inside the bus", y2 = "A bird is flying in the sky" will be combined into xxx
Thanks :)

@vinson2233
Hi, thank you for having interest in our work!

Hi, I really like your paper. But I have a question.
I understand how CutMix is applied to the image classification task, but I'm confused about how do you implement CutMix for Image Captioning task
Given λ that sampled from the uniform distribution (0, 1), how do you merge 2 captions?
It will be great if you could provide an example like :
with λ = 0.2, y1 = "A man wearing a blue jeans standing inside the bus", y2 = "A bird is flying in the sky" will be combined into xxx
Thanks :)

Actually, our Image Captioning experiments do not apply CutMix during training time. Only the pretrained model trained with CutMix is used to validate its transferrable ability.
I think your idea to combine two caption labels is quite interesting, and hope to see your further work!

Thanks for the fast response :)
So this section of the paper,

Transferring to MS-COCO image captioning:
We used Neural Image Caption (NIC) [43] as the base model for image captioning experiments. We have changed the backbone
network of encoder from GoogLeNet [43] to ResNet-50.

From my understanding is, NIC workflow is Image -> CNN - > LSTM -> Caption
So the CNN part replaced with your CutMix Pretrained model ?

From my understanding is, NIC workflow is Image -> CNN - > LSTM -> Caption
So the CNN part replaced with your CutMix Pretrained model ?

Yes, thanks.


FAPI Compliance Test - fapi-rw-id2 test failures - CheckIfAuthorizationEndpointError

After the fix for #255, running the tests again resulted in further progress. The access_token could be called OK once the audience in the jwt was set the the value of the access_token endpoint as reported by the as well known.
The next issue that was it was the authorize endpoint, which seems to now not like the aud value in the access token.
When the conformance tests make this call;
https://iam.dev.forgerock.financial:443/am/oauth2/realms/root/realms/alpha/authorize?request=eyJraWQiOiIzMjkxNTg0MjQ5NzkwMzgwMDk2MzM2OTE5Nzc3Mzk0OTM3ODk4MTMiLCJhbGciOiJQUzI1NiJ9.eyJhdWQiOiJodHRwczpcL1wvb2JkZW1vLmRldi5mb3JnZXJvY2suZmluYW5jaWFsOjQ0M1wvYW1cL29hdXRoMlwvcmVhbG1zXC9yb290XC9yZWFsbXNcL2FscGhhIiwic2NvcGUiOiJvcGVuaWQgYWNjb3VudHMiLCJjbGFpbXMiOnsiaWRfdG9rZW4iOnsiYWNyIjp7InZhbHVlcyI6WyJ1cm46b3BlbmJhbmtpbmc6cHNkMjpzY2EiLCJ1cm46b3BlbmJhbmtpbmc6cHNkMjpjYSJdLCJlc3NlbnRpYWwiOnRydWV9LCJvcGVuYmFua2luZ19pbnRlbnRfaWQiOnsidmFsdWUiOiJBQUNfYTZkODJkNjktNTQ3Zi00ZWYzLTk4MmEtYjg5NzhkNWY0OTU4IiwiZXNzZW50aWFsIjp0cnVlfX19LCJpc3MiOiI4OGRmNzVkMS0wYTkxLTQ5MWYtYTk5Ni01MzVhMGQ1YzRkMzgiLCJyZXNwb25zZV90eXBlIjoiY29kZSBpZF90b2tlbiIsInJlZGlyZWN0X3VyaSI6Imh0dHBzOlwvXC93d3cuY2VydGlmaWNhdGlvbi5vcGVuaWQubmV0XC90ZXN0XC9hXC9mb3JnZXJvY2stc2JhdC1kZXZcL2NhbGxiYWNrIiwic3RhdGUiOiJObkhLUUN2WEVUIiwiZXhwIjoxNjUwNjI4Nzg5LCJub25jZSI6IjFsOFkyeGFxcEMiLCJjbGllbnRfaWQiOiI4OGRmNzVkMS0wYTkxLTQ5MWYtYTk5Ni01MzVhMGQ1YzRkMzgifQ.NzReFE0qYDlIWkCrd2MJITg8uKoYDde325VK19tIoYG71DhrvRRS4E1pNSIqEn3MHgdL-9pVD8pKuhgs6G8ckPGoYA1mGxu24Bbr_DW9euddSa3tBEmCnBMq7RHrU3MDu7nSznhUrFmDh15puH8_4NhG4zCYhmtGCwPLIrdJocnYWNyDf5_q5OaFkvvV_76WV7VS8l6duiywHc8J8qy_VEj3zIBIdaJOAJpy6vGW1n4DZc4QkcQutYorl3StmHNstLLZ2GvSLfFxtvKJsLJEs7Ln_Abdq0dd6C2FvyV-8rIxhurZPx0v8GqUt9dMRnUY6l7Wn84421YULvD903_cqw&client_id=88df75d1-0a91-491f-a996-535a0d5c4d38&redirect_uri=https://www.certification.openid.net/test/a/forgerock-sbat-dev/callback&scope=openid accounts&response_type=code id_token
The response is #error_description=Audience%20validation%20failed.&error=invalid_request_object
The jwt contains an audience of;
"aud": "https://obdemo.dev.forgerock.financial:443/am/oauth2/realms/root/realms/alpha"
Clearly the url on which the request was made, and the value in the audience is different.
To fix this, we need to fix the AS well known value to have the correct url so that the conformance suite calls via IG and does not hit am directly. Then we need to ensure that the audience received in the request jwt is also in the AM OAuth2 Providers Additional Audience Values field.
Analysis
This is a really tough one. The FAPI compliance suite uses the issuer value from the well_known as the audience value in the authorize endpoint request jwt. Because of the fix for #255 the well_known endpoint now contains an issuer value that is an IG uri;

"issuer": "https://obdemo.dev.forgerock.financial/am/oauth2/realms/root/realms/alpha"

When this is used in a request to AM using the domain on which AM is operating then AM expect the audience to be https://iam.dev.forgerock.financial/am/oauth2/realms/root/realms/alpha
The OIDC connect specification says that the Authorization server should accept any value containing the issuer value. i.e. AM should accept the authorization_endpoint value from our well_known, which is;

"authorization_endpoint": "https://iam.dev.forgerock.financial/am/oauth2/realms/root/realms/alpha/authorize"

However, AM doesn't accept that (this is a bug in AM?), and anyway the FAPI compliance suite doesn't use the authrorization_endpoint value, it uses the issuer value, which according to the OIDC spec should be accepted :-(

The jwt contains an audience of "aud": "https://obdemo.dev.forgerock.financial:443/am/oauth2/realms/root/realms/alpha",

Am now taking a different approach. Rather than adding acceptable audience for the access_token config, and then struggling with the authorize endpoint I am trying to set the Base URI for AM, which results in all of the endpoints in the well know being obdemo rather than iam.
I now think this avenue of investigation was a dead end.


Msal node token cache

Coverage remained the same at 80.98% when pulling 547d91360b7ebc00279823a493eb7dcd23bac12c on msal-node-token-cache into<PHONE_NUMBER>9ed70219f04569bc9afaa2ded69294 on dev.


A way to export favorites?

Thanks for fun app, I enjoy browsing through quotes and this is app is apt for it 😄
I was wondering if there was a way to export favorites?

 Export favourites?

I didn't know the favorites sync between devices, but yeah that is not what I mean.
I meant something as basic like a text file export - all the favorites are pooled in to a text file and exported on device storage. This would enable me to review my favorites on my computer and copy the ones I really like elsewhere.

Hi Dinesh - Thanks for explanation. I'll start coding this feature. Don't hold your breath though: it takes a while for me to code new features!
You can track "progress" on: https://github.com/jameshnsears/QuoteUnquote/projects/1 & in any notifications on this issue.

App now has an Export feature, to local / remote storage. Software released; but might take a few days to appear in Google Play.
If you have any other suggestions / spot problems please raise another GitHub issue.


Practice App: Creating Serializers

What's up?
We have designed our ER Diagrams as documented in #94. Models are designed in #96. One way of returning our models in JSON format is to implement Serializers. Please keep in mind that scope of this work is limited, and aimed to give you a starting point. You can add new serializers in your branches if necessary.
To Do
Following actions must be taken:

[x] Create serializers:

[x] Create user serializer.
[x] Create comment serializer.
[x] Create tag serializer.
[x] Create art item serializer.
[x] Create follow serializer.

Deadline
Deadline: 14.05.2022 @22.00
Additional Information
Contributors:

@BElifb
@KarahanS

Reviewers
No response

In #93, serializers for ArtItem and Tag are created by @BElifb. Then I've added serializers for Comment and myUser before merging.
In #99, modified existing serializers to reflect the changes in the model, added Follow and SimpleUser serializers.

Updated the ArtItemSerializer to return more information about its tags by this commit. It used to return only the ID of the tag, now it returns all the fields including the name of the tag.

Updated the ArtItemSerializer to return more information aobut the owner by this commit. It used to return only the ID of the owner, not it returns username, name and surname as well.

Serilizers of our practice app seems to have all the functionality we needed to accomplish our endpoint tasks. one can open an issue if it is needed. It is finalized for milestone 2 report for now.


remove old code that assumes tag were created before building

remove old code that assumes tag were created before building. Old code picks up the TAG as version for istioctl, which is not what we want. We are passing it in explicitly.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: rkpagadala
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: douglas-reid
If they are not already assigned, you can assign the PR to them by writing /assign @douglas-reid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

bin/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hklai, rkpagadala
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

bin/OWNERS [hklai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment