[AutoRelease] t2-communication-2023-11-15-90986(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/4743
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=2047
BuildTargetingString
azure-mgmt-communication
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/4743


readbuffersize error, when proxy h265 rtsp stream

Which version are you using?
v1.8.0
Which operating system are you using?
linux

[ ] Linux amd64 standard
[ ] Linux amd64 Docker
[ ] Linux arm64 standard
[ ] Linux arm64 Docker
[ ] Linux arm7 standard
[ ] Linux arm7 Docker
[ ] Linux arm6 standard
[ ] Linux arm6 Docker
[ ] Windows amd64 standard
[ ] Windows amd64 Docker (WSL backend)
[ ] macOS amd64 standard
[ ] macOS amd64 Docker
[ ] Other (please describe)

Describe the issue
Description
Describe how to replicate the issue

readbuffersize error, when proxy h265 rtsp stream

start the server
publish with ...
read with ...

Did you attach the server logs?

yes / no
Did you attach a network dump?

yes / no

Please reopen the issue and attach either a server log or a network dump as stated in the instructions.


how to select feature in WebGLVectorTileLayer

I created a layer as this:
      class WebGLVectorTileLayer extends VectorTileLayer {
        createRenderer() {
          return new WebGLVectorTileLayerRenderer(this, {
            style: {
              fill: {
                fragment: result.builder.getFillFragmentShader(),
                vertex: result.builder.getFillVertexShader(),
              },
              stroke: {
                fragment: result.builder.getStrokeFragmentShader(),
                vertex: result.builder.getStrokeVertexShader(),
              },
              symbol: {
                fragment: result.builder.getSymbolFragmentShader(),
                vertex: result.builder.getSymbolVertexShader(),
              },
              attributes: {
                fillColor: {
                  size: 2,
                  callback: (feature) => {
                    return packColor([21, 255, 18, 1])
                  },
                },
                strokeColor: {
                  size: 2,
                  callback: (feature) => {
                    return packColor([255, 255, 255, 1])
                  },
                },
                strokeWidth: {
                  size: 1,
                  callback: (feature) => {
                    return 1
                  },
                },
              },
            },
          })
        }
      }
      let layer = new WebGLVectorTileLayer({
        source: new VectorTileSource({
          attributions: [],
          format: new MVT(),
          url: API_URL + 'mapManage/GetRasterMVT?z={z}&x={x}&y={y}',
          minZoom: MIN_ZOOM,
          maxZoom: MAX_ZOOM,
        }),
      })

now, how to select the feature in this layer?
please
thank you

The Vector Tile layer example https://openlayers.org/en/latest/examples/vector-tile-selection.html uses feature id, so a fix for #15251 would be needed before something similar can be attempted in WebGL.

It seems this is still not possible.  Although #15120 also updated WebGLVectorTileLayerRenderer to enable the VectorStyleRenderer to hit detect by default map.forEachFeatureAtPixel returns no hits and layer.getFeatures is not supported.

Hit detection on vector tile layers rendered using WebGL is currently being worked by @gberaudo on and a PR is likely to come in the near future.

We look forward to hearing from you soon. @jahow

So, my case is:

useGeographic()
WebGL layer with feature source and preloaded features in EPSG:4326
Default Select tool working with WebGL layer..

It seems that internally some "math" is not yet correctly working.

The value of "ref"  variable has nothing related to real "ref" as keys in batch_  data stucture of WebGL renderer... (looks like it should be 1,2,3... N
Source code is a bit of magic at the moment for me in WebGL renderer, so just inconsistency of calculated "ref" value is obvious, may be it helps.

@vitalus thanks for reporting, this problem has most likely been fixed by https://github.com/openlayers/openlayers/pull/15340; unfortunately, this hasn't made it to 8.2.0. Could you please let us know if the issue persists when using the ol@dev latest package? Thanks!

in version 9.0.0-dev.1701358929517  selection is working normally, seems.

It seems this is still not possible. Although #15120 also updated WebGLVectorTileLayerRenderer to enable the VectorStyleRenderer to hit detect by default map.forEachFeatureAtPixel returns no hits and layer.getFeatures is not supported.

Feature selection remains unavailable for WebGLVectorTileLayerRenderer, correct? It seems that the behavior described above is still the case (ol 9.2.4).

Hi, by any chance is there any development related to selecting features from WebGLVectorTileLayer?


Automated cherry pick of #2451: bugfix: 避免openstack磁盘扩容成功后未能挂载到虚拟机上

Cherry pick of #2451 on release/2.9.0.
#2451: bugfix: 避免openstack磁盘扩容成功后未能挂载到虚拟机上

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.9.0@95a22c9). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             release/2.9.0   #2481   +/-   ##
===============================================
  Coverage                 ?   7.69%
===============================================
  Files                    ?     336
  Lines                    ?   54019
  Branches                 ?       0
===============================================
  Hits                     ?    4157
  Misses                   ?   49475
  Partials                 ?     387

Flag
Coverage Δ

#aFlag
7.69% <ø> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95a22c9...b3fa0b2. Read the comment docs.

/approve
/lgtm


Unable to create bucket using scripts/setup

Unfortunately minio/minio doesn't have the mc CLI installed on it making this line from the setup file fail https://github.com/inveniosoftware/cookiecutter-invenio-rdm/blob/master/scripts/setup#L30
I don't have the perfect solution, but this worked for me.
I have created my own image for the S3 service with the following docker/minio/Dockerfile
FROM minio/minio

RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc -P /usr/bin \
    && chmod +x /usr/bin/mc

COPY ./create-bucket.sh .

The content of docker/minio/create-bucket.sh
#!/usr/bin/env sh

# Configure minio client to point to localhost

mc config host add local http://localhost:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}

# Delete the bucket in case it exists, start from scratch
if mc ls local | grep -q default-bucket; then
    mc rb local/default-bucket
fi

# Create the bucket
mc mb local/default-bucket

(NOTE: there is no bash, just sh)
the rest should remain the same besides the call inside scripts/setup
docker-compose exec s3 sh -c './create-bucket.sh'

Other solutions, that I can think of, range from adding a new "service" to the docker-service.yml file pulling from the minio/mc image;
mc:
    image: minio/mc
    link:
      - s3
    environment:
      MINIO_ACCESS_KEY: CHANGE_ME
      MINIO_SECRET_KEY: CHANGE_M

to tell the user to install and configure the mc locally, documentation
I'd love to hear your thoughts on this, and if you think what I did is "good enough", I'm happy to make a PR anytime 😇

Hello @egabancho Thanks for reporting!
Since we are using minio just for "demo" purposes. Meaning, it's not meant for production and it is just there to allow the user to test with S3 I would personally go with building a new docker image with mc installed in it. We already do something similar with Nginx.
@fenekku @zzacharo what are your thoughts?

I think that mc would be quite useful to be included inside a docker image for people to be able to play around with the S3 installation!

+1 on having an image with mc ready to go.

Hey guys, sorry for not doing anything here for a while. The other I realized that if we simply create a folder and point the min.io server to it it'll create a bucket automagically.
For example, min.io is started with a volume data:/data, if we simply create a local folder data/default-bucket it'll pick it up and we don't need to do anything special.
WDYT?

@egabancho sounds good to me, minio is just for "demo" purposes in local... so I would say yes! thanks for checking!


[Loc] update to data-workspace xlf

This PR fixes #

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.01%) to 47.969%

Files with Coverage Reduction
New Missed Lines
%

extensions/notebook/src/book/bookTreeView.ts
2
38.21%

Totals

Change from base Build<PHONE_NUMBER>:
-0.01%

Covered Lines:
15005

Relevant Lines:
29653

💛  - Coveralls


.netstandard support

This PR has too many file changes, and .NET Standard 2.0 support was introduced with 3.2.9.


wip: websocket client support

A wip implementation
Closes #650

Codecov Report

Merging #794 (1eec57b) into main (2d26f13) will decrease coverage by 0.95%.
The diff coverage is 18.51%.

@@            Coverage Diff             @@
##             main     #794      +/-   ##
==========================================
- Coverage   51.75%   50.79%   -0.96%
==========================================
  Files          68       69       +1
  Lines        2085     2128      +43
  Branches       63       63
==========================================
+ Hits         1079     1081       +2
- Misses       1006     1047      +41

Impacted Files
Coverage Δ

zio-http/src/main/scala/zhttp/http/Scheme.scala
50.00% <0.00%> (-10.00%)
:arrow_down:

...ala/zhttp/service/client/ClientSocketHandler.scala
0.00% <0.00%> (ø)

zio-http/src/main/scala/zhttp/service/Client.scala
43.75% <11.11%> (-13.40%)
:arrow_down:

zio-http/src/main/scala/zhttp/http/URL.scala
97.56% <50.00%> (ø)

...http/service/client/ClientChannelInitializer.scala
66.66% <63.63%> (-19.05%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d26f13...1eec57b. Read the comment docs.

Closing in favour of #933


Date race analysis

The entire package needs to be tested with -race.  The go race detector did not exist when this package was first written and published.

  Needs analysis.
$ STOMP_PORT=61613 STOMP_TEST11p=1.1 STOMP_HB11LONG=y STOMP_AMQ11=y go test -v -race -timeout 10m30s -run TestHB11NoSend
=== RUN TestHB11NoSend
2015/04/23 21:22:14.516358 ID:tjjackson-39054-1429838424047-2:1[TestHB11NoSend start sleep]
2015/04/23 21:22:14.516729 ID:tjjackson-39054-1429838424047-2:1[1 Send 0 Receive 6000]
==================
WARNING: DATA RACE
Read by goroutine 9:
  github.com/gmallard/stompngo.(*Connection).receiveTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:168 +0x305

Previous write by goroutine 10:
  github.com/gmallard/stompngo.(*Connection).readFrame()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/reader.go:97 +0x289
  github.com/gmallard/stompngo.(*Connection).reader()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/reader.go:36 +0x5a

Goroutine 9 (running) created at:
  github.com/gmallard/stompngo.(*Connection).initializeHeartBeats()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:109 +0xfb3
  github.com/gmallard/stompngo.(*Connection).connectHandler()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:148 +0x7af
  github.com/gmallard/stompngo.Connect()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:104 +0xc55
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:201 +0x588
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 10 (running) created at:
  github.com/gmallard/stompngo.Connect()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:111 +0xd82
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:201 +0x588
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc
==================
==================
WARNING: DATA RACE
Read by goroutine 9:
  github.com/gmallard/stompngo.(*Connection).log()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connection.go:179 +0x4b
  github.com/gmallard/stompngo.(*Connection).receiveTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:170 +0x823

Previous write by goroutine 7:
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:214 +0x960
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 9 (running) created at:
  github.com/gmallard/stompngo.(*Connection).initializeHeartBeats()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:109 +0xfb3
  github.com/gmallard/stompngo.(*Connection).connectHandler()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:148 +0x7af
  github.com/gmallard/stompngo.Connect()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:104 +0xc55
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:201 +0x588
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 7 (running) created at:
  testing.RunTests()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:560 +0xc96
  testing.(*M).Run()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:490 +0xed
  main.main()
      /tmp/go-build301441451/github.com/gmallard/stompngo/_test/_testmain.go:206 +0x28c
==================
==================
WARNING: DATA RACE
Write by goroutine 9:
  sync/atomic.CompareAndSwapInt32()
      /ad3/gma/gorepo/github/go/src/runtime/race_amd64.s:279 +0xb
  sync.(*Mutex).Lock()
      /ad3/gma/gorepo/github/go/src/sync/mutex.go:43 +0x50
  log.(*Logger).Output()
      /ad3/gma/gorepo/github/go/src/log/log.go:147 +0x9e
  log.(*Logger).Print()
      /ad3/gma/gorepo/github/go/src/log/log.go:178 +0x7e
  github.com/gmallard/stompngo.(*Connection).log()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connection.go:180 +0x1f6
  github.com/gmallard/stompngo.(*Connection).receiveTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:170 +0x823

Previous write by goroutine 7:
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:213 +0x85d
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 9 (running) created at:
  github.com/gmallard/stompngo.(*Connection).initializeHeartBeats()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:109 +0xfb3
  github.com/gmallard/stompngo.(*Connection).connectHandler()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:148 +0x7af
  github.com/gmallard/stompngo.Connect()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:104 +0xc55
  github.com/gmallard/stompngo.TestHB11NoSend()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:201 +0x588
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 7 (running) created at:
  testing.RunTests()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:560 +0xc96
  testing.(*M).Run()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:490 +0xed
  main.main()
      /tmp/go-build301441451/github.com/gmallard/stompngo/_test/_testmain.go:206 +0x28c
==================
2015/04/23 21:22:20.518452 ID:tjjackson-39054-1429838424047-2:1[HeartBeat Receive TIC TickerVal<PHONE_NUMBER>517013000 LastReceive<PHONE_NUMBER>516823072 Diff<PHONE_NUMBER>]
2015/04/23 21:22:26.517816 ID:tjjackson-39054-1429838424047-2:1[HeartBeat Receive TIC TickerVal<PHONE_NUMBER>517431895 LastReceive<PHONE_NUMBER>502080848 Diff 15351047]

Races when only receiving heartbeats.  Needs analysis.
$ STOMP_PORT=61613 STOMP_TEST11p=1.1 STOMP_HB11LONG=y STOMP_AMQ11=y go test -v -race -timeout 10m30s -run TestHB11NoReceive
=== RUN TestHB11NoReceive
2015/04/23 21:25:58.841925 ID:tjjackson-55840-1429838707424-2:1[TestHB11NoReceive start sleep]
2015/04/23 21:25:58.842801 ID:tjjackson-55840-1429838707424-2:1[2 Send 10000 Receive 0]
==================
WARNING: DATA RACE
Read by goroutine 11:
  github.com/gmallard/stompngo.(*Connection).log()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connection.go:179 +0x4b
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:127 +0x13d

Previous write by goroutine 7:
  github.com/gmallard/stompngo.TestHB11NoReceive()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:258 +0x960
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 11 (running) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d

Goroutine 7 (running) created at:
  testing.RunTests()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:560 +0xc96
  testing.(*M).Run()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:490 +0xed
  main.main()
      /tmp/go-build941597814/github.com/gmallard/stompngo/_test/_testmain.go:206 +0x28c
==================
==================
WARNING: DATA RACE
Write by goroutine 11:
  sync/atomic.CompareAndSwapInt32()
      /ad3/gma/gorepo/github/go/src/runtime/race_amd64.s:279 +0xb
  sync.(*Mutex).Lock()
      /ad3/gma/gorepo/github/go/src/sync/mutex.go:43 +0x50
  log.(*Logger).Output()
      /ad3/gma/gorepo/github/go/src/log/log.go:147 +0x9e
  log.(*Logger).Print()
      /ad3/gma/gorepo/github/go/src/log/log.go:178 +0x7e
  github.com/gmallard/stompngo.(*Connection).log()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connection.go:180 +0x1f6
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:127 +0x13d

Previous write by goroutine 7:
  github.com/gmallard/stompngo.TestHB11NoReceive()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:257 +0x85d
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc

Goroutine 11 (running) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d

Goroutine 7 (running) created at:
  testing.RunTests()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:560 +0xc96
  testing.(*M).Run()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:490 +0xed
  main.main()
      /tmp/go-build941597814/github.com/gmallard/stompngo/_test/_testmain.go:206 +0x28c
==================
2015/04/23 21:26:08.844634 ID:tjjackson-55840-1429838707424-2:1[HeartBeat Send data]
==================
WARNING: DATA RACE
Read by goroutine 11:
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:140 +0x417

Previous write by goroutine 9:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:142 +0x241

Goroutine 11 (running) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d

Goroutine 9 (running) created at:
  github.com/gmallard/stompngo.(*Connection).initializeHeartBeats()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:100 +0xe85
  github.com/gmallard/stompngo.(*Connection).connectHandler()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:148 +0x7af
  github.com/gmallard/stompngo.Connect()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/connect.go:104 +0xc55
  github.com/gmallard/stompngo.TestHB11NoReceive()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/hb_test.go:245 +0x588
  testing.tRunner()
      /ad3/gma/gorepo/github/go/src/testing/testing.go:452 +0xfc
==================
2015/04/23 21:26:18.844310 ID:tjjackson-55840-1429838707424-2:1[HeartBeat Send data]
==================
WARNING: DATA RACE
Write by goroutine 12:
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:137 +0x470

Previous write by goroutine 11:
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:137 +0x470

Goroutine 12 (running) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d

Goroutine 11 (finished) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d
==================
==================
WARNING: DATA RACE
Read by goroutine 12:
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:138 +0x4b4

Previous write by goroutine 11:
  github.com/gmallard/stompngo.(*Connection).sendTicker.func1()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:138 +0x4d0

Goroutine 12 (running) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d

Goroutine 11 (finished) created at:
  github.com/gmallard/stompngo.(*Connection).sendTicker()
      /ad3/gma/gosrc/src/github.com/gmallard/stompngo/heartbeats.go:141 +0x21d
==================

Some number of commits relating to data races have recently been merged into master branch.
Please update to at at least HEAD of current master: a488ae2
All current race tests are clean:
=============== AMQ 1.0 ===============
STOMP_PORT=61613 go test -v -race
...
PASS
ok  	github.com/gmallard/stompngo	3.313s
=============== AMQ 1.1 ===============
STOMP_PORT=61613 STOMP_TEST11p=1.1 STOMP_HB11LONG=y STOMP_AMQ11=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	483.231s
=============== AMQ 1.2 ===============
STOMP_PORT=61613 STOMP_TEST11p=1.2 STOMP_HB11LONG=y STOMP_AMQ11=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	483.160s
=============== Apollo 1.0 ===============
STOMP_PORT=62613 STOMP_APOLLO=y go test -v -race
...
PASS
ok  	github.com/gmallard/stompngo	3.480s
=============== Apollo 1.1 ===============
STOMP_PORT=62613 STOMP_TEST11p=y STOMP_HB11LONG=y STOMP_APOLLO=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	603.148s
=============== Apollo 1.2 ===============
STOMP_PORT=62613 STOMP_TEST11p=1.2 STOMP_HB11LONG=y STOMP_APOLLO=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	602.967s
=============== RMQ 1.0 ===============
STOMP_RMQ=y STOMP_PORT=41613 go test -v -race
...
PASS
ok  	github.com/gmallard/stompngo	2.615s
=============== RMQ 1.1 ===============
STOMP_TEST11p=1.1 STOMP_RMQ=y STOMP_PORT=41613 STOMP_HB11LONG=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	603.019s
=============== RMQ 1.2 ===============
STOMP_TEST11p=1.2 STOMP_RMQ=y STOMP_PORT=41613 STOMP_HB11LONG=y go test -v -race -timeout 10m30s
...
PASS
ok  	github.com/gmallard/stompngo	603.085s


Change SourceLocation to be serializable.

Needed for the DTH so we can send SourceLocations over the wire.

Gonna leave this open until the DTH changes are in.

Getting this in so Todd can start integrating tooling.


Added code to open external links in a new tab

Links that point to an external website now open in a new tab
 Mention [stepsize] in a comment if you'd like to report some technical debt. See examples here.

I've removed the https links from the 'reserved attributes'  page.


MinTrace's protection to Schafer-Strimmer covariance, eliminated stat…

Solved eigenvalue indetermination in the presence of small percentage of zero predictions.

Added protections for single series with constant values / std=0.
Added protections to standardize residuals when std=0.
Final epsilon diagonal for improved numerical stability.
Warning + unit test for percentage of zero residuals scenarios.

I took out the statsmodels, the only function we used from it was cov2corr that is particularly simple and does not merit such a big dependency.

Also, if this PR is going close #92, please let's add a test where it is verified that before the changes, the same error arises, and now with these changes, it is solved.
See, for example, this PR.
A user had an error using HierarchicalEvaluation when h=1, then a test was included showing that it works. In particular, these lines,

This particular case is a little bit tricky because we don't have the actual data. I'd start trying to reproduce the error with a small example. Just to be sure that these changes are actually solving the problem.

Just some thoughts about the test:
An element of the diagonal of the variance-covariance matrix would be zero if a time series has only one value, so to reproduce the error, we could use the objects used for testing purposes in nbs/methods.ipynb as follows,
diff_len_y_insample = S @ y_bottom
diff_len_y_hat_insample = S @ y_hat_bottom_insample
diff_len_y_insample[-1, :-1] = np.nan
diff_len_y_hat_insample[-1, :-1] = np.nan
print(diff_len_y_insample)
cls_min_trace = MinTrace(method='mint_shrink')
cls_min_trace(
    S=S,
    y_hat=S @ y_hat_bottom,
    y_insample=diff_len_y_insample,
    y_hat_insample=diff_len_y_hat_insample,
    idx_bottom=idx_bottom
)

The output of the previous code is,

This small example recovers the #92 bug. Observe that if we use ols, there is no problem (as expected),

The user's data has time series with a single value.


Wrong loading of weights on nested models

Loading of nested models including BatchNormalisation and 1D convolutions fail with the following traceback:
Traceback (most recent call last):
  File "/home/david/research/keras_bug/save_resave.py", line 45, in <module>
    model = load_model('a{}.h5'.format(i))
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 263, in load_model
    load_weights_from_hdf5_group(f['model_weights'], model.layers)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 915, in load_weights_from_hdf5_group
    reshape=reshape)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 556, in preprocess_weights_for_loading
    weights = convert_nested_model(weights)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 544, in convert_nested_model
    original_backend=original_backend))
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 556, in preprocess_weights_for_loading
    weights = convert_nested_model(weights)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 532, in convert_nested_model
    original_backend=original_backend))
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/saving.py", line 674, in preprocess_weights_for_loading
    weights[0] = np.transpose(weights[0], (3, 2, 0, 1))
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 575, in transpose
    return _wrapfunc(a, 'transpose', axes)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 52, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: axes don't match array

I have been able to trim it down to a minimum example:
https://gist.github.com/Dapid/cd1be6e9d9d8d61f7f225544f0967e35
Inspecting the shapes, it seems like it is trying to load the bias vector of a conv1d on both the bias and kernel, and of course the shapes don't match. I have been unable to trace it back to the original place.
The model was trained on Tensorflow.
Possibly related:
https://github.com/keras-team/keras/issues/10777
Also, inspecting the shapes of the loading weights, it is possible that some batch normalisation is not being loaded, which would explain https://github.com/keras-team/keras/issues/10780 among others.

Dose this problem solved? I have the same error. I save a model which contains several nested model, and when load the model, comes the error: axes don't match array.

Sergey Ovchinnikov added on Slack:

I think its related to the issue we previously talked about in the google-mailing-list (I believe someone suggested a bug fix). it comes down to trainable weights and untrainable weights of BatchNorm and in the order in which they are saved/loaded. Bug in the get_weights() function. For each layer, the function returns first the trainable and then the untrainable weights (if you make one of the models a "layer", regardless of the layer order of that model, first the trainable, followed by untrainable weights are returned).

Curious about the status of this issue- also have been running into the same problem as above.
Does there exist any workaround outside of not using batchnorm layers within nested models at all?

Geez, if after trained the model weights can't be loaded back, then what's the point of training the model??


CI Production Workflows for Docker and Docs

Closes #807
Code Changes

[x] Add a conditional step to check if the tag used matches a release
[x] Include the current image when pushing a production image
[x] Parse the  github.event.ref to publish the version specific docs
[x] Mimic the conditional step for docs publishing

Steps to Confirm

[x] Set up a testing repo to ensure the workflow worked
[x] Add a test nox command to print out the update IMAGE constant with version

Pre-Merge Checklist

[x] All CI Pipelines Succeeded
Documentation Updated:

[x] documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
[x] documentation issue created (tag docs-team to complete issue separately)

[x] Issue Requirements are Met
[x] Relevant Follow-Up Issues Created
[x] Update CHANGELOG.md

Description Of Changes
Spent some time in a test repo to ensure the workflow updates should work for us.
I'm unsure still if the nox changes are sufficient, or if there should be a separate constant used for the current working version.
Another additional question is if we want to retroactively publish docs versions that were missed 🤔

@SteveDMurphy I don't feel a need to publish old docs versions, I think it'll be good enough to have going forward but not worth back-filling


'Before' Image offset doesn't show at startup

Hi Folks
I got an issue I just experienced on the website I'm working on. I'm using Twentytwenty with an offset set at 0.2. When the twentytwenty container is shown, I can only see the second image ('After' in default configuration, 'Before' on mine but whatever) while the handle is correctly set à 20% of the width. (Screenshot)
Whenever I click on or drag the handle, everything goes back to normal and the plugin works.
Any ideas ?
Thanks

My initial thoughts are maybe there's an issue with how TwentyTwenty is being initialized, can you confirm it's on the $(window).load() event?
Also posting a link to your code would be very helpful, thanks @Acoleno!

Hey @mhayes! Thanks for your time
Bascially, it was into $(document).ready() event so I did move it into $(window).load() but as you can guess, it didn't solve the problem.
I'm working on an automatically generated wall gallery .
As you can deduce from the screenshot, each before/after image is wrapped into a fancybox.
Here's the HTML part:

	<div class="fancybox-wrapper-target"  id="matte-painting-<?php the_ID();?>">
		<div class="fancybox-matte-painting twentytwenty" style="width:<?php echo  $after_image['width'].'px';?>; min-height: <?php echo $after_image['height'].'px';?>">
			<img src="<?php echo $after_image['url'];?>" width="<?php echo $after_image['width'];?>" height="<?php echo $after_image['height'];?>"/>
			<img src="<?php echo $before_image['url'];?>" width="<?php echo $before_image['width'];?>" height="<?php echo $before_image['height'];?>"/>

And the related js:
$(window).load(function (){
		$("a.fancybox-matte-painting-link").each(function (){
	    	var target = $('div#matte-painting-'+ $(this).attr('data-fancybox-href'));
			$(this).fancybox({
	        'href'   : '#'+target.attr('id'),
	   		});
		});
		$('div.fancybox-matte-painting').each(function() {
			$(this).twentytwenty();
		});
	});

There's no specific css except some minor width & height ajustements.
Ask me if you need more information!

I thought that it might have something to do with manually setting the image dimensions on the Fancybox container but that doesn't seem to be the case. I put together an example that showcases Fancybox and TwentyTwenty working together. Here's the example page:
http://mhayes.github.io/example-twentytwenty-fancybox/
The corresponding code I used was not radically different from what you were using. It can be found here:
https://github.com/mhayes/example-twentytwenty-fancybox/blob/master/index.html
The biggest different could be that you can have to trigger a window resize event when the Fancybox opens as that will adjust dimensions on the slider. Take a peek at the atterLoad event in the code I mentioned above.
If you're still running into issues it would be useful to see the full site you're working on. If you posted a link to your development/staging environment that would be helpful.
Hope that helps you out, cheers!

I just integrated the afterload event, but it didn't made the trick.
I'm glad to see I didn't go into weird direction when it came to combine fancybox & twentytwenty functionnality. The issue shouldn't be so far.
I made the test with both jquery versions 1.12 and 2.2 and tried to deactivate other jquery plugins I installed. The issue is still there.
You can check this out on the staging environment (portfolio -> matte Painting)
The call is located at the bottom of funcion.js file.
Still really grateful for your time.

Hey @Acoleno I took a look at the functions.js file and it looks like if you add a setTimeout function on the after load event it will work as expected. It feels like there should be a more elegant solution than this, but it does appear to work in my browser. Here's the relevant code to try:
$("a.fancybox-matte-painting-link").each(function (){
  var target = $('div#matte-painting-'+ $(this).attr('data-fancybox-href'));
  $(this).fancybox({
      'href'   : '#'+target.attr('id'),
      afterLoad: function () {
        // Adding the the timeout
        setTimeout(function() {
            $(window).trigger("resize");
        }, 100);
      }
    });
});

I also noticed that the functions.js is being included before the other plugins (jquery.event.move.js and jquery.twentytwenty.js). It doesn't appear to be causing any issues, but I believe the order should be:

Hey @mhayes.
This totally made the trick !
I'll take a look at your other advices.
We can call this one solved. Thanks again.

You're welcome, glad it worked @Acoleno :)


cannot upload file

since friday i cannot upload any file with the script, i've never fail before, maybe it's because of the selenium, may you have the fix of this problem?

Hi @sakarimov , can you explain me a bit more the problem.
When does it fail?
Which social network?
Does the image never upload?
Firefox and os version?
Did you update your browser recently?
Thanks!

When does it fail?
since friday 02 of October
Which social network?
Instagram
Does the image never upload?
yes, when the step goes to upload file it stucks and keep loading without uploading, then goes to caption typing step, whereas previously, after the "upload file" step, the file will be uploaded immediately and the program will go to the caption typing step

image not uploaded but the typing goes on
Firefox and os version?
Firefox 81.0, Ubuntu 20.04
Did you update your browser recently?
i dont think so, maybe not

Thanks for your reply.
I know about this problem.

Which social network?
Instagram
Does the image never upload?
yes, when the step goes to upload file it stucks and keep loading without uploading

And a message of the problem is here:

JavaScript error: https://business.facebook.com/creatorstudio/?reference=visit_from_seo, line 46: TypeError: 'get width' called on an object that does not implement interface HTMLImageElement.

That is from your log file, you can see a similar output in the console of your browser.
I will try to explain a bit the issue.
I added an extension that implements different strategies to hide selenium. I took the extension from Instapy
One of that strategies is the Broken Image test.
So sometimes this code is executed before the upload finish and you get the error because the image object was not set properly.
A quick solution is to comment from Line 33 until Line 49.
I will be thinking how to solve it. Disable it, it's an option, for this project I think is not necessary. At the moment Facebook don't implement a test like that one.

it works, thank u so much, it helps, I've been dizzy for a few days, and can't find an answer, hope u the best @asperduti


⚠ Bofrost.at has degraded performance

In 0a3cf2f, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2624 ms

Resolved: Bofrost.at performance has improved in 32fcffa.


Is it possible to have more than one brand

I'm trying to implement some decent number checking and types:
import * as rt from 'runtypes';

export type RealNumber = rt.Static<typeof RealNumber>
export const RealNumber = rt.Number.withConstraint(n => !isNaN(n - n)).withBrand('number.real');

export type Integer = rt.Static<typeof Integer>
export const Integer = RealNumber.withConstraint(n => n % 1 === 0).withBrand('number.integer');

Th runtypes themselves look good, but the IntegerType is currently being resolved as never.
My assumption is that this is because there is a phantom brand field or something and that field cannot be two different strings.
If this is the case it would appear that what i am trying to do would not be possible.
I feel that what I am attempting to do here is a valid use case.  When messing around on my own with this sort of things before I have done something like this for managing multi-branding.
type Brand<T extends string> = {
 __brand: {
    [k in T]: T
    }
}

type Real = number & Brand<'number.real'>
type Integer = Real & Brand<'number.integer'>

const integer:Integer = -11 as Integer;
const real:Real = 1 as Real
const integersAreReal: Real = integer;

I would love to see something like this implemented.
If there is some way to do this already, if I am doing something wrong, or if you think for some reason this is a bad idea, please let me know.

Thanks, this should be definitely a thing. It wouldn't be hard to implement without breaking change, so I'm going to give it a shot this weekend.

@yuhr - I think I've beaten you to it.
https://github.com/pelotom/runtypes/pull/329
I messed around a little bit changing the reflection but it turned out to be unnecessary.


Question: What happens when you upgrade multiple times

If multiple requests are attempting to upgrade a single, specific, service multiple times, what would happen?
Sorry if this is in the documentation somewhere, but I could not find it?
My use case:
I'd like to dodge and race conditions if someone asked to deploy something N number of times, that hopefully it would just override and restart the previous deployment.  If not, curious how to handle this?

From what I understand, a helm has state of the current install and is idempotent. So doing two upgrades of a single specific service most likely will do nothing the second time. This is assuming you are not using extra command line flags to the upgrade command.

Thanks!


Additions

Hey,
thanks again for publishing this.
Maybe you can reserve a port for the exporter and get yourself added to the official list:

https://prometheus.io/docs/instrumenting/writing_exporters/#port-numbers
https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exporters.md

LBNL, I am also looking if there is a way to assign labels from the namespace to the pod metrics. Would you take this as a contribution? Or is this too specific?
Thanks again,
Till

Great suggestion about the port assignment. I didn't even know that registry existed.
I like the sound of your enhancement idea and would encourage a PR. From that I can better feel if it's too specific or not.

...oh, the dilemma of allocating a port is that the default is already 8080 (which is of course heavily over-used). As a user of this exporter, would you mind adjusting your prom config to a different port?

...oh, the dilemma of allocating a port is that the default is already 8080 (which is of course heavily over-used). As a user of this exporter, would you mind adjusting your prom config to a different port?

No, not at all. Just depends on how you interpret BC. No idea who uses this exporter, but it may require config changes for others.

On the additional labels: I think I gave up on the idea for now, since cadvisor would be more close to what I was trying to achieve.

Thanks for following up.


Allow loading migrations from a jar file

We are using Pillar inside a Play2 application where it is deployed as a jar. One can not read migrations from a directory inside a jar. We worked around this with something like the following. I am not creating a pull request as I am not sure on how to integrate this and if it's desired behaviour nor do I know how to correctly test this. sorry.
  private val registry = Registry(loadMigrationsFromJarOrFilesystem())
  private val migrator = Migrator(registry, new LoggerReporter)

  private def loadMigrationsFromJarOrFilesystem() = {
    val migrationsDir = "migrations/"
    val migrationNames = JarUtils.getResourceListing(getClass, migrationsDir).toList.filter(_.nonEmpty)
    val parser = Parser()

    migrationNames.map(name => getClass.getClassLoader.getResourceAsStream(migrationsDir + name)).map {
      stream =>
        try {
          parser.parse(stream)
        } finally {
          stream.close()
        }
    }.toList
  }

where JarUtils.getResourceListing is taken from the top answer here: http://stackoverflow.com/questions/6247144/how-to-load-a-folder-from-a-jar
Hope that helps

I agree it would be nice to be able to use migrations from resources on the classpath.
This alternative that functions as a library allows you to deploy your usual artifact instead of needing to somehow also deploy source (or a standalone repo of conf/) + the pillar executable into a server environment where it can reach the production Cassandra cluster. Support for this would pave the way for more easily using Pillar as a library for the same sort of use case when required.


v1: backport sort commands by name

this is backporting the commit from https://github.com/urfave/cli/issues/575
command: enable ordering commands by name
(cherry picked from commit ac772237b96509d5150567da31ba6c6c3897d452)

This was actually missing because I hadn't recently merged master into the v1 branch, but have done so now. Thank you for pointing this out @MaximilianMeister !

@jszwedko thanks ! do you plan to make a new release tag as well? and make the gopkg.in link point to there as well?

@MaximilianMeister definitely; I'm doing a sweep now, but will try to get a release out this week.

definitely; I'm doing a sweep now, but will try to get a release out this week.

great, thanks again!

@jszwedko btw do you plan to include this in a next stable release? would be great to have that commits when running govendor fetch gopkg.in/urfave/cli.v1


Browser to SS console

Menu actions to launch browser to hub and SS console

Added menu Launch  with item "SS console" and "Hub website". Google chrome browser  runs by  QProcess.start_detached() in --new_window and shows login page.


undefined is not a function (evaluating 'fs.existsSync(dirPath)')

I am trying to get Winston (Version 3.X) logger to work on React-Native using RN-Nodeify (what a package, thank you! <3) and I am stuck at an issue with fs.existsSync() when trying to output the logs to a file.

Postinstall script used:
"postinstall": "rn-nodeify --install util,buffer,os,fs,path,zlib,http,https,_stream_transform,assert,events,url,stream,process,tls,global,crypto,vm --hack"
Index.js - from line 1 (app entry point):
import "./shim";
global.Symbol = require("core-js/es6/symbol");
require("core-js/fn/symbol/iterator");
require("core-js/fn/map");
require("core-js/fn/set");
require("core-js/fn/array/find");
var fs = require("react-native-level-fs");
import { Navigation } from "react-native-navigation";
...

Is there any way to get round this issue with RN-Nodeify?

unfortunately fs.existsSync (along with many other synchronous methods) is not supported at the moment. RN introduced the ability to make calls over the bridge synchronously only a few versions ago, so you can probably add support to react-native-level-fs if you want :)

@mvayngrib Thanks for getting back to me! :)
Thanks for the heads up, I will have a look into react-native-level-fs. Closing this issue!


Update to install latest (4.6) Gradle version

Updating to install the latest Gradle version (4.6). Ubuntu currently doesn't have newer versions of Gradle available in their repo, so we have to download directly from the Gradle site.
Verified on Ubuntu 16.04 and 14.04.

The images have been updated on Docker Hub.


Don't use fs.promises to prevent warnings

Don't use fs.promises to prevent warnings from node.

Codecov Report

Merging #2602 into canary will increase coverage by <.01%.
The diff coverage is 55.55%.

@@            Coverage Diff            @@
##           canary   #2602      +/-   ##
=========================================
+ Coverage    7.95%   7.96%   +<.01%
=========================================
  Files         258     258
  Lines        9401    9405       +4
  Branches     1068    1069       +1
=========================================
+ Hits          748     749       +1
- Misses       8604    8607       +3
  Partials       49      49

Impacted Files
Coverage Δ

src/util/read-json-file.ts
0% <ø> (ø)
:arrow_up:

src/commands/deploy/legacy.js
0% <ø> (ø)
:arrow_up:

src/commands/deploy/index.js
0% <ø> (ø)
:arrow_up:

src/util/git.js
0% <ø> (ø)
:arrow_up:

src/util/index.js
0% <0%> (ø)
:arrow_up:

src/util/dev/server.ts
11.28% <0%> (ø)
:arrow_up:

src/util/hash.js
100% <100%> (ø)
:arrow_up:

src/util/read-metadata.js
100% <100%> (ø)
:arrow_up:

src/util/get-files.ts
86.5% <75%> (ø)
:arrow_up:

src/util/dev/builder-cache.ts
29.54% <0%> (-0.11%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad4e20c...de184d2. Read the comment docs.


invalid user when using chown with custom user

Hi!
First of all, this is an amazing tool! Thanks for the work you've done on it!
I've been setting up my Pi's for a while manually, and as you know it becomes a pain. Yesterday doing some research I discovered sdm, and here I am learning about it, to automate all the systems I use.
In this example I'm trying to set up a basic system with a hotspot, and that part I have it working already.
The issue comes when I try to do it under a different user than the default pi.
Reading the logs, I finally noticed the error:
chown: invalid user: ‘shadow:shadow'

Looking for related issue I found #126 , so I tried changing shadow:shadow to shadow.shadow, and later on also to only shadow, without specifying the group.
In all cases is the same outcome, Invalid user.
I don't know if I might be doing something wrong, or if it's actually an issue, so I decided to open this one.
The customize command I'm running is:
sudo sdm --customize --plugin L10n:host --plugin user:"adduser=shadow" --plugin mkdir:"dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700" --plugin copyfile:"from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif" --plugin hotspot:"config=hotspot.cfg" --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img

And the trace of the command is:
Mount IMG '2024-07-04-raspios-bookworm-arm64-lite.img'
mount: /dev/loop24 mounted on /mnt/sdm.
mount: /dev/loop25 mounted on /mnt/sdm/boot/firmware.
* Start Configuration
> Command Line: /usr/local/bin/sdm --customize --plugin L10n:host --plugin user:adduser=shadow
  --plugin mkdir:dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700 --plugin
  copyfile:from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif
  --plugin hotspot:config=hotspot.cfg --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img
* Host Information
   Hostname:  star
   Memory:    32592024 kB
   uname:     Linux star 6.8.11-amd64
   os-release Name: Kali GNU/Linux Rolling
   Version: 2024.2 (kali-rolling)
   Like OS: debian
   sdm Version: V12.4
* IMG Information
   Name: 2024-07-04-raspios-bookworm-arm64-lite.img
   Date: 2024-07-04
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12 (bookworm)
% sdm will use qemu 'aarch64'
% sdm will use systemd-nspawn on this 64-bit x86-64 host
  Retry the command with --chroot if this fails
* Plugins selected:
   * L10n
       Args: host
   * user
       Args: adduser=shadow
   * mkdir
       Args: dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700
   * copyfile
       Args: from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif
   * hotspot
       Args: config=hotspot.cfg
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 343072 1K-blocks (0.4GB, 0.3GiB) free
> Copy sdm to /usr/local/sdm in the IMG
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host:
> Plugin L10n: Read L10n configuration from host system
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       es
> Plugin L10n:   Locale:       en_US.UTF-8
> Plugin L10n:   Timezone:     Asia/Bangkok
> Plugin L10n:   WiFi Country:
> Plugin L10n: Save L10n configuration
* Plugin L10n: Complete Phase 0
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments:
  'adduser=shadow'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: shadow
> Plugin user: Create user 'shadow' home directory '/home/shadow' so available for other plugins
* Plugin user: Complete Phase 0
> Run Plugin 'mkdir' (/mnt/sdm/usr/local/sdm/plugins/mkdir) Phase 0 with arguments:
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase 0
> Plugin mkdir: Keys/values found:
   dir: /home/shadow/.ssh
   chown: shadow:shadow
   chmod: 700
> Plugin mkdir: Create directory '/home/shadow/.ssh'
* Plugin mkdir: Complete Phase 0
> Run Plugin 'copyfile' (/mnt/sdm/usr/local/sdm/plugins/copyfile) Phase 0 with arguments:
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase 0
> Plugin copyfile: Keys/values found:
   from: authorized_keys
   to: /home/shadow/.ssh
   chown: shadow:shadow
   chmod: 600
   mkdirif:
> Plugin copyfile: Add 'authorized_keys' to files to copy
> Plugin copyfile: Copy 'authorized_keys' to '/mnt/sdm/etc/sdm/assets/copyfile/.'
* Plugin copyfile: Complete Phase 0
> Run Plugin 'hotspot' (/mnt/sdm/usr/local/sdm/plugins/hotspot) Phase 0 with arguments:
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase 0
> Plugin hotspot: Keys/values found:
   config: hotspot.cfg
> Plugin hotspot: Read hotspot config 'hotspot.cfg' and update missing with defaults:
   channel: auto
   country: us
   device: wlan0
   dhcprange: <IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>
   domain: wlan.net
   enable: true
   hwmode: g
   leasetime: 24h
   passphrase: password
   ssid: MyNet
   wlanip: <IP_ADDRESS>
   type: routed
   hsmode: nm                                                   [Default]
> Plugin hotspot: Write hotspot config to /mnt/sdm/etc/sdm/assets/hotspot/config
* Plugin hotspot: Complete Phase 0
* Phase 0 Completed
* Enter image '2024-07-04-raspios-bookworm-arm64-lite.img' for Phase 1
* Start Phase 1 image customization
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 343000 1K-blocks (0.4GB, 0.3GiB) free at start of Phase 1 image customization
> Configure and enable sdm FirstBoot service (sdm-firstboot)
> Set hostname 'pearl'
> Start 'apt update'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 342996 1K-blocks (0.4GB, 0.3GiB) free at start of 'apt update'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 228016 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt update'
> Run Plugins Phase '1'
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host'
* Plugin L10n: Start Phase 1
% Plugin L10n: Keymap 'es' will be set during sdm FirstBoot
> Plugin L10n: Disable keyboard-setup service; will be re-enabled during sdm FirstBoot
Removed "/etc/systemd/system/sysinit.target.wants/keyboard-setup.service".
> Plugin L10n: Set locale to 'en_US.UTF-8'
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
> Plugin L10n: Set timezone to 'Asia/Bangkok'
* Plugin L10n: Complete Phase 1
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments: 'adduser=shadow'
* Plugin user: Start Phase 1
> Plugin user: Add user 'shadow'
useradd: group shadow exists - if you want to add this user to that group, use -g.
? Plugin user: useradd returned an error
* Plugin user: Complete Phase 1
> Run Plugin 'mkdir' (/usr/local/sdm/plugins/mkdir) Phase 1 with arguments:
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase 1
* Plugin mkdir: Complete Phase 1
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase 1 with arguments:
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase 1
> Plugin copyfile: Run /etc/sdm/assets/copyfile/run-phase1-authorized_keys-homeshadow.ssh
> Plugin copyfile: Process file 'authorized_keys'
> Plugin copyfile: Copy 'authorized_keys' from /etc/sdm/assets/copyfile/. to '/home/shadow/.ssh'
> Plugin copyfile: Set file '/home/shadow/.ssh/authorized_keys' owner 'shadow:shadow'
chown: invalid user: ‘shadow:shadow’
> Plugin copyfile: Set file '/home/shadow/.ssh/authorized_keys' protection '600'
* Plugin copyfile: Complete Phase 1
> Run Plugin 'hotspot' (/usr/local/sdm/plugins/hotspot) Phase 1 with arguments:
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase 1
> Plugin hotspot: Hotspot will be enabled by sdm FirstBoot
* Plugin hotspot: Complete Phase 1
* Phase 1 post-app installation/configuration
> Start 'apt upgrade'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 223164 1K-blocks (0.2GB, 0.2GiB) free at start of 'apt upgrade'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt upgrade'
> Start 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at start of 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of Phase 1 image customization
* Phase 1 Completed
> Run Plugins Phase 'post-install'
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments:
  'adduser=shadow'
* Plugin user: Start Phase post-install
* Plugin user: Complete Phase post-install
> Run Plugin 'mkdir' (/usr/local/sdm/plugins/mkdir) Phase post-install with arguments:
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase post-install
> Plugin mkdir: Change owner on '/home/shadow/.ssh' to 'shadow:shadow'
chown: invalid user: ‘shadow:shadow’
> Plugin mkdir: Change directory protection for '/home/shadow/.ssh' to '700'
* Plugin mkdir: Complete Phase post-install
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase post-install with arguments:
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase post-install
* Plugin copyfile: Complete Phase post-install
> Run Plugin 'hotspot' (/usr/local/sdm/plugins/hotspot) Phase post-install with arguments:
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase post-install
> Prefix hotspot: Configure hotspot 'Hotspot' with Network Manager
* Plugin hotspot: Complete Phase post-install
> Run graphics post-install configuration
> Enable SSH
> Plugin phase1: Enable SSH service
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221744 1K-blocks (0.2GB, 0.2GiB) free at end of image customization
> Customize elapsed time: 00:01:07
* Enter Shell Command Prompt
  'exit' to exit back to host system
root@sdm:/# exit
exit
* Customization complete
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

Do you have any idea what could be?
Again, with pi user I get it to work, because the user obv is already there, but it would be cool to be able to set others :)
Thanks!!

Glad that you're finding sdm useful! Here's what is happening, I'll leave it to you to decide your path forward.

In a brand new, untouched IMG there is already a group named shadow, and it's a system group (In Linux, group IDs less than 1000 are typically reserved for system groups).
When sdm creates a user, it uses the useradd command to create the user, which also tries to create a group with the same name. Unfortunately, there's already a group named shadow, so the useradd command fails:

Plugin user: Add user 'shadow'
useradd: group shadow exists - if you want to add this user to that group, use -g.
? Plugin user: useradd returned an error

Arguably, this should be a fatal error and stop the customization process, but at the moment it doesn't. Will look into that for a future release.
Since the user shadow was not created, all subsequent attempts that try to use the user shadow also fail: specifically, your mkdir and copyfile plugin invocations

The easiest fix for you is to rename the user shadow to something else. At the moment there's no workaround in the user plugin for this.
Here are some other notes and comments that I hope are helpful.
I'd encourage you to consider moving all your plugins into a pluglist file documented here. It eliminates clutter on the command line, makes it easier to see what plugins you're using, and you don't need all the double quotes around plugin arguments. So, your customize command line would be
sudo sdm --customize  --plugin @/path/to/pluglist --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img`

And /path/to/pluglist file would contain (I used xshadow instead of shadow):
L10n:host
user:adduser=xshadow
mkdir:dir=/home/xshadow/.ssh|chown=xshadow:xshadow|chmod=700
copyfile:from=authorized_keys|to=/home/xshadow/.ssh|chown=xshadow:xshadow|chmod=600|mkdirif
hotspot:config=hotspot.cfg

I don't know about you, but I think this is a lot easier to work with! [NOTE: I did not test the above]
I typically don't set the hostname during a customize, saving that until I burn the disk. That way, a customized IMG can be used for multiple target systems. Of course, you can still set the hostname on a burn if you've set it during customization, so not a problem, but I find that being rigorously prescriptive about the details keeps me from confusing myself 😉
I haven't looked at the hotspot plugin in quite a while. LMK if you run into any problems with it.

Oh man! Good catch! When I was writing the name I thought.. shadow is too common of a name, thinking of /etc/shadow, but I just went on with it ^_^ I wasn't aware of the existing group.
It works now! Thanks!
For organizing the plugins/commands I'm moving to the script ezsdm, so I can have different folders for each card, with the script ready to be run, with some assets that might be necessary for each set up. I also find that having it on a list is way more managable than on cluttered commands, and easier to version :)
Good tip also on the hostname :+1: , I'm still figuring the commands and how they interact on the different phases. Jumping soon on the code to understand some internals I'm still missing :p
I love it!! It's prob becoming my default setup from now :)
Thanks!!

Closing the issue, since it was never such a thing :pray:

Oh man! Good catch! When I was writing the name I thought.. shadow is too common of a name, thinking of /etc/shadow, but I just went on with it ^_^ I wasn't aware of the existing group.
It works now! Thanks!

Excellent ;) Thanks for including /etc/sdm/history, that makes it SO easy to see what's going on. I've been reading them for a few years so very good at scanning for problems.

For organizing the plugins/commands I'm moving to the script ezsdm, so I can have different folders for each card, with the script ready to be run, with some assets that might be necessary for each set up. I also find that having it on a list is way more managable than on cluttered commands, and easier to version :)

Great!

Good tip also on the hostname 👍 , I'm still figuring the commands and how they interact on the different phases. Jumping soon on the code to understand some internals I'm still missing :p
I love it!! It's prob becoming my default setup from now :)
Thanks!!
Thanks for the positive feedback! Happy to help with any further questions, problems, suggestions, etc.

Thanks! :pray:
I think I'm starting to figure out the best way to use the tool. In fact most of my cards share some base set up, like hotspot, aliases, etc.
So I think having an ezsdm for the base image makes sense. Then, specific projects can start from there. But this is where I'm not so sure how to proceed.
Can I run another --customize on top of an already customized image? I assume the last customize will override something that was set on the base if I specify so, but keeping the rest intact. Is that a valid approach?
The thing is, for more complex cards, I think I'll need to set them up manually with --explore. In this case the project folder will actually not hold its own ezsdm, but just an already customized base image.
I would like to be able to back up everything with only the script, without having to push the *.img's, but in the approach I explained above the "manual steps" would be lost.
It is still light-years better than what I have currently tho, which is some notes and bad memory :joy:

@sraver some answers and a bit about what I do.
Yes, you can run another --customize on an IMG, but I think there are better ways. That said, when you try to customize an already-customized IMG, sdm warns you. You can omit the warning by using --redo-customize.
I try to script everything, so none of my systems are done manually. At the moment I have 3 Pi "servers":

One with DHCP/DNS (not pihole!). It has a few other services, such as apt-cacher-ng. BTW, if you have multiple Pi systems and aren't using apt-cacher-ng, you should make getting that working a top priority. You're welcome in advance 😉
One runs a VPN server, serves a 1TB work-disk, and is a "cold" backup DHCP/DNS server. It copies the DHCP/DNS json database at regular intervals (currently daily) and rebuilds it's database if needed. Failover is manual because in 25 years of running my own DHCP/DNS server I've not had a system fail. If that first time ever happens, I can have the network back up and running quicky enough for my needs
One piwatch/flightradar24 system

My other Pi systems are all configured as workstations.
 And I always do a fresh install. Only the best fresh new bugs for MY systems 🤣, followed by 2 years of basically not futzing with it)
I have a "standard" configuration for my Pis. It's the "workstation" configuration. When I want to build a new server, regardless of what it is, I burn the disk for that server and run whatever plugins are needed to turn it into the server. For instance, to rebuild my DHCP/DNS server I run plugins to:

Install the DNS/DHCP server (using sdm's ndm plugin)
Install apt-cacher-ng using the like-named plugin
Install postfix using the like-named plugin
Install logwatch using an as-yet unreleased like-named plugin
Add a .nmconnection file using the network plugin that configures it's static IP address (kinda needed for the DHCP/DNS server)
Use the system plugin to configure /etc/exports
Run another plugin that does host-specific stuff not yet done via other plugins.

The burn command. $dev, $host, and $img are appropriately set.
sdm --burn $dev --hostname $host --expand-root --plugin @/l/work/mystuff/$host/$host.plugins  $img

I have similar pluglist files ($host.plugins) for each server.
I'd argue that there's very little, if anything that you can only do manually by exploring into an IMG. It's taken me a while to get to this point, of course. For Bullseye, I was partially there. I spent some time completely redoing the implementation for my servers to take advantage of pluglists, new plugins, and new system capabilities, such as NetworkManager.  I think it's much easier to achieve this nirvana with the current sdm and Bookworm.
The beauty of this, which I'm sure you recognize, is that bad memory is not a failure mode. Bad code might be 🤣 but I try to avoid that.
As far as backups, I don't do full backups. I figure out what config files I need to backup, and only back them up if they've changed. I have a script that runs nightly to do that.
re Ansible: I used it for a while, but it seemed fairly heavyweight and sluggish for my impatience to tolerate. That said, I see the value in it. If I were to use Ansible now, I would use sdm to install whatever bits needed to be installed on every system in my workstation IMG (ssh keys, etc). I'd install Ansible itself on whatever systems I want to be able to run ansible commands on, and I'd install the configuration data on an NFS share so that I can run ansible on any server as needed.
That's it in a nutshell. You'll undoubtedly do things differently than me.
TLDR: Figure out how to configure each and every system completely using sdm, and use as few customized IMGs as possible. Defer system-specific customization to when you burn the disk for that system.

Thanks for such detailed explanation!
Very helpful to get a glimpse of how you are using it, it helps understanding the thought process on the tool itself.
You opened a new rabbit hole for me now, ndm xD Previosuly I had been using pi-hole, but I'm in the search of something more lightweight and easily manageble, from the terminal if possible, and ndm seems to tick all the boxes!
I'm a feeling a bit stuck right now tho, not because any issue I'm finding in the tool, but more because of my lack my understanding of some pieces of the whole system I'm trying to implement.
Would you mind me writing an email to you to explain what I'm trying to achieve, and maybe I could get your thoughts on it? I don't mind sharing the final approach here, even though it would be completely off-topic on this very issue, but I feel I'm a bit lost, and I'd rather share a clean approach once I have it.
Regardless, thanks for your time, and all the tips and tricks! :heart:

ndm: yep, it does dhcp and dns really well, and it's lightweight. pihole has some nice features, but I've seen enough people having problems with it...
re ndm and backups: Correct, you still need to back up the configuration, but it's exactly one (json) file. With that you can easily rebuild and reinstall the dhcp and dns servers configs with 2 ndm commands. The json file is one of the files that I back up nightly, BTW, and my network doesn't change very often so once a day is fine. If you're changing the network a lot, you can back it up more often.
re email: 👍


Distribution Support Request

Details of the Distribution

Name of the Distribution: Ubuntu
Version of the Distribution: 20.04
Package Types

[X] Deb
[ ] RPM
[ ] Tar.gz
Snap - Please file issue in https://github.com/powershell/powershell-snap.  This issues type is unrelated to snap packages with a distribution neutral.

Processor Architecture (One per request):
[ ] Required - An issues has been filed to create a Docker image in https://github.com/powershell/powershell-docker
The following is a requirement for supporting a distribution without exception.

[ ] The version and architecture of the Distribution is supported by .NET Core.

The following are requirements for supporting a distribution.
Please write a justification for any exception where these criteria are not met and
the PowerShell committee will review the request.

[ ] The version of the Distribution is supported for at least one year.
[ ] The version of the Distribution is not an interim release or equivalent.

Progress - For PowerShell Team ONLY

[ ] Docker image created
[ ] Docker image published
[ ] Distribution tested
[ ] Update packages.microsoft.com deployment
[ ] Lifecycle updated
[ ] Documentation Updated

Looks like there's already an existing issue for this https://github.com/PowerShell/PowerShell/issues/12626


Add grpc request and response  metadata

Fixes #3358
Short description of the changes
Type of change

[x] New feature (non-breaking change which adds functionality)
[x] This change requires a documentation update

How Has This Been Tested?
I assert span results of a call to a unary method which contains custom metadata in both the request and response.
Checklist:

[x] Followed the style guidelines of this project
[x] Unit tests have been added
[x] Documentation has been updated

Please let me know if I added the changes to the correct changelog as I see there's another one in opentelemetry-js/experimental :)

Please let me know if I added the changes to the correct changelog as I see there's another one in opentelemetry-js/experimental :)

You  shoud move the entry to experimental/changelog.md since the package you modified is in experimental

Please let me know if I added the changes to the correct changelog as I see there's another one in opentelemetry-js/experimental :)

You shoud move the entry to experimental/changelog.md since the package you modified is in experimental

Moved

cc @open-telemetry/javascript-approvers if someone have some time to review this :)


SQL database source helpers has no fallback for tz in row_tuples_to_arrow

dlt version
0.4.13a0
Source name
sql_database postgres
Describe the problem
When using backend pyarrow with the postgres source, pipeline.run() failed due to the missing value of tz argument.
This is not convenient for users. I suggest we just go with "UTC" as a default value for tz.
╰─➤  python sql_database_pipeline.py
2024-06-12 14:03:52,284|[INFO                 ]|38392|8625376256|dlt|pipeline.py|_restore_state_from_destination:1452|The state was restored from the destination postgres(dlt.destinations.postgres):dlt_dev
2024-06-12 14:03:52,364|[INFO                 ]|38392|8625376256|dlt|__init__.py|bind:424|Bind incremental on dev_table with initial_value: None, start_value: None, end_value: None
Traceback (most recent call last):
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/extract/pipe_iterator.py", line 275, in _get_source_item
    pipe_item = next(gen)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database/helpers.py", line 219, in table_rows
    yield from loader.load_rows(backend_kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database/helpers.py", line 118, in load_rows
    yield from self._load_rows(query, backend_kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database/helpers.py", line 140, in _load_rows
    yield row_tuples_to_arrow(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database/schema_types.py", line 135, in row_tuples_to_arrow
    arrow_schema = columns_to_arrow(columns, tz=tz)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/common/configuration/inject.py", line 243, in _wrap
    config = resolve_config(bound_args, accept_partial_=accept_partial)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/common/configuration/inject.py", line 171, in resolve_config
    return resolve_configuration(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/common/configuration/resolve.py", line 67, in resolve_configuration
    return _resolve_configuration(config, sections, (), explicit_value, accept_partial)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/common/configuration/resolve.py", line 157, in _resolve_configuration
    _resolve_config_fields(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/common/configuration/resolve.py", line 293, in _resolve_config_fields
    raise ConfigFieldMissingException(type(config).__name__, unresolved_fields)
dlt.common.configuration.exceptions.ConfigFieldMissingException: Following fields are missing: ['tz'] in configuration with spec ColumnsToArrowConfiguration
        for field "tz" config providers and keys were tried in following order:
                In ExplicitValues key tz was not found.
Please refer to https://dlthub.com/docs/general-usage/credentials for more information

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 443, in extract
    self._extract_source(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 1141, in _extract_source
    load_id = extract.extract(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/extract/extract.py", line 399, in extract
    self._extract_single_source(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/extract/extract.py", line 324, in _extract_single_source
    for pipe_item in pipes:
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/extract/pipe_iterator.py", line 159, in __next__
    pipe_item = self._get_source_item()
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/extract/pipe_iterator.py", line 306, in _get_source_item
    raise ResourceExtractionError(pipe.name, gen, str(ex), "generator") from ex
dlt.extract.exceptions.ResourceExtractionError: In processing pipe dev_table: extraction of resource dev_table in generator table_rows caused an exception: Following fields are missing: ['tz'] in configuration with spec ColumnsToArrowConfiguration
        for field "tz" config providers and keys were tried in following order:
                In ExplicitValues key tz was not found.
Please refer to https://dlthub.com/docs/general-usage/credentials for more information

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database_pipeline.py", line 289, in <module>
    load_select_tables_from_database()
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/sql_database_pipeline.py", line 34, in load_select_tables_from_database
    info = pipeline.run(source_1,
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 228, in _wrap
    step_info = f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 273, in _wrap
    return f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 698, in run
    self.extract(
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 228, in _wrap
    step_info = f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 182, in _wrap
    rv = f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 168, in _wrap
    return f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 273, in _wrap
    return f(self, *args, **kwargs)
  File "/Users/dle/Documents/VSCode/data_platform_dlt/apps/dlt/postgres_tracking/.venv/lib/python3.9/site-packages/dlt/pipeline/pipeline.py", line 464, in extract
    raise PipelineStepFailed(
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at stage extract when processing package<PHONE_NUMBER>.316936 with exception:

<class 'dlt.extract.exceptions.ResourceExtractionError'>
In processing pipe dev_table: extraction of resource dev_table in generator table_rows caused an exception: Following fields are missing: ['tz'] in configuration with spec ColumnsToArrowConfiguration
        for field "tz" config providers and keys were tried in following order:
                In ExplicitValues key tz was not found.
Please refer to https://dlthub.com/docs/general-usage/credentials for more information

Expected behavior
No error should be thrown.
A default value of tz should be applied (e.g. "UTC").
Steps to reproduce
import dlt
from sql_database import sql_database

def load_select_tables_from_database() -> None:
    """Use the sql_database source to reflect an entire database schema and load select tables from it.
    """

    # Create a pipeline
    pipeline = dlt.pipeline(
        pipeline_name="dev_dlt_survey",
        destination='postgres',
        dataset_name="dlt_dev",
        # progress=dlt.progress.tqdm(colour="yellow"),
    )

    # Configure the source to load a few select tables incrementally
    source_1 = sql_database(schema="dlt_active_campaign",
                            backend="pyarrow").with_resources(
        "dev_table")

    source_1.dev_table.apply_hints(
        incremental=dlt.sources.incremental("sale_date"),
        primary_key="id",
    )

    info = pipeline.run(source_1,
                        write_disposition="merge",
                        refresh="drop_resources"
                        )
    print(info)

How you are using the source?
I'm considering using this source in my work, but bug is preventing this.
Operating system
macOS
Runtime environment
Local
Python version
3.9.11
dlt destination
postgres
Additional information
No response

@khoadaniel thanks for the report! the alpha version that you are using changes how dlt handles None passed to arguments with default values: https://github.com/dlt-hub/dlt/pull/1430#issue-2329339449
we will most probably release the current alpha as a "major" version and together with all the fixes needed in verified sources.
btw. this fix is correct :) I have exactly the same patch ready and since it is backward compatible I'll merge it before the release above


Reload PDF without losing scroll position?

Question 1: Is it possible to make PDFViewerApplication to reload the PDF without losing its scroll position?
I was able to trigger a reload by using PDFViewerApplication.open(), but the scroll position returns to the top.
If there is no such function, I could save the scrollTop property of #viewerContainer and then set it again after the PDF has been reloaded, but for that I would need to know when the PDF has been rendered.
Question 2: Is there a render-finished callback that I can register?

I realized that PDFViewerApplication.open() returns a Promise, so I tried the following:
const scrollTop = document.getElementById('viewerContainer').scrollTop
PDFViewerApplication.open(pdfname).then(
    () => {
      document.getElementById('viewerContainer').scrollTop = scrollTop
    }
  )

Unfortunately, it doesn't work, after the reload the scroll position returns to the top.

That's already the default behaviour, unless you've manually changed preferences/options, when you simply reload the viewer demo at https://mozilla.github.io/pdf.js/web/viewer.html

Right. The difference in my case is that I have the viewer.html in an iframe, see #11358. But that means I can trigger a reload using .contentWindow.location.reload(), correct?

Assuming that you're actually re-opening the exact same file, then I cannot imagine why directly calling PDFViewerApplication.open() would produce a different behaviour (than the demo above).

No, it's not the exact same file, otherwise the reload wouldn't change anything. The context is that slightly modified versions of the PDF are generated, based on a source document (Markdown processed by Pandoc). Because the PDF is different, reloading makes sense, and because the changes are almost always very small, keeping the scroll position makes sense.

generally no point where all pages are rendered

OK, makes sense. Would the documentinit event notify me when all the pages are represented by not-yet-rendered page placeholders, so that scrolling to the same position will work?

No, it's not the exact same file, otherwise the reload wouldn't change anything. The context is that slightly modified versions of the PDF are generated,

The functionality in question relies on the document fingerprint, which is defined in the PDF document itself. Hence if the document is modified, the PDF generator will create a new fingerprint since it's then a "new" PDF document. (There's generally no good/simply way of supporting exactly what you seem to want in this case, with the exception of a full browser-reload, without issues elsewhere.)

[...] and because the changes are almost always very small, keeping the scroll position makes sense.

In the general case, the "almost always very small" part cannot really be assumed to hold.

Would the documentinit event notify me when all the pages are represented by not-yet-rendered page placeholders, so that scrolling to the same position will work?

Yes; with the caveat that if disableAutoFetch was set (when calling getDocument) this may not work perfectly for documents with different page sizes.

Closing since this is answered. I'm also linking #6847 here which seems relevant.

Alright, I finally figured it out, and I describe the solution here in case some else needs this.
The default reload behavior of the pdf.js viewer is indeed to preserve the scroll position (and some other GUI states) if the file is exactly the same. However, it doesn't check the complete file contents (via a hash or fingerprint) but relies on the document ID that is typically (but not necessarily) embedded in a PDF file (see 14.4 in the specification).
In my case, the (contentwise slightly different) PDFs are generated by pdflatex / xelatex / lualatex, and apparently they generate the PDF ID based on the current time and the pathname of the document. Therefore, even if I don't change anything in the LaTeX document, just process it again to generate a PDF, this PDF has a different ID, and the pdf.js viewer does not treat it as the same document (preserving the scroll position), but as a new one (starting with the scroll position at the top).
Fortunately, there is a way around this. If the environment variable SOURCE_DATE_EPOCH is set to a Unix time stamp (number of seconds since 1 Jan 1970 00:00 UTC), then this time will be used for the PDF ID instead of the current time. Since the pathname of the document doesn't change either, this ensures an identical ID across regenerations of the PDF, and therefore the viewer shows its scroll-preserving behavior on reload.

I found another solution by overriding the fingerprint inside the PDFDocumentProxy:
// Binary pdf contents stored in data
let doc = await pdfjsLib.getDocument({data}).promise;
// Override the fingerprint after parsing but before passing to the view layer.
// This means that the view state (page, scroll offset, etc.) is preserved.
doc._pdfInfo.fingerprint = 'constantFingerprint';
PDFViewerApplication.load(doc);

Note we have to assign to doc._pdfInfo.fingerprint because doc.fingerprint is a getter. This doesn't appear to have ill effects despite accessing the presumably private _pdfInfo.
This is similar to the solution in #11496, but doesn't require patching pdf.worker.js. You could get fancier here and use a fingerprint based on the e.g. basename or path of the file.

When using frameworks like ng-pdfjs-viewer, this doens't work as pdf.js is somewhat hidden.
In our use case, we have a table containing scanned PDFs on the left and the actual PDF on the right of the selected scan. Each scan is the same type of document and user needs to manually verify the scanned data on e.g. page 2 zoomed in to easily check values. Then user traverses the rows and verifies the data is correct at the exact (same) position in all scans.
Our hack solution that works in combination with ng-pdfjs-viewer is:

copy the pdfjs folder into /assets/ This is happening either way so remove it out of the angular.json as you are doing it yourself)
modify /assets/pdfjs/pdf.js to check for a global variable

key: "fingerprint",
    get: function get() {
      const currentPdfFingerPrintOverride = window.parent.currentPdfFingerPrintOverride;
      if(currentPdfFingerPrintOverride){
        console.warn('PDF UI State Saving feature in pdf.js: Overriding PDF fingerprint to ',currentPdfFingerPrintOverride);
        return currentPdfFingerPrintOverride;
      }else{
        return this._pdfInfo.fingerprin
      }
    }
  }

Then somewhere in your Angular code or (anywhere you want in plain javascript) set the value

window.currentPdfFingerPrintOverride = fingerprint;


CMP CLI-based tests: small extensions and improvements

This extends and tweaks some of the test cases in 80-test_cmp_http_data/*.csv:

test_commands.csv: improve test for -reqin, adding -reqin_new_tid
test_verification.csv: add missing test case for -untrusted with non-matching cert
test_enrollment.csv: clean up test cases regarding (non-existing) directories,
which improves #15575.

[x] tests are added or updated

BTW, I am missing a label called, e.g., triaged: test.

Adding a new test is a feature. :grin:

Adding a new test or improvement (not a bug fix) of existing one are features. :grin:

This makes some sense, but the same could have been said for triaged: documentation.
Addressing a test directly would be more precise,
and then the template given when opening a PR would become obsolete:
- [x] documentation is added or updated
- [ ] tests are added or updated

the template given when opening a PR would become obsolete:

it would not as external contributors cannot set labels

the template given when opening a PR would become obsolete:

it would not as external contributors cannot set labels

True.
Still triaged: test may make sense, and even issue: test,
for differentiation from productive actual (library/app) features.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged - thanks @paulidale and @t8m


Add percent symbol

Makes instructions more readable and understandable.

I think that within 10 of your monthly budget means monthlyBudget-10 or monthlyBudget+10.
If your monthly budget is 100 and the monthly payment is in the range of 90...110, it returns true as we want -10 and +10 to behave the same way.

I totally agree that the instructions could be interpreted exactly as you state, but on
retrospection, it doesn't really make sense to interpret them in that manner.
Considering the example calls for Task 1, the first call is for a very inexpensive car
fitting a very limited budget.  The second example call is for a car costing over 2 million
fitting an entirely different budget - ten dollars either way on this kind of budget
really becomes almost meaningless.
A budget tolerance of 10%, on the other hand, on a budget of 10,000 (e.g. a max of 11,000)
is probably a reasonably consideration for someone of such means. Compare that to a budget
tolerance of 10% on a budget of 100 (e.g. a max of 110). Someone with this lower budget will
be more sensitive to a change of this magnitude.
If a purchase is within the budget there is really no reason to be frugal, making the
(budget - 10) value kind of pointless.  The budget + 10 does have real meaning since frugality
could mean the difference of acquiring the asset or not.
Just writing this explanation, highlights the grey areas involved by not including a
symbol whether it be a % or a currency symbol ¢, $, or £, etc.  It might be argued
that by the term budget, the author meant income, although these two should
never be confused.  IMHO, the percent symbol makes the most sense to add to the
instructions as it bears a close relation to a pragmatic budget and code written to this
specification will still pass the tests.


Dynamic binding problem with 3rd party library directive

I've been trying to integrate angular-bootstrap-calendar with ionic framework. There seems to be some issues which I can't seem to resolve. The example is in the codepen.io link below:
http://codepen.io/anand1st/pen/OMvWmq
I notice that the events appear on the calendar when the ng-controller is in a directive that's directly above the mwl-calendar i.e. in the example above, it's the ion-content. The moment I move the ng-controller up to ion-pane directive, the events don't show up in the calendar anymore.
The author of angular-bootstrap-calendar seems to think that this is a bug with ionic framework. How can dynamic binding work correctly with the ng-controller moved to the ion-pane directive?

This issue was moved to driftyco/ionic-v1#187


add button compatability

Hello, i know that this project hasnt been iterated on recently, but if you could approve this PR that would be very appreciated. Love the code, but I need the ability to activate a checkpoint through a button click.

Hey, sure thing. We might be archiving this soon anyway, so not too fussed. :)

Thank you for merging! When you get a chance can you update the npm package as well :D

Sorry, I recommend just installing from source.


err faild xhr

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

i can't update vscode
and
i can't install extensions

Closing as this is not reproducable.
If you find nice reproducable steps please open a new issue in the VS Code repository. Thank you


Casting nil embed when embed already populated

Hi,
I have an issue casting nil when the embed is not empty, If i pass in a new map, it works as expected; However when the map contains values, it fails.

Thank you for adding the failing test 😁


[Feature] Reaction diagram plotter print "total energy change" and "equilibrium potential"

Reaction diagram plotter print "total energy change" and "equilibrium potential".

Added by 78d660d60067e2ed601f96ec772d9eee26427271 and f4fdb27b380ee607ab7d9f28695a0b1358846b24. But the "equilibrium potential" calculation method is not added for the time being as I cannot think of a reliable method to detect total number of electron transfers.


error: build of ‘/nix/store/...-VirtualBox-GuestAdditions-5.0.12-4.5-rc5.drv’ failed

Issue description
I had upgrated from 15.09 to 16.03 branch. After that system rebooted. To update all packages I ran nix-env -u and got error while VirtualBox-GuestAdditions-5.0.12-4.5-rc5 compilation.
Log: http://pastebin.com/NTL4QDec

System: 16.03.526.567b65d (Emu)
Nix version: nix-env (Nix) 1.11.2
Nixpkgs version: "16.03.526.567b65d"

Sounds like a too new kernel for VirtualBox.

I did't install 4.5-rc5 kernel specially. It's runs automatically. Also I had see while nixos-rebuild switch --upgrade works - it's compiled 4.5 kernel.

Yes, nix-env -u does that.

Getting this during nixos-install! Booted from the iso available from https://nixos.org/nixos/download.html which is nixos-minimal-16.03.769.f98d57b-x86_64-linux.iso

@boobiesinc can you post logs?

@domenkozar
The logs looked like the ones OP posted (see from line 585)
Then, I tried to make it use 5.0.20 by modifying version and sha256 but I must've missed something since I was getting the name 5.0.20 but the same 5.0.12 sources, here's a pic:

But note however that I was using a reset --hard version of the repo to channels/16.03 that's why the version was 5.0.12 not 5.0.14(as I have linked)
I was using these lines in configuration.nix however:
linuxPackages = pkgs.linuxPackages_4_5; (this is likely why I was getting the issue, newer kernel with older vbox)
also
virtualisation.virtalbox.guest.enable = true;
and
virtualbox = linuxPackages.virtualbox.override {
    enableExtensionPack = false;
    pulseSupport = true;
  };

and
services.xserver.videoDrivers = [ "virtualbox" ];
and
virtualisation.virtualbox.host.enable = false;

I have just switched to booting nixos-minimal-16.09pre83100.25e3c09-x86_64-linux.iso
and using the latest master when doing the nixos-install like so:
NIX_PATH="nixpkgs=/mnt/etc/nixos/nixpkgs" nixos-install
I have kept the above lines in my configuration.nix, of course.
Noting that I see some linux-headers-4.4.10 during compilation of something(glibc 2.23).
Compiling...

Sounds like this has been fixed in the meantime.


All storage SDK documentation "Main Page"s have no content

This page links to the Azure SDK for C++ page, which has a left-side tab for "Storage". All of the links on that tab (azure-storage-blobs, azure-storage-common, etc.) lead to pages which claim to be generated by doxygen, and have the usual doxygen page chrome, but otherwise have no content at all. For example, this page.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 16019ec6-63b0-36f7-5597-628a843ad3e7
Version Independent ID: 55f83505-93ab-5c81-0446-cf3411d28342
Content: Quickstart: Azure Blob Storage library v12 - C++
Content Source: articles/storage/blobs/quickstart-blobs-c-plus-plus.md
Service: storage
Sub-service: blobs
GitHub Login: @kyle-patterson
Microsoft Alias: kylepa

The doxygen pages should include our class reference docs, navigation is currently under the "Classes" navigation item.
@RickWinter is there a way we can include the README from each library in the landing page for the respective libraries?

OK, you're telling me the problem is "the main page is empty", which I agree is a real problem. I'll change the issue title.
The "Concepts and Related Pages" page is almost completely empty, having only a revision history link; that, too, is a problem.

@kyle-patterson We can certainly add the readme to be the index.html content.  I've opened a feature-request to do so.
https://github.com/Azure/azure-sdk-for-cpp/issues/3121

@RickWinter Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.

#reassign: @tamram

@jasonzio - The C++ SDK documentation is provided "as-is". As @kyle-patterson pointed out, the Classes tab provides access to all of the classes in the SDK.
#please-close


Add pattern matching to input files

Closes #3
Update: this has been tested with a single .dsc file match. Basic testing of the concept suggests that this should scale properly.

Thanks you for looking into this
I think you mentioned in issue #1 that we can't upload multiple packages.
Is this to avoid hard-coding the .dsc file name?
Also, shellcheck seems to have found a few issues.

You can see what we are doing here, which has a few commits addressing a couple of different things that we wanted.
[diff]
It's possible that some other changes were made to this other than what is presented in this PR - you may wish to simply review this yourself to see if you wish to use any of it -  we are likely to continue to use our fork.

Wow, you do have a lot of local changes.
Cool, I'll take a look into this part, let me know if you have other interesting things there.