Module 1: Portal changes need updated instructions

Contact Details
No response
What happened?
Exercise: 1
Task: 2
Description of issue:

Step 10: Instead of "select Manage" it should be "Apply standard policy"
Between Steps 12 and 13: There is a new page "Turn on the policy after I finish"
Step 14: Instead of "select Manage" it should be "Apply strict policy"
Between Steps 16 and 17: There is a new page "Turn on the policy after I finish"

Lab
Lab 01 Exercise 01 Explore Microsoft 365 Defender
Relevant screenshots

Do you want to help us? 👏

[ ] Create a Pull Requestt by following this project's Contribution Guide

I saw this yesterday. I'll get it updated.

This is updated.


Drop oc/kubectl tests in tests/vm_test.go

What happened:
The cases testing oc and kubectl in vm_test.go (possibly also in other files) are questionable and should be either dropped or moved to tests/kubectl_test.go if deemed useful.
https://github.com/kubevirt/kubevirt/blob/bcd91649e91b0df8312d27725366823abdd2497c/tests/vm_test.go#L1499
What you expected to happen:
Drop unnecessary tests that test software (oc or kubectl) which is not developed in this repository.
How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.
Additional context:
Add any other context about the problem here.
Environment:

KubeVirt version (use virtctl version): N/A
Kubernetes version (use kubectl version): N/A
VM or VMI specifications: N/A
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): N/A
Kernel (e.g. uname -a): N/A
Install tools: N/A
Others: N/A

/remove-lifecycle stale
@0xFelix is doing a good job addressing this, but there's more to come.


fix(git-validator): if a folder only contains md files, checking this folder will fail

ac9714cbf8b761cdb60910a31d833e919c17fcb3


build: fix the dependency issue

Summary
While graduate the beta version, github action cannot recognize the latest change in React Native and Node.

Checklist

[ ] Does your PR title have the correct title format?
Does your PR have a breaking change?:

Sync with Justin, why we use 2.0.0-beta.0 in node and react is because there is no 2.0 for node and react-native. And we have not published them yet.  So this shouldn't be changed. Will close this pr.
However, it's preventing me to graduate the analytics-browser SDK. It might be the issue between this pr and the graduate process.


[Snyk] Security upgrade jszip from 3.7.1 to 3.8.0

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `yarn` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

package.json
yarn.lock

Vulnerabilities that will be fixed
With an upgrade:

Severity
Priority Score (*)
Issue
Breaking Change
Exploit Maturity

601/1000  Why? Recently disclosed, Has a fix available, CVSS 6.3
Arbitrary File Write via Archive Extraction (Zip Slip) SNYK-JS-JSZIP-3188562
No
No Known Exploit

(*) Note that the real score may have changed since the PR was raised.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Allow global resync & reconcilerImpl informers to be filtered via filterfunc

related to #1764
There were two missing places where the informers from the shared cache we're not being filtered.

in the globalresync functions (where the filterFunc being passed through was ~= return true
the informer in the LeaderAwareFuncs.PromoteFunc which was marked with a TODO to allow filtering

The former has been changed to allow for a controllerImpl.GlobalResyncFilterFunc to be set (and if not, the former default of return true is passed to allow objects.
The latter was complicated to pass as a function, if done as a controller.Option, it would be too late for the closure in the initial reconcilerImpl.  I've opted instead for a pkg/controller/controller.go variable that will store via the shared context and again -- if not set -- pass a default filter of return true filtering over the objects.
/cc @vaikas @n3wscott @matzew

thanks for the reviews @julz, should be fixed at this point.

Which state is this one in? Seems like it fell between the cracks @lberk ?

@markusthoemmes this is still a problem in eventing.  I was asked for further justification that this solves the root cause of the issue, I haven't had time to return to it since.

I'm going to close this out since it's been a number of months - feel free reopen

I’m pretty sure Options now has PromoteFilterFunc…


Livekit 2.0 Native crash Fatal signal 11 (SIGSEGV)

I am getting a  Fatal signal 11 (SIGSEGV). I think it started since upgrading to 2.0 of Livekit. the issue seems to be related to NetworkMonitor. I have disabledNetworkMonitor in options.
Reproduced both on Samsung A52 and Pixel 6
Seems to happen only when I end a call, but not always reproducable
private fun buildPeerConnectionFactory(): PeerConnectionFactory {
    return PeerConnectionFactory
        .builder()
        .setVideoDecoderFactory(DefaultVideoDecoderFactory(rootEglBase.eglBaseContext))
        .setVideoEncoderFactory(
            DefaultVideoEncoderFactory(
                rootEglBase.eglBaseContext,
                true,
                true
            )
        )
        .setOptions(PeerConnectionFactory.Options().apply {
            disableNetworkMonitor = true
        })
        .createPeerConnectionFactory()
}

2024-04-18 12:28:11.189 12226-12461 libc                    com.bambuser.onetoone.dev            A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xffffffc2000008 in tid 12461 (ConnectivityThr), pid 12226 (er.onetoone.dev)
2024-04-18 12:28:12.061 13151-13151 DEBUG                   pid-13151                            A  Cmdline: com.bambuser.onetoone.dev
2024-04-18 12:28:12.061 13151-13151 DEBUG                   pid-13151                            A  pid: 12226, tid: 12461, name: ConnectivityThr  >>> com.bambuser.onetoone.dev <<<
2024-04-18 12:28:12.062 13151-13151 DEBUG                   pid-13151                            A        #00 pc<PHONE_NUMBER>390c44  /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!libjingle_peerconnection_so.so (offset 0x1603000) (BuildId: 6ca9af36c844c199)
2024-04-18 12:28:12.062 13151-13151 DEBUG                   pid-13151                            A        #01 pc<PHONE_NUMBER>41b5c8  /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!libjingle_peerconnection_so.so (offset 0x1603000) (BuildId: 6ca9af36c844c199)
2024-04-18 12:28:12.062 13151-13151 DEBUG                   pid-13151                            A        #02 pc<PHONE_NUMBER>41b4b0  /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!libjingle_peerconnection_so.so (offset 0x1603000) (Java_livekit_org_webrtc_NetworkMonitor_nativeNotifyOfNetworkConnect+32) (BuildId: 6ca9af36c844c199)
2024-04-18 12:28:12.062 13151-13151 DEBUG                   pid-13151                            A        #08 pc<PHONE_NUMBER>061cc4  [anon:dalvik-classes33.dex extracted in memory from /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!classes33.dex] (livekit.org.webrtc.NetworkMonitor.notifyObserversOfNetworkConnect+0)
2024-04-18 12:28:12.062 13151-13151 DEBUG                   pid-13151                            A        #13 pc<PHONE_NUMBER>061ab4  [anon:dalvik-classes33.dex extracted in memory from /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!classes33.dex] (livekit.org.webrtc.NetworkMonitor.-$$Nest$mnotifyObserversOfNetworkConnect+0)
2024-04-18 12:28:12.063 13151-13151 DEBUG                   pid-13151                            A        #18 pc<PHONE_NUMBER>060350  [anon:dalvik-classes33.dex extracted in memory from /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!classes33.dex] (livekit.org.webrtc.NetworkMonitor$2.onNetworkConnect+0)
2024-04-18 12:28:12.063 13151-13151 DEBUG                   pid-13151                            A        #23 pc<PHONE_NUMBER>060fb8  [anon:dalvik-classes33.dex extracted in memory from /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!classes33.dex] (livekit.org.webrtc.NetworkMonitorAutoDetect$SimpleNetworkCallback.onNetworkChanged+0)
2024-04-18 12:28:12.063 13151-13151 DEBUG                   pid-13151                            A        #28 pc<PHONE_NUMBER>060dac  [anon:dalvik-classes33.dex extracted in memory from /data/app/~~lFNIYzULyDLD2VfRjyfeiA==/com.bambuser.onetoone.dev-QgpzWBLO_PUuaPsu0lnOkw==/base.apk!classes33.dex] (livekit.org.webrtc.NetworkMonitorAutoDetect$SimpleNetworkCallback.onCapabilitiesChanged+0)

Which version of LiveKit specifically is being used here? 2.0.0? Can you also check which version of io.github.webrtc-sdk is being used (run gradle dependencies).

implementation "io.livekit:livekit-android:2.0.0"
io.github.webrtc-sdk:android-prefixed:114.5735.07@aar

How often would you say the error occurs, percentage wise?

Wait, you wrote a buildPeerConnectionFactory method, but how does that PeerConnectionFactory get injected into our LiveKit SDK?

On my Pixel 6 android 14, it is happening often. I did about 6 test calls, it crashed twice. The crash is a bit confusing because it happens a few seconds after I end a call. so once it crashed after a call, the other time it crashed right after I started a new one, but I am assuming it is because I ended a call and started a new one very fast

private val peerConnectionFactory by lazy { buildPeerConnectionFactory() }
val localVideoSource: VideoSource = peerConnectionFactory.createVideoSource(false)
So I use that to create video sources. sorry about the confusion

Is there a reason why you're not going through LocalParticipant.createVideoTrack()? It sounds like the association between two separate PeerConnectionFactories are causing the issue.

I am sending video with custom VideoCapturer
private suspend fun startLocalVideo(
        context: Context
    ) {
        val localVideoSource = peerConnectionFactory.createVideoSource(false)
        val surfaceTextureHelper =
            SurfaceTextureHelper.create(Thread.currentThread().name, rootEglBase.eglBaseContext)
        videoCapturer = CustomVideoCapturer { ImageRepo.getInstance().getSegmentedImageForWebRTC() }
        videoCapturer?.let { videoCapturer ->
            (videoCapturer as VideoCapturer).initialize(
                surfaceTextureHelper,
                context,
                localVideoSource.capturerObserver
            )
            localVideoTrack =
                room?.localParticipant?.createVideoTrack(
                    RTCClient.LOCAL_TRACK_ID,
                    videoCapturer
                )
            localVideoTrack?.let { track ->
                room?.localParticipant?.publishVideoTrack(
                    track
                )
            }

            videoCapturer.startCapture(1280, 960, 24)
        }
    }

Is there a way I can get the capturerObserver from Livekit instead? or will i need to revert to 1.0 Livekit for now?

createVideoTrack has an overload to pass in your own custom video capturers. You should try that.

I am doing that :). the problem is my custom video capturer need access to capturerObserver to send frames. and I get that from PeerConnectionFactory. its all in the code above.

val localVideoSource = peerConnectionFactory.createVideoSource(false)
(videoCapturer as VideoCapturer).initialize(
                surfaceTextureHelper,
                context,
                localVideoSource.capturerObserver
            )

    @Synchronized
    override fun initialize(
        surfaceTextureHelper: SurfaceTextureHelper,
        context: Context,
        capturerObserver: CapturerObserver
    ) {
        checkNotDisposed()

        this.context = context
        this.capturerObserver = capturerObserver
        this.surfaceTextureHelper = surfaceTextureHelper
    }

// and later
 capturerObserver?.onFrameCaptured(videoFrame)

LiveKit handles calling VideoCapturer.initialize with the appropriate capturerObserver from our internal PeerConnectionFactory. You don't need to call it yourself.

Hmm, I am a bit confused at so what my CustomVideoCapturer is supposed to do then. I have a function on it called trySendToServer that I call with a delay of 42 millis. there I fetch a bitmap from my ImageRepository, I do some GLES20/yuvConverter stuff and the part that I understand as crucial is                         capturerObserver?.onFrameCaptured(videoFrame)
Do you maybe have an example of how I can send my own frames with my custom video capturer?

The point is you can pass in your CustomVideoCapturer to createVideoTrack without calling initialize.
We will call initialize for you and supply you with the capture observer to use.
Basically, just remove your own creation of the video source and calling of initialize.

I understand, thank you


Adding ConfigContext to DeviceType

Environment

Python version:  3.5.2
NetBox version:  2.6.1

Proposed Functionality
Please add ConfigContext to DeviceType that would appear in the nested Device.config_context.
Use Case
Often there are certain models of equipment that have certain attributes that apply to all devices of that model but not necessarily to all of that Platform/Role. Adding ConfigContext will allow that information to be accessible when querying for a list of Devices without additional API calls to get Custom Field data for each type.
Database Changes
class ConfigContext(models.Model):
...
    device_types = models.ManyToManyField(
        to='dcim.DeviceType',
        related_name='+',
        blank=True
    )

External Dependencies
None.

Device types define physical hardware. Config context data has no relevance to them.

If that's the case, then can Custom Fields be exposed on the Device list API?


Development(extracted vCO Package file)

vCO package file already extracted and updated.

apologies for the wrong pull request i created.


Additional tooling access methods, trait impls, and a bug fix

This PR includes a variety of small fixes and improvements that I found myself needing as I went to upgrade the ion-cli to HEAD. I'd like to include these in the v1.0-rc4 release. I have completed the ion-cli upgrade in my local workspace using this branch, so I'm reasonably confident that these are the last items I need to get in.

Renames the RawReaderType enum to IonEncoding because it is generally useful in other contexts too.
Adds the ability for system and raw readers using AnyEncoding to report the encoding they're currently using.
Adds value_span and annotations_span methods to LazyValue. It already had a span method whose bytes included both the annotations and the value.
The binary 1.0 and 1.1 writers will now write f64s as f32s to save space when it can be done losslessly.
Adds WriteAsIon implementations for LazyList, LazySExp, and LazyStruct.
Adds IntoIterator impls for LazyList, LazySExp and LazyStruct. Previously, they only existed for borrowed (&) references to those types.
Fixes a bug in the StreamingRawReader that could cause a value's annotations span to be overwritten if reading that value consumed all of the data remaining in the buffer.
Renames the feature-gated LazyValue::lower method (which returns a LazyExpandedValue) to LazyValue::expanded() so I could add a raw() method alongside it that returns the underlying LazyRawValue when applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I noticed there were some functions where you removed the feature gate, and others where you kept the feature gate. How did you decide for which functions to remove vs retain the feature gate?

Some methods are on a fuzzy line between being reader-writer and tooling-apis. For example, LazyExpandedValue::context returns the EncodingContextRef used to decode that value. It's definitely part of the reader-writer API, but whether it's specifically tooling-access is debatable.
I decided I was ok with removing the feature gate from methods like that because they're still behind some feature gate, so either way there's no guarantee of stability being made.


A question about Cascade R-CNN code details

According to the code below, and as far as I comprehended, roi_labels states the GT labels that are assigned to the proposals.
https://github.com/open-mmlab/mmdetection/blob/374d4c0bed3e3c01e2988213b54310500479e3df/mmdet/models/roi_heads/cascade_roi_head.py#L268
However, these codes below give some changes to the roi_labels. If the GT label is bg, it is substituted with the most probable class label that is not bg.
https://github.com/open-mmlab/mmdetection/blob/374d4c0bed3e3c01e2988213b54310500479e3df/mmdet/models/roi_heads/cascade_roi_head.py#L279-L281
What is the point of doing this? Shouldn't Groundtruth labels be given any changes?

@vombategeht This is because the training process is dynamic. For the background proposal, it may become a positive sample after training. At this time, it needs to be dynamically adjusted to a positive sample.

Thanks for your response!
But unfortunately I still don't get why Faster R-CNN does not apply the same technique. Isn't it same for Faster R-CNN that bg proposals might become positive samples after training?
I also wonder bbox_results['bbox_targets'][0] are GT labels or not. I thought they are, but now I'm confused. Are they predicted labels for bounding boxes or unchangeable GT labels?

@vombategeht This is the refining process.


What about constructors?

Rust does not have a built-in feature for constructors, but the norm is to define an associated function new(...) for that type.
Dart has constructors as a feature. One can define up to one constructor and multiple named constructors. (A factory acts like a static method, returning an instance of a class, but has the same Syntax as a constructor to the class user)
Currently, on the Dart side, there is a constructor being generated that initializes all fields on the Dart side. But sometimes, the constructor on the Rust side is mandatory.
Solution
If a constructor on the Rust side should be used, generate a factory constructor that calls that constructor and do not expose the default constructor to the public API, if the struct does not have attribute #[frb(default_constructor)].
What should be considered a constructor?

An associated function should be considered a constructor,

if it does not have attribute #[frb(no_constructor)] and
if its signature is pub fn new(...) -> Self, where Self can also be the struct type, or
if it returns Self or the struct type and has attribute #[frb(constructor(default))]

An associated function should be considered a named constructor,

if it does not have attribute #[frb(no_constructor)] and
if it returns Self or the struct type and its name is not new. The name of the constructor should be

<name>, if it has attribute #[frb(constructor("<name>"))], or
the name of the function, otherwise

if its signature is pub fn new(...) -> Self, where Self can also be the struct type and it has attribute #[frb(constructor("<name>"))], where the name of the constructor is <name>.

Alternatives
Try to ignore the Dart constructor and only call MyStruct.newMyStruct(...). This is bad, if you have a public API
Additional context

Example
pub struct Point {
    pub x: f64,
    pub y: f64,
}

impl Point {
    // Constructor
    pub fn new(x: f64, y: f64) -> Point {
        Self { x, y }
    }

    // Named constructor
    pub fn all(v: f64) -> Point {
        Self { x: v, y: v }
    }

    // Renamed named constructor
    #[frb(constructor("with_magnitude"))]
    pub fn with_length(x: f64, y: f64, length: f64) -> Point {
        let scale = length / (x * x + y * y).sqrt();
        Self {
            x: x * scale,
            y: y * scale,
        }
    }

    // Not a constructor
    #[frb(no_constructor)]
    pub fn add(a: &Point, b: &Point) -> Point {
        Self {
            x: a.x + b.x,
            y: a.y + b.y,
        }
    }
}

Totally agree that MyStruct.newMyStruct is quite suboptimal!
However, as for the constructor, there is one problem: Dart requires any dart constructor to be synchronous. Thus, if we have #[frb(sync)] fn new() -> Self, it is easy to be a Dart constructor; but if it is just fn new() -> Self, then we cannot :( What do you think?
Btw, just a small tweak about the conditions you suggest: We may not consider all non-method be constructors, since it may just be a normal static method. But we can allow users to explicitly make it a constructor when using something like frb(constructor)

However, as for the constructor, there is one problem: Dart requires any dart constructor to be synchronous. Thus, if we have #[frb(sync)] fn new() -> Self, it is easy to be a Dart constructor; but if it is just fn new() -> Self, then we cannot :( What do you think?

Oh yea... I totally overlooked that! 🤦♂ However, I think it would be fine to implicitly mark constructors as sync, since constructors are rarely that complicated that the extra overhead of calling it asynchronously is even worth it. But if a constructor is marked with #[frb(async)], you could generate a static method (acting like a constructor) instead of a factory constructor, that returns a Future<Self> and, in the case of an unnamed constructor, is named new.
Alternatively, if you decide, that this is an antipattern, I think it would also be fine, if there is the added requirement, that a constructor must be marked with #[frb(sync)], otherwise generate a static method.

I am a bit worried about making it sync by default, since that may make users confused. The explicit frb(sync) LGTM, but that is a bit verbose... So yes, I am also hesitate among the two choices!

If we use option 1, we can unify the DX a little bit more by adding:

[ ] When using getters (#[frb(getter)]), automatically make them sync as well. (Since getters are usually synchronous)


Improve workflow

#12  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @s-cork and the rest of your teammates on  Graphite


Support ^ (jump to the first non-blank character of the line)

This plugin supports ^ now. Please update this plugin.

what about g_ it is more or less the same thing as ^ but it goes to the last non white space char


Limit style choice

Coverage remained the same when pulling e9ddb1daf138643341291de18809e0ca0bc7b9cc on limit-style-choice into b9bada9f28f1b6c6c4da1c6a87ec3219fab7a559 on master.


🖐 When is image (re)pulled by dagger-engine

Dagger Cloud run URL
docker-container://dagger-engine
What happened? What did you expect to happen?
I tried some examples from dagger.io.
And for me is unclear in what cases the image is repulled by the dagger-engine.
Manually pulled images seem to be ignored?
The problem is even more critical if you imagine kubernetes pod running the dagger-engine. Because there will be no info after restart of pod. Mounted docker.sock should have wished effect... but If it will repull, dagger is not of any use...

@Pymann, from what I understand, you've been:

Pulling an image in Docker using the docker CLI
Referencing the same image in a Dagger pipeline
Realizing that the pipeline would download the Dagger image even though you have it in locally in Docker

If this is what you meant, it doesn't work that way: docker's cache and dagger's cache are not related, even when you mount the socket.

The problem is even more critical if you imagine kubernetes pod running the dagger-engine. Because there will be no info after restart of pod. Mounted docker.sock should have wished effect... but If it will repull, dagger is not of any use...

Regarding the effect on Kubernetes pod, let's ping @jlongtine and @gerhard to see how they handled that use case

@Pymann I need to know a bit more about the architecture you're trying to use to comment. How are you trying to run the Dagger Engine in Kubernetes?


ModelMapper - mapping of empty object issue

I noticed a problem when mapping an object which contains a variable assigned to an object with null values, The mapped object contains mentioned variable set to null.
I know this is difficult to explain, therefore I managed to create very simple github repository to simulate the issue (with failing test).
https://github.com/samo717/modelMapperSim

Thanks for reporting this issue, it's conflict to the unit test GH6.
But your scenario makes sense to me, fixed by #601.


Grey boxes / shadows around blockquotes (Mac Preview)

As described here: https://old.reddit.com/r/homebrewery/comments/ch3v0d/psa_grey_boxesshadows_around_notes_stat_blocks_etc/
This issue is still a thing.
OS: macOS 11.5.1
Chrome: 92
Seems like certain renderers like the macOS Preview App still have problems with the exported box-shadows.
Stupidly, if I replace the box-shadows with a drop-shadow filter, everything renders fine in every PDF App I have:
.phb blockquote {
	/* workaround for grey boxes around blockquotes */
	box-shadow: none;
	filter: drop-shadow(2px 4px 6px #888);
}

Same holds for other elements using box-shadows like elements with the class descriptive:
.phb .descriptive {
	box-shadow: none;
	filter: drop-shadow(0 0 6px #faf7ea);
}

This is a known bug in Mac Preview; it struggles a lot with partially transparent elements in a lot of cases. Unfortunately we do not have any affiliation with Mac to be able to fix the problem on their end.
The solution... Is to not use a buggy app to view your PDFs.
If you insist on using Preview, you can take a look at thia post for some workarounds:
https://www.reddit.com/r/homebrewery/comments/jx6e42/green_side_note_gets_an_ugly_blackish_border_once/?utm_medium=android_app&utm_source=share
Chrome is also working on a possible workaround but I wouldn't hold your breath.
https://bugs.chromium.org/p/chromium/issues/detail?id=995265

Sure. It's definitely a bug in Preview. Nevertheless, it can easily be fixed by changing every box-shadow to a drop-shadow. It's a fast and easy change and it won't make a difference visually.
The only issue I see that could arise when performing such a change could be an incompatibility to some existing "brews". "Brews" where authors already changed the visual appearance of the shadows by injecting own styles.

The only differences between box-shadow and filter:drop-shadow are no ability for "inset" shadows and one less distance parameter in drop-shadow.   Likely true that any place that uses box-shadow right now could instead use drop-shadow.  But haven't tested.  Couldn't be done in Legacy due to breaking brews, but v3.

@adroste There is pretty apparent visual difference between box shadow and drop shadow. Box shadow extends around the top and bottom, but drop shadow does not. This is not something we are going to change to accommodate someone else's bugs.
However, as you have noted, you can manually change the shadow behavior on a per-brew basis by adding custom CSS to your document. If you are satisfied with that appearance, you can make that adjustment on your own brews via the Style tab. That's what it's for after all! :+1:

Oddly, I don't get this issue on macOS 11.5, Preview 11, printing to PDF from Chrome.  Box-Shadow is just fine for me.
But why not make an adjustment for v3?  Drop-shadow can extend all the way around an element if the spread is larger than the offset drop-shadow(0 0 6px) for example.  Further, the current box-shadow doesn't extend all the way around the box due to it's offset...the first note is the default box-shadow, and the second has drop-shadow:

The only visible difference is that drop-shadow follows the contour of the alpha channel on the border image, where box-shadow does not (only follows the edge of the elements box).  Arguably, this is a point in favor of drop-shadow.
Changing it for legacy wouldn't work, but adusting for v3 before any brews exist would be a non issue....seems like the time to squash this issue.

Ehhhh... maybe for v3. It looks pretty close. How well does it work for our other shadowed elements? (monster stat blocks, etc.)

I was trying to think of other shadowed elements and forgot monster blocks, but I don't think there is anything else...?  Not class tables, not descriptive boxes....

@Gazook89 Descriptive text boxes also have a white-ish box shadow: box-shadow: 0 0 6px #faf7ea;

See PR #1577

Following the issues encountered in #1661 , we might need to undo the changes to V3 back to using box-shadows since drop-shadow makes text unselectable and corrupts image quality.

This actually seems to make things objectively worse. I'm also thinking we might want to go back and undo the change to V3 as well. Selectable text and good image quality in almost every PDF viewer is preferable to just catering to the weird edge case in Mac Preview.

I have undo the changes in my pull request : #2127

@calculuschild
Did some further research and it's true, the drop-shadow works because chrome will print the box as an image with transparency. This reduces the visual quality of the text and prevents selection. A fix suggested here (https://github.com/puppeteer/puppeteer/issues/5284#issuecomment-727826838) just applies a filter property document-wide resulting in a pdf consisting of screenshots 👎.
With all the introduced trouble in mind, I think there is a reasonable way to "properly" misuse the filter property. If you move the box-shadows to another layer (e.g. with absolute positioning) and apply a noop-filter, you end up with pdf where the shadow is overlayed as graphic with the border-image but the text content remains selectable text/vector.
.phb blockquote {
  box-shadow: none;
  position: relative;
}

.phb blockquote::after {
  content: '';
  position: absolute;
  /* border-width = 11px, could be set as css variable */
  left: -11px;
  right: -11px;
  top: -11px;
  bottom: -11px;
  box-shadow: 1px 4px 14px #888;
  /* force chrome to print shadow as graphic */
  -webkit-filter: opacity(1);
}

@adroste This is an interesting approach. Unfortunately, there are already a few cases where the after (and before) pseudoelements are already claimed for other purposes. We might be able to redesign those elements to make this work, but in the meantime I think we may need to go back to box-shadows with #2127, and direct users to instead report the issue to Apple so they can fix Preview.

The shadow must just be applied at another level. It must not be via ::before or ::after. It can be any element anywhere as long as you keep track of the positioning and border of the container-element you want to create a shadow for. This is especially easy if you apply the styling on a nested absolute positioned element with the container set to relative. This way the solution could be css only.
Apple will fix s***. I'm reporting bugs my whole life and they just don't care. There are certain security related issues that Apple introduced in macOS 12.0 that still are not fixed yet despite new daily reports from other community members. If something does not work in the Apple world you have to find a workaround yourself or ignore it.
When it comes to macOS/iOS/iPadOS Preview is the only 'good' PDF viewer. Acrobats rendering might be more correct, but the UI and performance is broken since Steve introduced Retina displays in 2012/2013. Additionally, the preview renderer is deeply integrated into every part of every OS: Safari on macOS, all web browsers on iOS/iPadOS, QuickLook, Finder, Files etc.
To sum it up, Homebrewery PDFs are broken on every Apple device. We all know it's Apples fault, but they will not fix it.
We can just try to make the Homebrewery rendering more compatible with simple hacks like the one mentioned above.

It can be any element anywhere as long as you keep track of the positioning and border of the container-element you want to create a shadow for.

Right, but then this involves introducing nested elements everywhere you want a shadow. Rather than
Content
, now the text editor becomes cluttered with

Content

. That's putting more burden Homebrewery users where before/after allow the shadow to be "built-in" to the element.

Absolutely. I just looked at #2127. If it's just about these 4 places, it should be fairly easy to implement and test:

The before pseudo element is not used for the note, descriptive text box and monster stat block.
The class-table decoration has a shadow directly on the before pseudo element to darken the background image. This shadow does not affect any text or content and therefore must not be changed. For the sake of consistency it could also safely be changed to a box-shadow + noop filter without a noticable effect.

Is this still an issue?  I noted way back when that I did not have the same problem with gray borders in Preview on my Mac that @adroste had, but I'm just coming back around to see what issues and PR's can be closed (this and #2127).
For my brews created in Chrome on MacOS, viewed in Preview....
With Drop-Shadow

With Box-Shadow

The filter:drop-shadow(); still prevents text selection.  So if box-shadow issues (gray boxing) are no longer present, then my vote is to switch back to box-shadow.

Just a note for future readers: another alternative is to just go back to using drop-shadow and unset the box-shadow.

Running into this bug -- using absolute latest MacOS 14.4.1 on absolute latest Macbook (M3-Max, a month old.)
You can read about the details on my reddit post.
I tried 'fetching' the PDF on Chrome on Windows 11, and it doesn't help -- the same troublesome box-shadow tag is present, and every MacOS PDF renderer still can't render it properly.
The point of my posting here is that the comment from @calculuschild is no longer true:  using the latest MacOS doesn't help, and using Windows doesn't help.  You may want to re-open this issue.

@Gazook89 can you post some CSS that shows a workaround -- e.g. how to go back to drop-shadow?  I'm not a CSS guy.  :-)

We have not found any evidence so far that this issue happens outside of mac preview, but if what you say is true this is a major problem.
Could you send a share link or code snippet with this issue? for testing purposes.
If we still find this issue, we will probably need to swap box shadow for a drop shadow, even though that in itself has other problems

If we still find this issue, we will probably need to swap box shadow for a drop shadow, even though that in itself has other problems

Just for the record, we will not be switching back todrop-shadow. We already tried that. While it fixes this one issue limited to a specific Preview bug, it creates other issues that appear on every platform.

Hey all, I have a minimal test case that showcases the Mac bug!
It's some sort of weird interaction between a {{ div }} block, and the {{ note }} and {{monster}} blocks.
Reproduction recipe:

Create a single page that has a div, followed by a note.  Example here.  It looks totally fine during live-rendering, on both Mac and Windows.  (In the example, the div block has some attributes, but this bug happens even with just a plain {{ ... }} div.)
'Get PDF' and download the PDF.
The resulting PDF looks broken in the absolute latest MacOS -- there's a grey block behind the note, blocking text.

Note that this bug above happens on both Mac and Windows Chrome.  In other words, the workaround of "just fetch the PDF from Windows Chrome" doesn't help here.

Note that this bug above happens on both Mac and Windows Chrome.

Windows 11 with Chrome 124:

@5e-Cleric I modified their comment for clarity. What they mean is, the PDF still appears broken on Mac, even if originally generated on Windows.

Windows 11 with Chrome 124:

What he is saying is that if you were to upload your Windows-generated PDF, and I downloaded it to my Mac, the issue would be apparent to me even though you generated it on Windows.
So obviously it's a Mac bug.

So obviously it's a Mac bug.

Yes. The PDF is generated correctly in either case (I can view his Mac-generated PDF on Windows and it looks fine), but Mac fails to render the shadow. So specifically a Mac viewing issue.

That earlier comment stems from this discussion on Gitter:
https://app.gitter.im/#/room/#naturalcrit_Lobby:gitter.im/$BzzlvUIGmeBPFcdLuPqK60wGy-XmgI5bvRLO2xy2bKE
Could be we missed something there.
In any case, it looks like Google already narrowed down this issue in this comment. I don't think we can necessarily do anything about it. https://issues.chromium.org/issues/41476907#comment33

At this point, I'm trying to figure out how, after building a 20 page module in homebrewery, I can create a PDF that works everywhere.  I don't even know how to work around this. :-(   MacOS Preview isn't a 'niche' set of users.

Possibly you could run it through a pdf optimizer and get better result to share. I haven’t tried it

Hey all!  Pinging here again, wondering if I can persuade devs to reopen and work on this issue again. To recap:

Any time a {div} and a {note} are on the same page, the resulting PDF isn't rendered correctly by MacOS Preview (or any of the zillion things that depend on the MacOS-level PDF rendering code).
It doesn't matter what OS or browser the PDF in generated from; it always renders broken on Mac.
This doesn't seem like an obscure edge-case to me.  Mac-users aren't a teensy minority like they were 25 years ago. :-)

I completely agree that this is Apple's bug, not Homebrewery's.  But my gut says there must be some sort of HTML/JS/CSS workaround that can be inserted temporarily, just as you did before.
I'm a software dev myself and have run many open source projects, so of course I know how ridiculous it is that anyone "demand" that a bug be fixed.  I should submit a patch myself!  But sadly, I'm a backend engineer, and my knowledge of JS/CSS is pretty rudimentary.  :-(    That said, let me know if there's anything I can do to help.  Maybe I can lend one of you a Mac for testing?  :-)

@sussman This is a Mac issue, the only thing we could do in our own is to delete shadows entirely. This issue will not be fixed by us, you can fix it by yourself changing the box shadows to filter drop shadows (i think) but we won't do that fix because it causes other minor issues.
This issue will remain closed until a new option is available.

Thanks for the explanation.  Can you explain to me how I can change box shadows to filter drop shadows?  Or point to an FAQ perhaps?  (I'm also curious what sort of minor issues it causes!)

Thanks for the explanation. Can you explain to me how I can change box shadows to filter drop shadows? Or point to an FAQ perhaps? (I'm also curious what sort of minor issues it causes!)

Look at this PR for the necessary changes and the issues that arise because of them:
https://github.com/naturalcrit/homebrewery/pull/1661

Just adding to this old issue that another user on Discord is bumping into same issue.  Working from their brew (v3, uses box-shadow),

Generate PDF from Mac Firefox 131.0, view in Preview 11:  No shadow problem
Generate PDF from Mac Chrome 128.0, view in Preview 11:  shadow problem

Viewed both versions in browser, no problem.

Perhaps we could put a toggle in the preview toolbar to indicate user is using MacOS, which then switches the styles from one type of shadow to another.


added unit tests for Indent and Shape

I think having more unit tests will make it easier to spot regression, solve bug, etc, because when the test failed, it's closer to the components being tested. It will also help people understand how a particular function works.
This PR is just an initial work toward that, and to get feedback on the idea.

Thanks for the PR!
I'm generally in favour of unit tests. I think it is good to pick the things we test though - I'd prefer a smaller number of tests testing trickier things, rather than a tonne of tests testing every last function.


Unhandled Exception

I always get this error, When I am trying to use it with C#. :
Unhandled Exception: Cannot print exception string because Exception.ToString() failed.

Can you please provide minimal example that reproduce behaviour that you described?

Thanks.
For example, I used this code from https://fslab.org/Deedle/csharpseries.html
var numNames = new SeriesBuilder<int, string>() {   { 1, "one" }, { 2, "two" }, { 3, "three" } }.Series; numNames.Print();
// Debugging...
Unhandled Exception:
Cannot print exception string because Exception.ToString() failed.
I don't know if it is because I am using Visual Studio 2019 and the newest version of .NetFrame or .NetCore.

Okay, I see. You have additionally to add package FSharp.Core 4.5 to your project:
    <PackageReference Include="FSharp.Core" Version="4.5.0" />

Thank you for the answer.

Sure. Anyway I think it should be explicit dependency

@FoggyFinder Thanks for pointing out the dependency. I’ll add it explicitly to nuget next time when we publish next version.

@liugaocn Deedle 2.1.0 has been published with explicit dependency on FSharp.Core. You do not need to add FSharp.Core manually anymore.


Kinesis consumer returns StreamName should not be empty

Problem
After upgrading to v3.0.9 using readFromKinesisStream  with the default FanOut retrieval mode returns the following error. The streamName in KinesisConsumerSettings is set properly.
[info] ERROR 2021-03-01 15:09:49,630 s.a.kinesis.coordinator.Scheduler - Worker.run caught exception, sleeping for 1000 milli seconds!
[info] java.lang.NullPointerException: StreamName should not be empty
[info] 	at software.amazon.awssdk.utils.Validate.notEmpty(Validate.java:290)
[info] 	at software.amazon.kinesis.common.StreamIdentifier.singleStreamInstance(StreamIdentifier.java:85)
[info] 	at software.amazon.kinesis.retrieval.fanout.FanOutRetrievalFactory.createGetRecordsCache(FanOutRetrievalFactory.java:63)
[info] 	at software.amazon.kinesis.coordinator.Scheduler.buildConsumer(Scheduler.java:915)
[info] 	at software.amazon.kinesis.coordinator.Scheduler.createOrGetShardConsumer(Scheduler.java:890)
[info] 	at software.amazon.kinesis.coordinator.Scheduler.runProcessLoop(Scheduler.java:416)
[info] 	at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:325)
[info] 	at fs2.aws.kinesis.consumer$.$anonfun$readChunksFromKinesisStream$7(consumer.scala:239)

The application tries to read from one stream which has one shard.
Switching to Polling works fine.

Hi. I'm sorry for this bug, we introduced it when we adopted the integration with LocalStack
Please try version 3.0.11 i just released with fix
Also consider using https://github.com/laserdisc-io/fs2-aws/blob/master/fs2-aws/src/main/scala/fs2/aws/kinesis/Kinesis.scala
the usage example https://github.com/laserdisc-io/fs2-aws/blob/master/fs2-aws-examples/src/main/scala/KinesisExample.scala
I'm planning to open the Discussion to compare these 2 APIs and choose one of them at the end

i would love encourage you to participate in this discussion https://github.com/laserdisc-io/fs2-aws/discussions/607

Thank you for fixing this so quickly. I will upgrade and try the new implementation as well.


[Paging] In absence of next operation, next HTTP verb should be GET

Today, if the initial call is POST and there is no next operation, we use the same verb (POST again). We discussed in Teams that it should be GET.

And yes a test in testserver would be awesome :p


[bug] neighbors not appearing on boot, disappearing over time

Version
Checklist:

[x] I am not using Home Assistant. Or: a developer has told me to come here.
[x] I have checked the troubleshooting section and my problem is not described there.
[x] I have read the changelog and my problem is not mentioned there.

Build/Run method

[x] Docker
[ ] PKG
[ ] Snap package
[ ] Manually built (git clone - yarn install - yarn run build )

Zwavejs2Mqtt version: 6.1.1
Z-Wave JS version: 8.9.0-beta.1
Describe the bug
Immediately after rebooting none of my zwave devices have neighbors. Including the controller. Sometimes, if I manually navigate to the network graph, a refresh of the neighbors will be triggered and then the neighbors will show up. But if I start Zwave2Mqtt and do not navigate to the network graph UI the neighbors are never populated. When I am able to get the neighbors to populate, they eventually disappear over time and outbound messages to my zwave devices stop working.
To Reproduce
Steps to reproduce the behavior:

Restart the Docker container. Observe that the controller's neighbors, and all other device neighbors, are missing.
Navigate to Network Graph UI. Observe that (sometimes) the neighbors are refreshed and populate. Sometimes they are not.
Wait 1-3 days.
Observe that neighbors disappear from the controller, and outbound messages to zwave devices stop working.

Expected behavior
Neighbors are available after reboot and do not disappear over time. Won't you be my neighbor?
Additional context
The result of this is that commands to my zwave devices eventually stop working. They typically work right after the Docker container has started, but somewhere between a few hours to a few days outbound commands to devices stop working.
Inbound data from zwave devices continue to work. I see updates from all of the devices.
Also, I've found that even if I am able to get the neighbors to populate, they do eventually disappear over time. So, even if I boot the server, go immediately into the UI, refresh the neighbors by going to the Network Graph, eventually a few hours/days later the neighbors disappear again.
Sometimes going to the Network Graph UI will trigger a neighbor refresh, but the neighbors do not update. This results in a "flat" network graph with no connections and no update to the node data.
Debugging steps I've attempted

Rebooting the server that host the Docker containers
Removing and re-inserting the zwave stick (Aeotec Gen5)
Backing up and re-flashing the zwave stick
Healing the zwave network

Log files
Below are the logs / node info from a reboot last night where I did not go to the control panel for 6+ hours after restart.
At the end of the logs I tried to go to the network graph which triggered an unsuccessful neighbor refresh.
zwavejs2mqtt kmwoley.zip

Update. After running for several hours, messages stopped being delivered to my zwave devices. It happened overnight around 10pm.
For example, triggering a binary switch from ZwaveJS2MQTT shows this in the ZwaveJS2MQTT log. Note that it never gets a return for the writeValue call. This is the last thing I see.
2021-12-18 06:01:07.714 DEBUG SOCKET: Event ZWAVE_API emitted to DNYdjCB8FT29jxMQAAAP
2021-12-18 06:01:07.715 INFO ZWAVE: Calling api writeValue with args: [
  {
    nodeId: 44,
    commandClass: 37,
    endpoint: 0,
    property: 'targetValue'
  },
  false,
  {},
  [length]: 3
]
2021-12-18 06:01:07.715 INFO ZWAVE: Writing false to 44-37-0-targetValue

However, there is nothing logged in the ZwaveJS logs corresponding to that action. It's like ZwaveJS never is called.
I dumped the node_dump.json - it looks like everyone has neighbors, but I noticed the controller has fewer neighbors now than in the dump I provided earlier.
nodes_dump - 6am before reboot.zip
After rebooting - I confirmed that the controller has no neighbors, navigated to the Network Graph to trigger the neighbor refresh, and then confirmed that the controller now shows neighbors.
I then confirmed that I can trigger the binary switch update.
ZWaveJS2MQTT Log:
2021-12-18 06:36:42.832 DEBUG SOCKET: Event ZWAVE_API emitted to Bd4egvrA7486cDL5AAAD
2021-12-18 06:36:42.833 INFO ZWAVE: Calling api writeValue with args: [
  {
    nodeId: 44,
    commandClass: 37,
    endpoint: 0,
    property: 'targetValue'
  },
  false,
  {},
  [length]: 3
]
2021-12-18 06:36:42.834 INFO ZWAVE: Writing false to 44-37-0-targetValue
2021-12-18 06:36:42.915 INFO ZWAVE: Node 44: value updated: 37-0-currentValue true => false
2021-12-18 06:36:42.917 INFO ZWAVE: Node 44: value updated: 37-0-targetValue true => false
2021-12-18 06:36:42.917 INFO ZWAVE: Success zwave api call writeValue true
2021-12-18 06:36:44.050 INFO ZWAVE: Node 44: value updated: 37-0-currentValue false => false

And the zwave log:
2021-12-18T14:36:42.857Z SERIAL » 0x010a00132c03250100251ad2                                          (12 bytes)
2021-12-18T14:36:42.858Z DRIVER » [Node 044] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      26
                                  └─[BinarySwitchCCSet]
                                      target value: false
2021-12-18T14:36:42.860Z SERIAL « [ACK]                                                                   (0x06)
2021-12-18T14:36:42.866Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-12-18T14:36:42.867Z SERIAL » [ACK]                                                                   (0x06)
2021-12-18T14:36:42.868Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-12-18T14:36:42.908Z SERIAL « 0x010700131a000005f4                                                 (9 bytes)
2021-12-18T14:36:42.909Z SERIAL » [ACK]                                                                   (0x06)
2021-12-18T14:36:42.910Z DRIVER « [REQ] [SendData]
                                    callback id:     26
                                    transmit status: OK
2021-12-18T14:36:42.914Z CNTRLR   [Node 044] [~] [Binary Switch] currentValue: true => false        [Endpoint 0]
2021-12-18T14:36:43.923Z SERIAL » 0x010900132c022502251bd2                                            (11 bytes)
2021-12-18T14:36:43.924Z DRIVER » [Node 044] [REQ] [SendData]
                                  │ transmit options: 0x25
                                  │ callback id:      27
                                  └─[BinarySwitchCCGet]
2021-12-18T14:36:43.926Z SERIAL « [ACK]                                                                   (0x06)
2021-12-18T14:36:43.932Z SERIAL « 0x0104011301e8                                                       (6 bytes)
2021-12-18T14:36:43.933Z SERIAL » [ACK]                                                                   (0x06)
2021-12-18T14:36:43.933Z DRIVER « [RES] [SendData]
                                    was sent: true
2021-12-18T14:36:43.974Z SERIAL « 0x010700131b000005f5                                                 (9 bytes)
2021-12-18T14:36:43.975Z SERIAL » [ACK]                                                                   (0x06)
2021-12-18T14:36:43.975Z DRIVER « [REQ] [SendData]
                                    callback id:     27
                                    transmit status: OK
2021-12-18T14:36:44.047Z SERIAL « 0x01090004002c03250300fb                                            (11 bytes)
2021-12-18T14:36:44.049Z CNTRLR   [Node 044] [~] [Binary Switch] currentValue: false => false       [Endpoint 0]
2021-12-18T14:36:44.051Z SERIAL » [ACK]                                                                   (0x06)
2021-12-18T14:36:44.052Z DRIVER « [Node 044] [REQ] [ApplicationCommand]
                                  └─[BinarySwitchCCReport]
                                      current value: false

The next thing I'm trying is to remove the most recent zwave device I added to my network to see if that changes anything.

I have the exact behavior to. After a couple of days my network graph will look like this:

To get it to work again I need to restart the docker container or just go to ZWave Settings and press save. After aprox. one minute the network is up and running again.

I have EXACT same issues, I have Aeotec Gen 5, this started occurring around the time the below update might have occurred, other notes:

I can most often see neighbors but at some point they go away.
Works ok, perhaps slow after a server reboot - slowly degrades and looses touch w devices.
I node will report a change to HA and HA will update it in the UI, However - HA can't control a device.
Direct device updates from Z-wave JS DON'T update devices
Devices don't respond to PINGS
Yet devices report status to HA/UI/Z-Wave JS properly.

ll/Most z-wave functionality
zwavejs2mqtt: 6.1.1
zwave-js: 8.9.0-beta.1
home id:<PHONE_NUMBER>
home hex: 0xdefb762d

Does reverting to a previous version fixes the problem? Try with 6.0.3

I reverted to 6.0.1 with same problem. Did that test in the end of November.

no clue, I tried to check the nodes dump and I cannot find any issue, @AlCalzone ?

Might be worth a test with the latest beta.3
Neighbor information is reported by the stick though, so the problem might be hardware or firmware related. See #3905

One thing I've noticed is that I see a DRIVER » [REQ] [SendDataAbort] message around the time communication to the zwave devices fails. I ran two experiments where I repeatedly sent an on/off every 5 min to a binary switch and tested to see if it was received. Both times I saw that the switch stopped reacting to the commands after SendDataAbort.
Logs are attached for both of those experiments.
Experiment 1
Subject: Node 059
Failed after 2021-12-18T18:50 in the zwave log (10:50 in the zwavejs2mqqt)
2021-12-18T18:51:02.125Z DRIVER » [REQ] [SendDataAbort]
zwavejs2mqtt_2021-12-18-1235.log
zwavejs_2021-12-18-1235.log
Experiment 2
Subject: Node 059
Failed after 2021-12-19T03:01 in the zwave log (19:01 in the zwavejs2mqqt)
2021-12-19T03:08:12.528Z DRIVER » [REQ] [SendDataAbort]
zwavejs_2021-12-18-2124.log
zwavejs2mqtt_2021-12-18--2124.log
@AlCalzone - I'd be happy to try the latest beta.3. I see these instructions for installing from github. I'm running the zwavejs2mqqt docker container. Can you point me to instructions on how to get beta.3 installed into the docker container?
I'm on an Aeotec Gen5 stick - I don't think the 700 series issues apply.

Is this related to #2081 ?

Can you point me to instructions on how to get beta.3 installed into the docker container?

@robertsLando ?

Use master tag instead of latest

Super - thanks. Testing this now. It may take a few days to see if it makes
a difference.
On Tue, Dec 21, 2021 at 1:15 AM Daniel Lando @.***>
wrote:

Use master tag instead of latest
—
Reply to this email directly, view it on GitHub
https://github.com/zwave-js/zwavejs2mqtt/issues/2074#issuecomment-998604461,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOWG6GGVCBOZDT7JN7LG2TUSBATRANCNFSM5KJMY4EQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

Closing for now, if the issue persists feel free to reopen

@AlCalzone - it does not look like beta.3 has resolved this issue. It may have gotten better (I'm seeing longer times between failure) but the issues described in this bug still persist.
Logs attached. These logs show about 7 hours between the first boot of docker with beta.3 and 2021-12-21T19:01:41.659Z DRIVER » [REQ] [SendDataAbort], after which commands to the devices stopped responding.
zwavejs_2021-12-21.log.gz
I rebooted and have been running now for 36 hours without issue. I didn't change anything besides rebooting. I'll keep monitoring for failure, but given that I've seen it fail once already makes me think it's not truly fixed.
@robertsLando - I doesn't look like I have permissions to reopen this issue. Can you please do so? Or should this bug be filed on the node-zwave-js project?

Hey folks - checking back in here. I ran for a full week without issues. Thanks for your efforts on all parts of this stack - your efforts are greatly appreciated @robertsLando & @AlCalzone !


str:[n] property does not get truncated, instead schema gets changed

Expected behavior and actual behavior.
In my schema I set the property "twochars" to a str type with a maximum width of 2. http://toblerity.org/fiona/manual.html#field-types says "If this value is used in the schema of a file opened for writing, values of that property will be truncated at [2] characters." Instead I see the schema of the newly written file being adapted to whatever string length a feature might have had.
Steps to reproduce the problem.
Here is minimal example code.
import fiona
from shapely.geometry import *
from fiona.crs import from_epsg
from fiona import prop_width
import tempfile
import os.path

feature = {
    "geometry": mapping(Point(0,0)),
    "properties": {"twochars": "ABCDEFG"}
}

print("Feature property: {}".format(feature['properties']['twochars']))

schema = {
    "geometry": "Point",
    "properties": {"twochars": "str:2"}
}
print("Schema property: {}".format(schema['properties']['twochars']))
print("Schema property width: {}".format(prop_width(schema['properties']['twochars'])))

driver = "ESRI Shapefile"
extension = ".shp"

with tempfile.TemporaryDirectory() as directory:
    temporary_file = os.path.join(directory, "file"+extension)

    with fiona.open(temporary_file, "w", driver=driver, schema=schema, crs=from_epsg(4326)) as f:
        f.write(feature)

    with fiona.open(temporary_file) as f:
        print("File's schema property: {}".format(f.schema['properties']['twochars']))
        print("File's Schema property width: {}".format(prop_width(f.schema['properties']['twochars'])))
        for feature in f:
            print("File's feature property: {}".format(feature['properties']['twochars']))

From my understanding it should print
Feature property: ABCDEFG
Schema property: str:2
Schema property width: 2
File's schema property: str:2
File's Schema property width: 2
File's feature property: AB

with the "ABCDEFG" being truncated to just two characters, "AB", but instead it prints
Feature property: ABCDEFG
Schema property: str:2
Schema property width: 2
File's schema property: str:7
File's Schema property width: 7
File's feature property: ABCDEFG

Operating system
Archlinux
Fiona version and provenance
Fiona-1.7.9.post1-cp36-cp36m-manylinux1_x86_64.whl from pip.

This is clearly a bug. Thanks for the report.

@kannes it is not so clearly a bug after all – it is a GDAL feature. See http://www.gdal.org/drv_shapefile.html

Starting with GDAL/OGR 1.10, the driver knows to auto-extend string and integer fields (up to the 255 bytes limit imposed by the DBF format) to dynamically accommodate for the length of the data to be inserted.

There's nothing we can do about this now in Fiona. It looks like the work around is to use a more strict format driver.

Dang! Well, I guess it is safe to say that people who use Shapefiles won't care about attribute consistency too much anyways >:)
Thanks for looking into it!


Draft: Replace Mutual TLS pinning with TLS binding authentication

Pending:

[ ] Remove channel from existing interfaces (SendSubmit, SendConsensus)
[x] Implement Server logic of Auth Service
[ ] Define signer for signature
[ ] Add new unit testcase and update existing unit testcases

I merged this PR changes into PR #7. Hence closing this one.


Pass in 'opts' when call opts.loading.finished() in destroy()

This would make it easier for user who uses custom the opts.loading.finished to access 'opts' props (like opts.state).
On the other hand, I found the other places -- where opts.loading.finished() is called -- already pass in 'opts' as the first argument. So, this code change would still make the argument list consistent to existing one.

Infinite Scroll v3 is out!. It's an all new re-write, so take a look at how to upgrade from v2.  With the brand new release, this contribution is no longer applicable.
Thank you for your interest and effort in Infinite Scroll. Sorry I had to pass on this one. 🌈🐻


compressed images tarball support

Is your feature request related to a problem? Please describe.
We currently support the /var/lib/rancher/k3s/agent/images drop-in directory for pre-loading images. This is of key importance to supporting air-gapped installations where tarballs can be quite large and in the case of RKE2 must be compressed to be uploaded as a release artifact (otherwise we will exceed the 2GB upload limit for individual files to GH releases). As currying such large, uncompressed tarballs is less-than-ideal, it would be handy if k3s/rke2 could detect and decompress tarballs in the agent images drop-in directory.
Describe the solution you'd like
On start-up, prior to spinning up any subprocesses and therefore prior to loading images into containerd, attempt to decompress tarballs in-place. This would amount to a one-time cost per drop-in tarball at start-up.
Describe alternatives you've considered
Currying uncompressed tarballs and/or decompressing as part of the drop-in installation process.
Additional context
We should support the most common compression algorithms:

bzip2
gzip
lz4

We should also log at INFO level every file encountered in the images drop-in directory and indicate whether they will be skipped or processed.
Additionally, when processing files we should not assume that we own anything we encounter. This means that we may need to setup a k3s-local data directory for recording the hashes of tarballs we have successfully processed (to prevent re-processing at every start).

@fkautz it would be great if also add support for zstr or least xz for maximum compression

maybe with
https://godoc.org/github.com/DataDog/zstd
or
https://godoc.org/github.com/klauspost/compress/zstd

Validated using commitid 077bacc9fcbfd3e7713f44d7fa4104d2b0a542b2

4 types working: tar, tar.gz, tar.bz2, tar.lz4
Images are shown properly when executing sudo k3s crictl images:

$ sudo ls /var/lib/rancher/k3s/agent/images/
alpine.tar.lz4	busybox.tar.gz	multiport.tar  ubuntu.tar.bz2

$ sudo k3s crictl images
IMAGE                                      TAG                 IMAGE ID            SIZE
docker.io/library/alpine                   latest              a24bb4013296f       5.85MB
docker.io/library/busybox                  latest              6858809bf669c       1.45MB
docker.io/library/ubuntu                   latest              4e2eef94cd6b9       76.3MB
docker.io/maxross/multiport                testing             5e3c24369c24c       366MB
...

Confirmed there are ImageCreate and ImageUpdate events in the containerd.log file to extracting these images
The images don't appear to add multiple times upon stopping and restarting k3s or re-running install scripts
@dweomer I don't see any logs mentioning every file encountered. Also I'm not sure we created a k3s-local-data dir at all for this. Are these both still required as part of this issue?

@rancher-max no, consider those possible future enhancements. If the base functionality works that's good enough.


Question on performance

hi，thanks for your work.
I have obtained bad performance on MMT and SpCL by only using your code and default setting .
For instance,
MMT(duke-to-market):
************************* Start validating market1501 on epoch 49 *************************n
Val: [ 0/18]	Time 0.997 (0.997)	Data 0.353 (0.353)
Val: [10/18]	Time 0.407 (0.493)	Data 0.000 (0.032)
Mean AP: 4.2%
CMC Scores:
top-1           6.1%
top-5          15.2%
top-10         21.7%
Validating time:  0:00:22.068357
SpCL (unsupervised on market1501):
************************* Start validating market1501 on epoch 49 *************************n
Val: [ 0/18]	Time 0.475 (0.475)	Data 0.299 (0.299)
Val: [10/18]	Time 0.100 (0.147)	Data 0.000 (0.027)
Mean AP: 15.8%
CMC Scores:
top-1          24.0%
top-5          44.2%
top-10         56.2%
Validating time:  0:00:14.673135
I suspect that I have overlooked something important. Can you give me a advise?

That's weird. Could you please provide the config files you used?

SpCL(market-to-duke):
Args:Namespace(config='SpCL/config.yaml', launcher='none', resume_from=None, set_cfgs=['TRAIN.val_freq', '10'], tcp_port='5017', work_dir='config')
cfg.LOCAL_RANK: 0
cfg.DATA_ROOT: /open-reid/datasets
cfg.LOGS_ROOT: /data//open-reid/logs
cfg.MODEL = edict()
cfg.MODEL.backbone: resnet50
cfg.MODEL.pooling: gem
cfg.MODEL.embed_feat: 0
cfg.MODEL.dropout: 0.0
cfg.MODEL.dsbn: True
cfg.MODEL.sync_bn: True
cfg.MODEL.samples_per_bn: 16
cfg.MODEL.mean_net: False
cfg.MODEL.alpha: 0.999
cfg.MODEL.imagenet_pretrained: True
cfg.MODEL.source_pretrained: None
cfg.DATA = edict()
cfg.DATA.height: 256
cfg.DATA.width: 128
cfg.DATA.norm_mean: [0.485, 0.456, 0.406]
cfg.DATA.norm_std: [0.229, 0.224, 0.225]
cfg.DATA.TRAIN = edict()
cfg.DATA.TRAIN.is_autoaug: False
cfg.DATA.TRAIN.is_flip: True
cfg.DATA.TRAIN.flip_prob: 0.5
cfg.DATA.TRAIN.is_pad: True
cfg.DATA.TRAIN.pad_size: 10
cfg.DATA.TRAIN.is_blur: False
cfg.DATA.TRAIN.blur_prob: 0.5
cfg.DATA.TRAIN.is_erase: True
cfg.DATA.TRAIN.erase_prob: 0.5
cfg.DATA.TRAIN.is_mutual_transform: False
cfg.DATA.TRAIN.mutual_times: 2
cfg.TRAIN = edict()
cfg.TRAIN.seed: 1
cfg.TRAIN.deterministic: True
cfg.TRAIN.datasets = edict()
cfg.TRAIN.datasets.market1501: trainval
cfg.TRAIN.datasets.dukemtmcreid: trainval
cfg.TRAIN.unsup_dataset_indexes: [1]
cfg.TRAIN.epochs: 50
cfg.TRAIN.iters: 400
cfg.TRAIN.LOSS = edict()
cfg.TRAIN.LOSS.losses = edict()
cfg.TRAIN.LOSS.losses.hybrid_memory: 1.0
cfg.TRAIN.LOSS.temp: 0.05
cfg.TRAIN.LOSS.momentum: 0.2
cfg.TRAIN.val_dataset: dukemtmcreid
cfg.TRAIN.val_freq: 10
cfg.TRAIN.SAMPLER = edict()
cfg.TRAIN.SAMPLER.num_instances: 4
cfg.TRAIN.SAMPLER.is_shuffle: True
cfg.TRAIN.LOADER = edict()
cfg.TRAIN.LOADER.samples_per_gpu: 16
cfg.TRAIN.LOADER.workers_per_gpu: 2
cfg.TRAIN.PSEUDO_LABELS = edict()
cfg.TRAIN.PSEUDO_LABELS.freq: 1
cfg.TRAIN.PSEUDO_LABELS.use_outliers: True
cfg.TRAIN.PSEUDO_LABELS.norm_feat: True
cfg.TRAIN.PSEUDO_LABELS.norm_center: True
cfg.TRAIN.PSEUDO_LABELS.cluster: dbscan
cfg.TRAIN.PSEUDO_LABELS.eps: [0.58, 0.6, 0.62]
cfg.TRAIN.PSEUDO_LABELS.min_samples: 4
cfg.TRAIN.PSEUDO_LABELS.dist_metric: jaccard
cfg.TRAIN.PSEUDO_LABELS.k1: 30
cfg.TRAIN.PSEUDO_LABELS.k2: 6
cfg.TRAIN.PSEUDO_LABELS.search_type: 0
cfg.TRAIN.PSEUDO_LABELS.cluster_num: None
cfg.TRAIN.OPTIM = edict()
cfg.TRAIN.OPTIM.optim: adam
cfg.TRAIN.OPTIM.lr: 0.00035
cfg.TRAIN.OPTIM.weight_decay: 0.0005
cfg.TRAIN.SCHEDULER = edict()
cfg.TRAIN.SCHEDULER.lr_scheduler: single_step
cfg.TRAIN.SCHEDULER.stepsize: 20
cfg.TRAIN.SCHEDULER.gamma: 0.1
cfg.TEST = edict()
cfg.TEST.datasets: ['dukemtmcreid']
cfg.TEST.LOADER = edict()
cfg.TEST.LOADER.samples_per_gpu: 32
cfg.TEST.LOADER.workers_per_gpu: 2
cfg.TEST.dist_metric: euclidean
cfg.TEST.norm_feat: True
cfg.TEST.dist_cuda: True
cfg.TEST.rerank: False
cfg.TEST.search_type: 0
cfg.TEST.k1: 20
cfg.TEST.k2: 6
cfg.TEST.lambda_value: 0.3
cfg.launcher: none
cfg.tcp_port: 5017
cfg.work_dir: /open-reid/logs/config
cfg.total_gpus: 4
The training is in a un/semi-supervised manner with 2 dataset(s) (['market1501', 'dukemtmcreid']),
where ['dukemtmcreid'] have no labels.
=> Loaded trainval from Market1501
ids | # images | # cameras

751 |    12936 |         6

=> Loaded trainval from DukeMTMCreID
ids | # images | # cameras

  1 |    16522 |         8

Epoch: [49][390/400]	Time 0.304 (0.353)	hybrid_memory 4.683 (4.941)
************************* Start validating dukemtmcreid on epoch 49 *************************n
Val: [ 0/23]	Time 0.506 (0.506)	Data 0.317 (0.317)
Val: [10/23]	Time 0.140 (0.181)	Data 0.000 (0.029)
Val: [20/23]	Time 0.104 (0.149)	Data 0.000 (0.015)
Mean AP: 11.2%
CMC Scores:
top-1          16.3%
top-5          25.9%
top-10         33.0%
Validating time:  0:00:15.829343
************************* Finished validating *************************

Finished epoch  49  mAP: 11.2%  best: 11.6%

=> Loaded checkpoint '/open-reid/logs/config/model_best.pth'
=> Loaded query from DukeMTMCreID

ids | # images | # cameras

702 |     2228 |         8

=> Loaded gallery from DukeMTMCreID
ids | # images | # cameras

1110 |    17661 |         8
******************************* Start testing dukemtmcreid *******************************
Test: [  0/156]	Time 0.646 (0.646)	Data 0.346 (0.346)
Test: [ 10/156]	Time 0.155 (0.199)	Data 0.000 (0.032)
Test: [ 20/156]	Time 0.135 (0.175)	Data 0.000 (0.017)
Test: [ 30/156]	Time 0.160 (0.167)	Data 0.000 (0.012)
Test: [ 40/156]	Time 0.141 (0.164)	Data 0.000 (0.009)
Test: [ 50/156]	Time 0.134 (0.161)	Data 0.000 (0.007)
Test: [ 60/156]	Time 0.144 (0.158)	Data 0.000 (0.006)
Test: [ 70/156]	Time 0.145 (0.155)	Data 0.000 (0.005)
Test: [ 80/156]	Time 0.134 (0.153)	Data 0.000 (0.005)
Test: [ 90/156]	Time 0.157 (0.153)	Data 0.000 (0.004)
Test: [100/156]	Time 0.127 (0.152)	Data 0.000 (0.004)
Test: [110/156]	Time 0.146 (0.151)	Data 0.000 (0.003)
Test: [120/156]	Time 0.136 (0.150)	Data 0.000 (0.003)
Test: [130/156]	Time 0.136 (0.150)	Data 0.000 (0.003)
Test: [140/156]	Time 0.146 (0.149)	Data 0.000 (0.003)
Test: [150/156]	Time 0.138 (0.148)	Data 0.000 (0.003)
Mean AP: 5.3%
CMC Scores:
top-1           9.7%
top-5          19.3%
top-10         24.1%
Testing time:  0:00:38.719870
******************************* Finished testing *******************************

Hi,
The config looks normal, could you please provide the number of clusters before each epoch? You can achieve it by cat log.txt | grep Statistics.
By the way, I re-trained SpCL (unsupervised market1501) upon receiving your issue. I have just trained for 8 epochs, however, the first validation result looks normal:
 * Finished epoch   4  mAP: 34.5%  best: 34.5% *

It seems that your problem does not occur in my experiments. We need to figure your problem step by step, so first, please provide the number of clusters in your experiments.

You could also provide your validation results during training by cat log.txt | grep mAP

==> Statistics for dukemtmcreid on epoch 0: 429 clusters, 9747 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 1: 254 clusters, 10355 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 2: 201 clusters, 10515 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 3: 267 clusters, 10335 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 4: 300 clusters, 10050 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 5: 345 clusters, 9691 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 6: 373 clusters, 9547 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 7: 377 clusters, 9509 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 8: 401 clusters, 9552 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 9: 411 clusters, 9258 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 10: 379 clusters, 9408 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 11: 408 clusters, 9263 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 12: 397 clusters, 9340 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 13: 381 clusters, 9256 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 14: 395 clusters, 9300 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 15: 399 clusters, 9270 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 16: 387 clusters, 9417 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 17: 399 clusters, 9407 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 18: 375 clusters, 9605 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 19: 404 clusters, 9461 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 20: 379 clusters, 9608 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 21: 389 clusters, 9447 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 22: 390 clusters, 9400 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 23: 382 clusters, 9283 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 24: 416 clusters, 9287 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 25: 380 clusters, 9281 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 26: 379 clusters, 9266 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 27: 374 clusters, 9211 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 28: 375 clusters, 9326 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 29: 384 clusters, 9153 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 30: 374 clusters, 8984 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 31: 364 clusters, 9068 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 32: 368 clusters, 8995 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 33: 384 clusters, 8882 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 34: 373 clusters, 9063 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 35: 370 clusters, 8893 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 36: 387 clusters, 8783 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 37: 373 clusters, 8950 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 38: 362 clusters, 8891 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 39: 379 clusters, 8958 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 40: 357 clusters, 9065 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 41: 370 clusters, 8963 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 42: 381 clusters, 8998 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 43: 378 clusters, 9001 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 44: 378 clusters, 8926 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 45: 361 clusters, 8986 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 46: 348 clusters, 9006 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 47: 395 clusters, 9114 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 48: 383 clusters, 8940 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 49: 390 clusters, 8802 un-clustered instances, 0 unused instances

The number of clusters is abnormal. It seems the label noise has been amplificated. Is it the market->duke UDA experiment with SpCL? Let me try it.

Yes.  However, the number of clusters is abnormal, thus I also want to sure it again.  The config files I used is:
MODEL:
architecture
backbone: 'resnet50'
pooling: 'gem'
embed_feat: 0
dropout: 0.
dsbn: True
sync_bn: True
samples_per_bn: 16
mean_net: False
alpha: 0.999
pretraining
imagenet_pretrained: True
source_pretrained: null
DATA:
height: 256
width: 128
norm_mean: [0.485, 0.456, 0.406]
norm_std: [0.229, 0.224, 0.225]
TRAIN:
# augmentation
is_autoaug: False
is_flip: True
flip_prob: 0.5

is_pad: True
pad_size: 10

is_blur: False
blur_prob: 0.5

is_erase: True
erase_prob: 0.5

# dual augmentation for MMT
is_mutual_transform: False
mutual_times: 2

TRAIN:
seed: 1
deterministic: True
datasets
datasets: {'market1501': 'trainval', 'dukemtmcreid': 'trainval'}
unsup_dataset_indexes: [1,]
epochs: 50
iters: 400
LOSS:
losses: {'hybrid_memory': 1.,}
temp: 0.05
momentum: 0.2
validate
val_dataset: 'dukemtmcreid'
val_freq: 5
sampler
SAMPLER:
num_instances: 4
is_shuffle: True
data loader
LOADER:
samples_per_gpu: 16
workers_per_gpu: 2
pseudo labels
PSEUDO_LABELS:
freq: 1 # epochs
use_outliers: True
norm_feat: True
norm_center: True
cluster: 'dbscan'
eps: [0.58, 0.6, 0.62]
min_samples: 4 # for dbscan
dist_metric: 'jaccard'
k1: 30 # for jaccard distance
k2: 6 # for jaccard distance
search_type: 0 # 0,1,2 for GPU, 3 for CPU (work for faiss)
cluster_num: null

# cluster: 'kmeans'
# cluster_num: [500,]
# dist_cuda: True

optim
OPTIM:
optim: 'adam'
lr: 0.00035
weight_decay: 0.0005
SCHEDULER:
lr_scheduler: 'single_step'
stepsize: 20
gamma: 0.1
TEST:
datasets
datasets: ['dukemtmcreid',]
data loader
LOADER:
samples_per_gpu: 32
workers_per_gpu: 2
ranking setting
dist_metric: 'euclidean'
norm_feat: True
dist_cuda: True
post processing
rerank: False
search_type: 0 # 0,1,2 for GPU, 3 for CPU (work for faiss)
k1: 20
k2: 6
lambda_value: 0.3
Thank you for your help. I have a new understanding.

I cannot find any problem in your config file, and here is mine, you could double check:
DATA_ROOT: '../datasets/'
LOGS_ROOT: '../logs/'

MODEL:
  # architecture
  backbone: 'resnet50'
  pooling: 'gem'
  embed_feat: 0
  dropout: 0.

  dsbn: True

  sync_bn: True
  samples_per_bn: 16

  mean_net: False
  alpha: 0.999

  # pretraining
  imagenet_pretrained: True
  source_pretrained: null

DATA:

  height: 256
  width: 128
  norm_mean: [0.485, 0.456, 0.406]
  norm_std: [0.229, 0.224, 0.225]

  TRAIN:
    # augmentation
    is_autoaug: False

    is_flip: True
    flip_prob: 0.5

    is_pad: True
    pad_size: 10

    is_blur: False
    blur_prob: 0.5

    is_erase: True
    erase_prob: 0.5

    # dual augmentation for MMT
    is_mutual_transform: False
    mutual_times: 2

TRAIN:
  seed: 1
  deterministic: True
  # mixed precision training for PyTorch>=1.6
  amp: False

  # datasets
  # datasets: {'market1501': 'trainval',}
  datasets: {'market1501': 'trainval', 'dukemtmcreid': 'trainval'}
  unsup_dataset_indexes: [1,]

  epochs: 50
  iters: 400

  LOSS:
    losses: {'hybrid_memory': 1.,}
    temp: 0.05
    momentum: 0.2

  # validate
  val_dataset: 'dukemtmcreid'
  val_freq: 5

  # sampler
  SAMPLER:
    num_instances: 4
    is_shuffle: True

  # data loader
  LOADER:
    samples_per_gpu: 16
    workers_per_gpu: 2

  # pseudo labels
  PSEUDO_LABELS:
    freq: 1 # epochs
    use_outliers: True
    norm_feat: True
    norm_center: True

    cluster: 'dbscan'
    eps: [0.58, 0.6, 0.62]
    min_samples: 4 # for dbscan
    dist_metric: 'jaccard'
    k1: 30 # for jaccard distance
    k2: 6 # for jaccard distance
    search_type: 0 # 0,1,2 for GPU, 3 for CPU (work for faiss)
    cluster_num: null

    # cluster: 'kmeans'
    # cluster_num: [500,]
    # dist_cuda: True

  # optim
  OPTIM:
    optim: 'adam'
    lr: 0.00035
    weight_decay: 0.0005

  SCHEDULER:
    lr_scheduler: 'single_step'
    stepsize: 20
    gamma: 0.1

TEST:

  # datasets
  datasets: ['dukemtmcreid',]

  # data loader
  LOADER:
    samples_per_gpu: 32
    workers_per_gpu: 2

  # ranking setting
  dist_metric: 'euclidean'
  norm_feat: True
  dist_cuda: True

  # post processing
  rerank: False
  search_type: 0 # 0,1,2 for GPU, 3 for CPU (work for faiss)
  k1: 20
  k2: 6
  lambda_value: 0.3

As for the market->duke with SpCL, my experiment runs normally.
==> Statistics for dukemtmcreid on epoch 0: 411 clusters, 9919 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 1: 524 clusters, 8844 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 2: 621 clusters, 7554 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 3: 658 clusters, 7021 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 4: 689 clusters, 6330 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 5: 713 clusters, 5949 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 6: 733 clusters, 5614 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 7: 769 clusters, 4810 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 8: 747 clusters, 4571 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 9: 757 clusters, 3696 un-clustered instances, 0 unused instances
==> Statistics for dukemtmcreid on epoch 10: 741 clusters, 3490 un-clustered instances, 0 unused instances

 * Finished epoch   4  mAP: 24.6%  best: 24.6% *
 * Finished epoch   9  mAP: 54.5%  best: 54.5% *

Ah, I think I found your problem, i.e. launcher='none' in your log.

How did you run the script? Please provide your running command.

Non-distributed training with a single GPU:
python ${METHOD}/main.py ${METHOD}/config.yaml --work-dir ${WORK_DIR} --launcher "none" --set [optional arguments]
mine:    python ./SpCL/main.py SpCL/config.yaml --launcher "none" --set TRAIN.val_freq 10

Ok.
First, I did not try to train with a single GPU and my provided models are all trained on 4 GPUs. I am not sure how much would it affect when training with different numbers of GPUs. In my experiments, I found that 4 GPUs could achieve optimal performance, so it is recommended to use 4 GPUs. It is also the default setting in the training scripts.
Second, if you still want to use one GPU for training,

please still use dist_train.sh by modifying GPUS=${GPUS:-4} to GPUS=${GPUS:-1};
please modify TRAIN.LOADER.samples_per_gpu from 16 to 64.

I have modified the docs (https://github.com/open-mmlab/OpenUnReID/blob/master/docs/GETTING_STARTED.md#train) for better understanding.

Yes, you are right.
Distributed training with multiple GPUs:
sh dist_train.sh ${METHOD} ${WORK_DIR} [optional arguments]
I try it, but I obtain the error result:
sh dist_train.sh SpCL SpCL/config.yaml TRAIN.val_freq 10

PYTHON=python
METHOD=SpCL
WORK_DIR=SpCL/config.yaml
dist_train.sh: 9: dist_train.sh: Bad substitution

The 9 line is: PY_ARGS=${@:3}. So i try it with a single GPU.  For this issue with multiple gpu can you give me an advise?

Try sh dist_train.sh SpCL spcl_log TRAIN.val_freq 10
${WORK_DIR} means the log path.

Have you solved the problem?

Sorry,
sh dist_train.sh SpCL ../logs TRAIN.val_freq 10

PYTHON=python
METHOD=SpCL
WORK_DIR=../logs
dist_train.sh: 9: dist_train.sh: Bad substitution

sh dist_train.sh SpCL ../logs/config TRAIN.val_freq 10

PYTHON=python
METHOD=SpCL
WORK_DIR=../logs/config
dist_train.sh: 9: dist_train.sh: Bad substitution

Try to use bash instead of sh.
Plus, use path instead of ../logs/path, since the LOGS_ROOT has already be defined in the config.

Ok,  the problem is solved.  Thank you very much.
bash dist_train.sh SpCL ../logs TRAIN.val_freq 10

hello, what is the result of spcl + algorithm on a single GPU
@yjh576


typo: recoveryservices/resource-manager/Microsoft.RecoveryServices

polymorhpic -> polymorphic
friendlyname -> friendly name
operationss -> operations
upgradation -> upgrade

Automation for azure-sdk-for-js

Automation for azure-sdk-for-python

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-node

Automation for azure-sdk-for-java

Automation for azure-sdk-for-go


Expose ability to get ACR refresh token (suitable for docker login or helm registry login) from ContainerRegistryClient

Is your feature request related to a problem? Please describe.
I have a need to obtain a ACR refresh token suitable for use in invoking helm via the command line. For example, using the Azure CLI, I can run the following commands as described in az acr login with --expose-token:
TOKEN=$(az acr login --name <acrName> --expose-token --output tsv --query accessToken)
echo $TOKEN | helm registry login myregistry.azurecr.io \
            --username 00000000-0000-0000-0000-000000000000 \
            --password-stdin

I would like to be able to generate a similar token using the JavaScript SDK.
Describe the solution you'd like
Ideally, I would like to call getToken or getRefreshToken directory from the ContainerRegistryClient class. Ideally, this could look something like:
const AzIdentity = require('@azure/identity');
const AzContainerRegistry = require('@azure/container-registry');

const creds = new AzIdentity.DefaultAzureCredential();
const client = new AzContainerRegistry.ContainerRegistryClient('https://myregistry.azurecr.io', creds);
const token = await client.getRefreshToken();

Describe alternatives you've considered
Having looked at the code, this seems to be possible today by access the internal AuthenticationImpl class, but requires the user to understand the inner workings of the ACR authentication token exchange flow and the fact that the refresh token is what is actually needed by helm (not an access token). Additionally, this means that there does not seem to be any need to go through the entire challenge flow. The following produces a refresh token that allows successful login using docker login or helm repository login:
const AzIdentity = require('@azure/identity');
const AzContainerRegistry = require('@azure/container-registry');

const creds = new AzIdentity.DefaultAzureCredential();
const client = new AzContainerRegistry.ContainerRegistryClient('https://myregistry.azurecr.io', creds);
const authClient = client.client.authentication;
const aadAccessToken = await azCreds.getToken("https://containerregistry.azure.net/.default");
const acrRefreshToken = await authClient.exchangeAadAccessTokenForAcrRefreshToken(
  'access_token',
  'myregistry.azurecr.io', {
    accessToken: aadAccessToken.token,
  }
);

Since az acr login is exposed directly from the Azure CLI, I think this could be generally useful to be exposed via the JavaScript SDK and would avoid having to utilize the internal AuthenticationImpl class.

Hi @sgerace, thanks for the suggestion. We'll take it into consideration. It would be great to understand a bit more about the scenario where you think that this would be useful -- what benefit do you think adding this to the SDK give you over the CLI-based solution you have already?

I recognize that our particular situation might be a bit unique, but at a high level, we have a management service (container) running inside of our AKS cluster that is written in Node.js and utilizes the Kubernetes Javascript SDK to interact with cluster resources. Think of it as a deployment manager that orchestrates deployment and configuration of multiple environments within the cluster. Our AKS cluster is configured to Authenticate with Azure Container Registry (ACR) using a managed identity, so the AKS cluster nodes have AcrPull permission on the ACR. Thus far, the management service has only needed to operate on the Kubernetes API, however, we now have the need to invoke helm commands. Ideally, we would have used a JavaScript helm library (similar to to the helm Go library), but, unfortunately one does not exist for JavaScript (at least to our knowledge). As such, we've bundled the helm CLI into our image so we can invoke it (via spawn from our Node.js process. The issue is that helm is not aware that it is running in the cluster and has not way to authenticate with the ACR (to pull charts). As such, we had a few options: (1) we could bundle the az CLI into our image, which would require us to pull in python, etc., or (2) we could replicate the process being performed by az acr login using the JavaScript Azure SDK to obtain a refresh token that we could then provide to the helm CLI via helm registry login (again, via spawn). Bundling the helm CLI into the image is relatively lightweight since helm is a single executable (and seemed unavoidable), but we were looking to avoid adding the az CLI if we could help it.
I think the existence of az acr login --expose-token is a good indication that the ability to get a refresh token is a generally useful function, so it seems reasonable to me that automating this task via some JavaScript tooling would be useful to support (by exposing a function that performs a similar task).


[k8s-extension] Update extension CLI to v1.3.3

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
The precondition is to put your code inside this repository and upgrade the version in the pull request but do not modify src/index.json.

k8s-extension


Streamline Gslb Status

Remove redundant ManagedHosts
Show external IP addresses in HealthyRecords even if local service
is Unhealthy
Expose GeoTag
Resolves #116

It looks like that now
status:
  geoTag: eu
  healthyRecords:
    app3.cloud.example.com:
    - <IP_ADDRESS>
    - <IP_ADDRESS>
    - <IP_ADDRESS>
  serviceHealth:
    app1.cloud.example.com: NotFound
    app2.cloud.example.com: Unhealthy
    app3.cloud.example.com: Healthy


Long strings

Steps to reproduce the issue:

Start by typing a long string into a type = .standard field till the text can't fit anymore into it.
Issue 1: The field does not scroll horizontally nor reduces it's font size, and because of this newly entered characters become invisible to the user.
Now press the left arrow key
Issue 2: The part of the long string that was invisible now is visible when it should've been always visible to the user.

Expected result:

The font should be reduced to accommodate long strings or there should be some sort of horizontal scroll while the user is typing to keep showing the newly entered characters.
Pressing the right and left arrow keys on my keyboard should allow me to move freely inside the field.

Thank you @mradzinski for reporting. Will update soon.

Hello @mradzinski
I have just taken a look at your issue and I can't reproduce. The behaviour is correct.

This gif is extracted from the example app. I encourage you to open it and check its usage.

In addition, if you want to add any other behaviour (like reducing the font), I suggest you create your own subclass of UITextField/UITextView, and use:
let myTextView = MyTextView()
animatedTextInput.configure(with type: .generic(myTextView))

This will add your own implementation of UITextField/UITextView as a subview. Another suggestion is that you extend AnimatedTextInputType and create your own `AnimatedTextInputConfigurator.


bpmn-js-properties-panel should be a normal "dependency"

Describe the Bug
I'm not sure if I'm doing everything right but just adding "camunda-bpmn-js": "0.13.0-alpha.8", to package.json and trying out to use the Modeler fails because bpmn-js-properties-panel is missing. It is added as a peer dependency which means that it won't be installed by yarn. I think it should be moved to dependencies.
Steps to Reproduce
Expected Behavior
Environment

OS: Linux
Modeler version: 0.13.0-alpha.8

P.S. The same applies to @bpmn-io/properties-panel.

Hi,
thanks for creating this issue. I will try to explain why both the @bpmn-io/properties-panel and bpmn-js-properties-panel need to be peer dependencies. If there's any flaw in my reasoning, please feel free to point it out so that we can achieve right solution.
To start with, we want developers using camunda-bpmn-js to be able to provide properties panel extensions in their custom builds. Also, we don't want require developers to use non-standard deduplication tools, e.g. npm dedupe.
At first, we included both projects as direct dependencies. This resulted in a structure of node_modules similar to one below:
├── camunda-bpmn-js
│   ├── node_modules
│   │   ├── bpmn-js-properties-panel
│   │   │   ├── node_modules
│   │   │   │   ├── @bpmn-io/properties-panel

When one added bpmn-js-properties-panel to the project dependencies in order to import useService hook, it would then look like this:
├── bpmn-js-properties-panel
│   ├── node_modules
│   │   ├── @bpmn-io/properties-panel
├── camunda-bpmn-js
│   ├── node_modules
│   │   ├── bpmn-js-properties-panel
│   │   │   ├── node_modules
│   │   │   │   ├── @bpmn-io/properties-panel

Because useService relies on preact context, it would not work in such a setup. The properties panel would use the nested context while custom extension would take the closest module which resulted in an error. The same applied to @bpmn-io/properties-panel. This is what we have experienced.
With peer dependencies, the node_modules looks like this:
├── @bpmn-io/properties-panel
├── bpmn-js-properties-panel
├── camunda-bpmn-js

So we have only one instance of each dependencies which rely on context and hooks. Thus, extensibility is maintained.
If you have a better idea how to achieve this, we are happy to receive suggestions.

@barmac - thank you for the detailed answer. I was asking if this was connected with the other issue and you linked it while I was typing. :)
As for a better idea... I definitely don't have one. :smile: This is a very "hacky" way of making sure that the same preact context is used but I'm totally unfamiliar with both preact and writing plugins for bpmn-js so I really can't say more. I do understand your pain though and understand that this is the best you can do for now so thank you for the information!
If you would like, let's leave the issue open so maybe someone someday has a solution in mind. :)

Stupid question: Why not make preact a peer dependency of all these packages? 🤔

The reason we decided against it is that if we make preact dependency, we force the library users to install a preact version specified in the peer dependencies. We don't want to enforce preact version for the code which is not related to the properties panel.


Parallel rsyncs fail

Running multiple rsyncs in parallel/after each other fails because the .ssh folder already exists. I fixed this in a fork by appending a per-action string to the .ssh folder, but I guess the folder name could also be just randomly generated string?

thanks for raising this, I hadn't thought about that.
I hacked this together on a shoestring, and so I'm generally worried that the ssh usage isn't up to par here.
I can see three alternative ways of addressing this:

The action always cleans up after itself and deletes the .ssh contents it created.
The action just learns to take whatever .ssh is already in there.
As you implemented, each action writes its own random string .ssh. (That's not something we usually do with ssh keys though, right? – might be hard to reason about / feels odd).

I'm wondering whether, generally, .ssh credentials should persist between actions or not.
On the one hand, actions are supposed to be pretty isolated, I guess, so that would favor just cleaning up after itself.
What are your thoughts?

In my opinion this action is using ssh files as a deployment tool rather than the standardized "hosting keys and authorized_keys" way. For this reason I think the third option would make the most sense here, and preferably the keys would also be cleared up after the action.
I agree with what you said about each action being as isolated as possible, so randomized name definitely makes the most sense to me.

on closer inspection of the entrypoint.sh and the github actions docs, I'm not 100% sure that this isn't actually a github bug / undocumented behavior, as per below message to github support.
I'm going to wait for their response, and then fix this as discussed (if still relevant).

So far, I was under the impression as per the file system docs that (only?) /github/workspace persists across actions (not workflows), but I now have user reports that /github/home also appears to persist across actions (not workflows).
Is this correct?
Could you perhaps clarify this in the documentation?
The current documentation could be understood to mean that only /github/workspace persists, because only it has mentions in the docs:

An action can modify the contents of this directory, which subsequent actions can access.


Add defaults to RUST_SRC_PATH

If RUST_SRC_PATH is not set, use /usr/src/rust/src or /usr/local/src/rust/src.

/usr/local usually should override /usr, it should check /usr/local first.

Done

Thanks @Yamakaky


An emacs mode for F*

I'm currently using TuaregMode with some twists (https://github.com/FStarLang/FStar#emacs), but it would be nice to have something more tailored to F*.

Maybe we could from something like this ?
git://github.com/fsharp/fsharpbinding.git

TuaregMode crashes a lot once files differ from OCaml.

@markulf  I've heard this from Nik as well, that Tuareg crashes a lot, still I haven't experienced that myself even if I used it quite a lot. The only crash I've got was when reflowing a comment that's not yet closed. Could your crashes be a Windows thing?

Just for the record (indirectly related to this issue):

@strub and @nikswamy started building an Atom (https://atom.io/) mode for F* (https://github.com/FStarLang/FStarIDE)
@mlr-msft seems to be working on a vim plugin (https://github.com/FStarLang/FStar/issues/202#issuecomment-93781441)
Personally I think it's fine for different people to experiment with their favorite editors at this point.

Just for the record (indirectly related to this issue):
@strub https://github.com/strub and @nikswamy https://github.com/nikswamy started building an Atom (https://atom.io/ https://atom.io/) mode for F* (https://github.com/FStarLang/FStarIDE https://github.com/FStarLang/FStarIDE)
This is very cool, thanks =)
@mlr-msft https://github.com/mlr-msft seems to be working on a vim plugin (#202 (comment) https://github.com/FStarLang/FStar/issues/202#issuecomment-93781441) Personally I think it's fine for different people to experiment with their favorite editors at this point.

Clement Pit-Claudel has contributed a prototype fstar mode:
https://github.com/FStarLang/fstar.el