Improved and fixed multiple issues with resource SPTrustedIdentityTokenIssuer

Changed parameter ClaimsMappings in SPTrustedIdentityTokenIssuer to consume a JSON array, to allow an easier syntax
Changed SPTrustedIdentityTokenIssuer to throw an exception if certificate specified has a private key, since SharePoint doesn't accept it
Fixed issue with SPTrustedIdentityTokenIssuer to stop if cmdlet New-SPTrustedIdentityTokenIssuer returns null
Fixed issue with SPTrustedIdentityTokenIssuer to correctly get parameters ClaimProviderName and ProviderSignOutUri
Fixed issue with SPTrustedIdentityTokenIssuer to effectively remove the SPTrustedAuthenticationProvider from all zones before deleting the SPTrustedIdentityTokenIssuer

[x] Change details added to Unreleased section of changelog.md?
[x] Added/updated documentation and descriptions in .schema.mof files where appropriate?
[ ] Examples updated for both the single server and small farm templates in the examples folder?
[x] New/changed code adheres to [Style Guidelines]?(https://github.com/PowerShell/DscResources/blob/master/StyleGuidelines.md)?
[x] Unit and Integration tests created/updated where possible?

This change is

I've started the review of this one, but have a few questions before we get too far in to it. Details below.

Review status: 0 of 6 files reviewed at latest revision, 6 unresolved discussions.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 102 [r1] (raw file):

                if ($cert.HasPrivateKey) {
                    throw "The certificate must be installed without the private key."

Do we have a link to TechNet or some other reference that explains why it can't have a private key that we can include with this exception?

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 106 [r1] (raw file):

                $claimsMappingsArray = @()
                $MappingsList = $params.ClaimsMappings| ConvertFrom-Json

Why are we relying on putting JSON in here instead of working with the CIM instances which are more of a DSC standard here?

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 111 [r1] (raw file):

                    $runParams.Add("IncomingClaimTypeDisplayName", $_.Name)
                    $runParams.Add("IncomingClaimType", $_.IncomingClaimType)
                    if ($_.LocalClaimType -eq $null)

$null must always be on the left side of an equality check in PowerShell as per the style guidelines, can you swap this back please?

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 137 [r1] (raw file):

                $trust = New-SPTrustedIdentityTokenIssuer @runParams

                if (!$trust) {

The opening brackets for if statements need to be placed on a new line (as per style guidelines)

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 138 [r1] (raw file):

                if (!$trust) {
                    #throw "SharePoint failed to create the SPTrustedIdentityTokenIssuer."

Why is this commented out? We should be throwing an exception if something has gone wrong

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 164 [r1] (raw file):

                    $zone = $_
                    $providers = Get-SPAuthenticationProvider -WebApplication $wa.Url -Zone $zone -ErrorAction SilentlyContinue
                    if (!$providers) {

The opening brackets for if statements need to be placed on a new line (as per style guidelines)

Comments from Reviewable

Actually I also have a question regarding style guidelines: should the bracket just after 'Where-Object' and 'ForEach-Object' also be in a new line?
According to style guidelines I should probably do so, but I didn't see this in other resources

Review status: 0 of 6 files reviewed at latest revision, 6 unresolved discussions.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 102 [r1] (raw file):
Previously, BrianFarnhill (Brian Farnhill) wrote…
> Do we have a link to TechNet or some other reference that explains why it can't have a private key that we can include with this exception?

I couldn't find anything on TechNet, but here is the error recorded in SharePoint logs:
*wsmprovhost.exe (0x13D8)    0x06EC  SharePoint Foundation   PowerShell  6tf2    High    System.ArgumentException: Certificate file must contain only one valid X509 certificate without private keys.*
Note that in DSC, that will only return $null on cmdlet New-SPTrustedIdentityTokenIssuer without any clue of the actual error, so I added this test to ease troubleshooting

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 106 [r1] (raw file):
Previously, BrianFarnhill (Brian Farnhill) wrote…
> Why are we relying on putting JSON in here instead of working with the CIM instances which are more of a DSC standard here?

All of this is (was) very new to me and I spent a lot of time to troubleshoot/understand this.
But I think that an array of HashTables is not possible, and no other solution actually works:
If ClaimsMappings param is an array of HashTables
Example:
ClaimsMappings = @( @{Name = "Email"; IncomingClaimType = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"}, @{Name = "Account name"; IncomingClaimType = "http://schemas.microsoft.com/ws/2008/06/identity/claims/role"; LocalClaimType = "http://schemas.xmlsoap.org/customSPGroupClaimType"} )

Node MOF file is generated successfully but it creates a 1 dimensional array of key/value pairs, e.g. Name/"Email", Name/"Account name"
If ClaimsMappings param is an array of CimInstance
Example:
ClaimsMappings = @(
    (New-CimInstance -ClassName "MSFT_KeyValuePair" -ClientOnly -Property @{
        Name = "Email";
        IncomingClaimType = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"
    }),
    (New-CimInstance -ClassName "MSFT_KeyValuePair" -ClientOnly -Property @{
        Name = "Account name";
        IncomingClaimType = "http://schemas.microsoft.com/ws/2008/06/identity/claims/role";
        LocalClaimType = "http://schemas.xmlsoap.org/customSPGroupClaimType"
    })
)

It doesn't work at all, the generation of node MOF file fails with various errors depending on how type is declared in resource schema file.
I made a lot of research, and it's the 2nd issue discussed in https://powershell.org/forums/topic/hashtable-as-parameter-for-custom-dsc-resource/
This is this article that gave me the idea to use JSON as workaround

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 138 [r1] (raw file):
Previously, BrianFarnhill (Brian Farnhill) wrote…
> Why is this commented out? We should be throwing an exception if something has gone wrong

Ok, I put this in comment to raise this discussion this with you.
I couldn't decide if I should throw an exception if creation fails without exception (New-SPTrustedIdentityTokenIssuer just returns $null), or simply return...

Comments from Reviewable

Review status: 0 of 6 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 102 [r1] (raw file):
Previously, Yvand (Yvan Duhamel) wrote…
> I couldn't find anything on TechNet, but here is the error recorded in SharePoint logs:
> *wsmprovhost.exe (0x13D8)    0x06EC  SharePoint Foundation   PowerShell  6tf2    High    System.ArgumentException: Certificate file must contain only one valid X509 certificate without private keys.*
> Note that in DSC, that will only return $null on cmdlet New-SPTrustedIdentityTokenIssuer without any clue of the actual error, so I added this test to ease troubleshooting

OK that makes sense - can I get you to make sure we explain that in the description readme.md file just to make it clear

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.psm1, line 106 [r1] (raw file):
Previously, Yvand (Yvan Duhamel) wrote…
> All of this is (was) very new to me and I spent a lot of time to troubleshoot/understand this.
> But I think that an array of HashTables is not possible, and no other solution actually works:
>
> ## If ClaimsMappings param is an array of HashTables
> Example:
> ```PowerShell
> ClaimsMappings = @( @{Name = "Email"; IncomingClaimType = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"}, @{Name = "Account name"; IncomingClaimType = "http://schemas.microsoft.com/ws/2008/06/identity/claims/role"; LocalClaimType = "http://schemas.xmlsoap.org/customSPGroupClaimType"} )
> ```
> Node MOF file is generated successfully but it creates a 1 dimensional array of key/value pairs, e.g. Name/"Email", Name/"Account name"
>
> ## If ClaimsMappings param is an array of CimInstance
> Example:
> ```PowerShell
> ClaimsMappings = @(
>     (New-CimInstance -ClassName "MSFT_KeyValuePair" -ClientOnly -Property @{
>         Name = "Email";
>         IncomingClaimType = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"
>     }),
>     (New-CimInstance -ClassName "MSFT_KeyValuePair" -ClientOnly -Property @{
>         Name = "Account name";
>         IncomingClaimType = "http://schemas.microsoft.com/ws/2008/06/identity/claims/role";
>         LocalClaimType = "http://schemas.xmlsoap.org/customSPGroupClaimType"
>     })
> )
> ```
> It doesn't work at all, the generation of node MOF file fails with various errors depending on how type is declared in resource schema file.
> I made a lot of research, and it's the 2nd issue discussed in https://powershell.org/forums/topic/hashtable-as-parameter-for-custom-dsc-resource/
> This is this article that gave me the idea to use JSON as workaround
>

OK, so the array of key value pairs was never going to cut it then, it's meant to be used like this:
        ClaimsMappings = @(
            MSFT_KeyValuePair {
                Key = "name"
                Value = "value"
            }
            MSFT_KeyValuePair {
                Key = "name1"
                Value = "value1"
            }
            MSFT_KeyValuePair {
                Key = "name2"
                Value = "value2"
            }
        )

So it wasn't ever going to give the data structure you needed (also you shouldn't need to use New-CIMInstance to get your MOFs created like that).
Now my immediate concern is that this could form a breaking change since we are changing a property type, but technically speaking it never would have worked under the old model so we aren't breaking something that worked. @ykuijs what do you think here, do we need to make this a 2.0 thing?
In terms of moving forward here, I'm super hesitant to put the JSON array in here and would rather stick to a properly document MOF object of our own here to make sure the property names are all handled correctly. We could do something similar to SPWebAppPolicy where we have a second MOF type in the schema and take an array of that in here (Check out an Example of it in use as well as the MOF schema to see how we do that.) Again @ykuijs what do you think here?

Comments from Reviewable

Isn't the solution I used in SPWebAppPolicy an option in this case? Sounds like a similar issue where this sub class solution would be ideal:
SPWebAppPolicy WebAppPolicy
{
WebAppUrl = "http://sharepoint.contoso.com"
Members = @(
MSFT_SPWebPolicyPermissions {
Username           = "contoso\user1"
PermissionLevel    = "Full Control"
ActAsSystemAccount = $true
}
MSFT_SPWebPolicyPermissions {
Username        = "contoso\Group 1"
PermissionLevel = "Full Read"
IdentityType    = "Claims"
}
)
}

Yes it's definitely a good solution that I didn't consider because my research led me to the JSON array, which I didn't find bad.
I understand you prefer a custom object so I'll update the pull request to use this.
I'll also update the code style as done in other pull request "Update to follow PowerShell team style guidelines"
@BrianFarnhill indeed current version never worked, so should I update version number of the resource?
I'll also update the readme.md to explain private key issue.

@Yvand No need to increment version numbers - we'll make that call and handle it before we ship it. Overall this is looking pretty good at this point. Can I just confirm that you've been able to test this and see it working correctly with this code base before I look to merge it in?

Reviewed 6 of 6 files at r3.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

@BrianFarnhill Yes I tested it intensively and I can assure you this version works fine :)
But I don't understand why appveyor fails on last commit, I don't even see the full error message, do you know what's wrong?

@kwirkykat Can I get you to reset the appveyor webhooks for us on this repo? I've got permissions here but not in appveyor. Looks like the webhook is returning HTTP 400's and I'm not sure why.

@BrianFarnhill @Yvand Triggered this one to run again manually.
I updated the SharePointDsc AppVeyor webhook as well.

Thanks @kwirkykat
@Yvand Just a couple of little things to get it passing the tests. What you have is a valid MOF but there are issues with our doco generators - they are bugs I'll get fixed but for the time being if you could make these last couple of minor changes that should see the tests pass and we'll merge this in.

Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.schema.mof, line 7 at r3 (raw file):

    [Key, Description("Display name of the incoming claim type")] String Name;
    [Required, Description("URI of the incoming claim type")] String IncomingClaimType;
    [Write, Description("URI of the local claim type; not required if same as IncomingClaimType")] String LocalClaimType;

Can you take the ; out of the description here?

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.schema.mof, line 19 at r3 (raw file):

    [Required, Description("URL of the identity provider where user is redirected to for authentication")] String SignInUrl;
    [Required, Description("Identity claim type that uniquely identifies the user")] String IdentifierClaim;
    [Required, EmbeddedInstance("MSFT_SPClaimTypeMapping"), Description("Array of MSFT_SPClaimTypeMapping to use with cmdlet New-SPClaimTypeMapping")] String ClaimsMappings[];

Can you swap this so the EmbeddedInstance comes after the description

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.schema.mof, line 24 at r3 (raw file):

    [Write, Description("Sign-out URL")] String ProviderSignOutUri;
    [Write, Description("Present if the SPTrustedIdentityTokenIssuer should be created, or Absent if it should be removed"), ValueMap{"Present","Absent"}, Values{"Present","Absent"}] String Ensure;
    [Write, EmbeddedInstance("MSFT_Credential"), Description("POWERSHELL 4 ONLY: The account to run this resource as, use PsDscRunAsCredential if using PowerShell 5")] String InstallAccount;

Can you swap this so the EmbeddedInstance comes after the description

Comments from Reviewable

Review status: 5 of 6 files reviewed at latest revision, 3 unresolved discussions.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.schema.mof, line 7 at r3 (raw file):
Previously, BrianFarnhill (Brian Farnhill) wrote…
> Can you take the ; out of the description here?

Done.

Modules/SharePointDsc/DSCResources/MSFT_SPTrustedIdentityTokenIssuer/MSFT_SPTrustedIdentityTokenIssuer.schema.mof, line 19 at r3 (raw file):
Previously, BrianFarnhill (Brian Farnhill) wrote…
> Can you swap this so the EmbeddedInstance comes after the description

Done.

Comments from Reviewable

Reviewed 1 of 1 files at r4.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


[6.5.0] Packages needlessly uninstalled during sync due to non-canonicalized names (dots vs dashes)

Packages with dots in them are not compiled to the same file in version 6.4.0 and 6.5.0, where dots are replaced with dashes.
E.g. zope.event becomes zope-event
Environment Versions

OS Type: Arch Linux
Python version: Python 3.10.2
pip version: pip 20.3.4 from /usr/lib/python3.10/site-packages/pip (python 3.10)
pip-tools version: pip-compile, version 6.5.0

Steps to replicate

On version 6.4.0

requirements.in

pip-tools==6.4.0
gevent==21.12.0

requirements.txt (after pip-compile)

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile
#
click==8.0.3
    # via pip-tools
gevent==21.12.0
    # via -r requirements.in
greenlet==1.1.2
    # via gevent
pep517==0.12.0
    # via pip-tools
pip-tools==6.4.0
    # via -r requirements.in
tomli==2.0.0
    # via pep517
wheel==0.37.1
    # via pip-tools
zope.event==4.5.0
    # via gevent
zope.interface==5.4.0
    # via gevent

# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools

On version 6.5.0

requirements.in

pip-tools==6.5.0
gevent==21.12.0

requirements.txt (after pip-compile)

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile
#
click==8.0.3
    # via pip-tools
gevent==21.12.0
    # via -r requirements.in
greenlet==1.1.2
    # via gevent
pep517==0.12.0
    # via pip-tools
pip-tools==6.5.0
    # via -r requirements.in
tomli==2.0.0
    # via pep517
wheel==0.37.1
    # via pip-tools
zope-event==4.5.0
    # via gevent
zope-interface==5.4.0
    # via gevent

# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools

Expected result

The zope.event package being listed as zope.event in generated requirements.txt
Running pip-sync twice in a raw will not do anything on second run (except printing Everything up-to-date

Actual result

The zope.event package has been renamed to zope-event in generated requirements.txt
Running pip-sync does the following even if nothing has changed (i.e. it does so in two consecutive runs)

Found existing installation: zope.event 4.5.0
Uninstalling zope.event-4.5.0:
  Successfully uninstalled zope.event-4.5.0
Found existing installation: zope.interface 5.4.0
Uninstalling zope.interface-5.4.0:
  Successfully uninstalled zope.interface-5.4.0
Collecting zope-event==4.5.0
  Using cached zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting zope-interface==5.4.0
  Using cached zope.interface-5.4.0-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: setuptools in ./env/lib/python3.10/site-packages (from zope-event==4.5.0->-r /tmp/tmpcg_uqvf4 (line 1)) (58.1.0)
Installing collected packages: zope-interface, zope-event
Successfully installed zope-event-4.5.0 zope-interface-5.4.0
WARNING: You are using pip version 21.2.4; however, version 22.0.3 is available.
You should consider upgrading via the '/tmp/foo/env/bin/python -m pip install --upgrade pip' command.

This is expected and explained at https://github.com/jazzband/pip-tools/pull/1567#issuecomment-1030069457

@webknjaz
This is introduced there but can (I think) easily be resolved by using canonicalize_name in key_from_req. I have done so just now in the process of merging the new release back into #1329, and I think this particular valid issue is resolved in that PR.

Oh to be clear, I'm not referring to the issue as titled, but this part

Running pip-sync does the following even if nothing has changed (i.e. it does so in two consecutive runs)

I may re-title this.

The fix specifically, in key_from_req:
-    key = key.replace("_", "-").lower()
-    return key
+    return str(canonicalize_name(key))

I'm confirming that #1329 is fixing the pip-sync consecutive runs issue.
The . are still converted into - but you know better than I do about this one, so it's ok with me.
Many thanks!

Given how large #1329 is, I think it probably makes sense to pull 1eac955 out into a separate PR (and add a test 😉) in order to get it out sooner than later.

I'll take it out and rebase if someone merges the tiny change alone, it should have been part of the recent merge.
If no one does by the time I get back to this, I'll submit a small PR myself.

Please see the small PR at #1572.

Released as part of pip-tools v6.5.1.

Confirmed fixed with v6.5.1, thank you! :tada:

I can still reproduce.

@alexlitvinenko this is normal: . are replaced to - since v6.5.0.
What was not normal and is now fixed, was that running pip-sync twice in a raw would try to perform some tasks instead of displaying Everything up-to-date.

thanks @Rogdham
I've raised this question because Pycharm had raised it.

But, I see that pip correctly interprets  zope-interface in requirements.txt.
Please, consider this as just a warning.

That would be a bug in Pycharm I think.

FWIW: the Pycharm bug tracker


Display better error messages using in-app notification banners

Current state
Currently, Robrix can only show errors, warnings, and other non-content information by updating the Status label in the rooms_list. This is okay for early errors like login failures, but it doesn't work at all when any rooms at all are shown, as the status label will be pushed to the very bottom and replaced by a count of joined rooms found so far.
Clearly, the existing "solution" of abusing the Status label for this purpose is inadequate.
Desired solution
Instead, we should utilize some sort of pop-up banner to notify the user of any relevant info. This can be either a banner along the top or bottom, or a "toast"-like notification that auto-disappears after several seconds, or any other common notification paradigm.
In the future, this same mechanism could also be used for in-app notifications about actual Matrix content, such as informing the user of new messages that arrived in a hidden/background room that the user should be aware of.
While this mechanism doesn't yet exist in upstream Makepad, the Moxin app has implemented something very similar to notify the user that a download has completed. Here's what Moxin's in-app popup modal looks like when it is shown in the top-right corner of the app:

Note that the link button that will navigate to a specific part of the app should be optional, depending on the nature of the error message or other info that is included in the notification.
To implement this, we can copy the code for Moxin's PopupDialog widget and modify it to suit our needs: https://github.com/moxin-org/moxin/blob/5b21144ced0b8161c4565cba241ae0e3fd8ddf3a/src/shared/download_notification_popup.rs#L1

Hello, I will try to fix that.

I will look into this. Makepad widget has it's own popup notification, shall we use that instead? https://github.com/makepad/makepad/blob/rik/widgets/src/popup_notification.rs

@Guocork The popup list is ready. https://github.com/project-robius/robrix/pull/211#issuecomment-2435187398


[DM-34335] Support UID/GID assignment using Google Firestore

Add support for configuring Firestore (using workload identity for authentication and authorization) as a repository of UID and GID assignments.  When this option is enabled, Gafaelfawr will ignore UID and GID information from the upstream OpenID Connect ID tokens or from LDAP and instead assign UIDs and GIDs on first use of that username or group name, or look up the UID and GID from Firestore if one was already assigned to that username or group name.
To implement this support, retrieval of user information was extracted from the OpenID Connect authentication provider into a new internal service, and the cache implementation previously used only for internal and notebook tokens was made more generic and also used to cache UIDs and GIDs.  UID and GID information is cached for the lifetime of the process, since UIDs and GIDs are never reused and should never change.
UID and GID assignment is done from the ranges documented in DMTN-225, including assigning UIDs to bot users from a different range than the UIDs for regular users.
This PR also defers group information from LDAP so that it's no longer stored with the token data and instead is retrieved dynamically when needed, such as in /auth responses and responses to the user-info API. This allows changes to group memberships to be seen dynamically without requiring reauthentication, although the scopes of a token will not change.
Some refactoring work remains to be done, most notably:

Further clean up the API of the caching layer and make it more generic
Add helper function to do OpenID Connect login tests
Improve the user information service class structure to get rid of RuntimeError
Move code to determine scopes from groups to the user information service

Those will be mostly mechanical changes and thus not interesting to review, so I'll do them in a separate self-merged PR.
Some follow-up work remains, namely:

Cache group information from LDAP for a short period of time to avoid excess queries from a quick sequence of /auth calls
Support the new user information service with GitHub authentication
Support getting the user's full name and email address from LDAP

Those will be handled in subsequent PRs, probably after the 4.1.0 release.
This PR also refreshes the JavaScript dependency information to reflect recent dependabot updates and uses --legacy-peer-deps to work around the fact that react-aira-modal seems to be dead upstream (we need to eventually do something about that).

Note that DMTN-225 is currently slightly out of date (it doesn't have a bot UID range and therefore the other ranges are a bit off). I'm fixing that in another PR.

I think I can make the cache even better. I'll send you that as a separate PR when I figure out how I want to do it.
For bot users, for now you can keep doing what you have been doing and it will continue to work, but eventually I want to have Gafaelfawr automatically assign UIDs for those users when created via the API as well. The tricky part is what to do on deployments that don't use Firestore, since in that case the API client needs to still provide the UID. That's a bit hard to make clear in the API.


[stable/fluent-bit] Make tests use fluent-bit image specified in values file

Is this a new chart
No.
What this PR does / why we need it:
Currently, when .Values.backend.type is set to "forward", test pod uses hard-coded image "fluent/fluentd:v1.4-debian-1". This makes test(s) inconsistent with the actual version of fluent-bit specified in values.yaml (e.g. I'm currently using fluent/fluent-bit:1.5.2).
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

/assign @towmeykaw

Sorry, I've missed that it's actually a fluentd image is used for tests. Closing.


Error when dumping ROM

Hi all, I am trying to test the emulator because it is an interesting project for me.
Unfortunately I run into an error at step 2 of the Quick Start, when I try to dump the ROMs.
I have cloned the espressif/esp-idf repository.
My system is Ubuntu 14.04 on a VirtualBox.
Any help will be highly appreciated.
My output when executing sudo ~/esp/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 -b 921600 -p /dev/tty0 dump_mem 0x40000000 0x000C2000 rom.bin follows.

esptool.py  v2.3.1
Connecting...
Traceback (most recent call last):
File "/home/username/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2637, in
_main()
File "/home/username/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2630, in _main
main()
File "/home/username/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2356, in main
esp.connect(args.before)
File "/home/username/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 407, in connect
last_error = self._connect_attempt(mode=mode, esp32r0_delay=False)
File "/home/username/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 365, in _connect_attempt
self._port.setDTR(False)  # IO0=HIGH
File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 535, in setDTR
fcntl.ioctl(self.fd, TIOCMBIC, TIOCM_DTR_str)
IOError: [Errno 22] Invalid argument

Are you sure that tty0 is correct port? It is usually a system terminal.

I also think /dev/tty0 is not correct. You should use the same port as when you program the device.
Another option is to use the (Dumping the ROM with the main/main.c program) section of the README.
If you do this you have to enable
xTaskCreate(&dump_task, "dump_task", 2048, NULL, 5, NULL); in app_main()


Suggestion: OCR block

We should have a specialized image block that runs OCR (in JS or Python) on an uploaded image and stores the associated text.
Example uses:

Uploading a page of a lab notebook
Uploading a picture of a label on a bottle/sample

Maybe we can add this to the current media block... either running automatically or on the click of a button


Hoist String/Substring extensions to StringProtocol

This is work in progress, and would have been merged much more gradually if I hadn't run into so many apparently-unrelated CI failures.

@swift-ci Please test

@swift-ci Please benchmark

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - be5a4095157c7dc1db86bb183dace925a718a0a4
Test requested by - @dabrahams

Build comment file:
Build failed before running benchmark.

@swift-ci Please benchmark

@swift-ci Please benchmark

@swift-ci Please test OS X Platform

@swift-ci Please test OS X Platform

Build comment file:
Build failed before running benchmark.

@swift-ci Please clean smoke benchmark

@swift-ci Please clean smoke benchmark

@swift-ci Please smoke benchmark

Build comment file:
Build failed before running benchmark.

@swift-ci Please smoke benchmark

@swift-ci Please smoke test

Build comment file:
Build failed before running benchmark.

@swift-ci Please smoke test

@swift-ci Please benchmark

@swift-ci Please benchmark

@swift-ci Please benchmark

@swift-ci Please smoke test linux platform

Build comment file:
Build failed before running benchmark.

@swift-ci Please smoke test linux platform

@swift-ci Please test mac os platform

@swift-ci Please test mac os platform

@swift-ci Please test mac os platform


Question 10

Need explanatory text of terms singleton, non-singleton and more qualifying criteria/description for modern resilient and independently deployable.
There was a discussion around whether we need to add more options for customer coming from other cloud platforms

Minor tweaks but needs more consideration


Investigate adding request management to the client.

The HTTPNetworking target has an HTTPClient that is responsible for managing network requests.
Currently the client provides the ability to add adaptors, retriers and validators to each request. However, one thing that could be beneficial is the ability to cancel requests and see what requests are in-flight.

After consideration of the problem. I have decided not to implement this logic inside the HTTPNetworking target.
The concept of task management and cancellation is a relevant one, and a fairly common use case for networking. However, I have decided that I would rather rely on tighter integration with Swift concurrency than on an internal management scheme or abstraction around Swift concurrency for the following reasons.

In its current state, HTTPNetworking handles Task cancellations appropriately, and furthermore, aggressively checks for cancellation before each intensive step in the process. As a result, consumers of HTTPNetworking can easily manage their request's lifecycle with familiar and foundational language concepts such as Task and TaskGroup

Adding some sort of task management to the HTTPClient would involve updating the run() function to be non-async and instead internally dispatching a Task and then returning it fore management purposes. Alternatively, a second run() function could be added that returns a Task while maintaining the existing async run() function. Both of these approaches lack the clarity and conciseness I feel is important to HTTPNetworking. In the case of a single run() function returning a Task, this results in a call-site that adds additional unnecessary bloat for a use-case that I find is less common than simply awaiting a request's response. In the case of multiple run() functions, I believe consumers may have difficulty choosing the correct run() function at quick glance, leading to unintentional complexity.


Gluon block thread safety bug

import mxnet as mx
import threading
import queue
import gluonnlp as nlp

q = queue.Queue()
count = 2
net = mx.gluon.nn.Dense((10))
net.initialize(mx.init.Uniform(), ctx=[mx.gpu(0), mx.gpu(1)])
mx.nd.waitall()

def fn(net, ctx):
    with mx.autograd.record():
        x = mx.nd.random.uniform(low=0, high=1000, shape=(10, 128), ctx=ctx)
        q.put(net(x))
    mx.nd.waitall()
    return

threads = []
for i in range(count):
    t = threading.Thread(target=fn, args=(net, mx.gpu()))
    threads.append(t)
for i in range(count):
    threads[i].start()
for i in range(count):
    threads[i].join()

ys = []
for i in range(count):
    ys.append(q.get())
print(ys)

Error:
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 1154, in forward
    params = {k: v.data(ctx) for k, v in self._reg_params.items()}
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 1154, in <dictcomp>
    params = {k: v.data(ctx) for k, v in self._reg_params.items()}
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/parameter.py", line 565, in data
    return self._check_and_get(self._data, ctx)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/parameter.py", line 236, in _check_and_get
    "num_features, etc., for network layers."%(self.name))
mxnet.gluon.parameter.DeferredInitializationError: Parameter 'dense0_weight' has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters. You can also avoid deferred initialization by specifying in_units, num_features, etc., for network layers.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 976, in _deferred_infer_shape
    self.infer_shape(*args)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 1074, in infer_shape
    self._infer_attrs('infer_shape', 'shape', *args)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 1058, in _infer_attrs
    inputs, out = self._get_graph(*args)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 928, in _get_graph
    out = self.hybrid_forward(symbol, *grouped_inputs, **params)  # pylint: disable=no-value-for-parameter
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 94, in __exit__
    self._name_scope.__exit__(ptype, value, trace)
AttributeError: 'NoneType' object has no attribute '__exit__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "test.py", line 29, in fn
    q.put(net(x))
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 696, in __call__
    out = self.forward(*args)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 1156, in forward
    self._deferred_infer_shape(x, *args)
  File "/home/ubuntu/src/mxnet/python/mxnet/gluon/block.py", line 980, in _deferred_infer_shape
    raise ValueError(error_msg)
ValueError: Deferred initialization failed because shape cannot be inferred. 'NoneType' object has no attribute '__exit__'

This is because self.namescope is shared and overriden: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/block.py#L92


session object empty on Apollo Server API Route

Trying to get Server Actions working with my current setup which uses Apollo Server integrated into a NextJS API Route.
I can see the cookie being created on the browser, but pressing the Login button a second time and logging the session object still comes out empty.
Made a repo that shows the corresponding problem.
https://github.com/jorgev259/next-apollo-iron-test

Hey @jorgev259 I had a look at your example. It seems the issue lies where some code around appolo is not forwarding the cookies from the browser correctly down to your code. If you create a normal API route and read cookies it works, but when you try to read cookies with req.cookies then it fails and I have no idea why. This is specific to graphql/appolo.
There are many issues online about "Appolo Next.js cookies" and I will let you have a look at that! Good luck :)

Here's one such relevant thread https://community.apollographql.com/t/apollo-client-not-sending-cookies-to-backend-server-on-localhost/6450/6


MWE front and center

I saw your post on the Boost mailing lists; I'm super excited for your library!
As a small point of feedback, I know nothing about unicode, and don't want to. Going through the docs of your library, it seemed like it was aimed at people who are very technically adept, understand Unicode and share your frustration with the lack of tooling.
But I was just looking for how to spit out the symbol for a Laplacian in an error message!
Could you put a simple minimal working example showing how to spit out some mathematical symbols, maybe some subscripts/superscripts, with Greek letters, front and center? In the README would be fine.
(Mathematical symbols aren't required, of course. I guess an emoji would be just fine as well.)

You are definitely the target audience for this library, but I don't really understand what you're asking.  Are you asking how you represent these symbols, how you print them, both, or something else?  If you can post pseudocode of what you want to do, indicating the parts you don't know how to do, that might help.

How about this as a representative of what I want to do?
if (matrix_has_negative_eigenvalue) {
boost::uft8 err_msg = "-∇² is positive definite, but found discretization with λ<0";
throw std::domain_error(err_msg);
}

This will already work on any system that supports UTF-8 in the terminal.  That's all major platforms now, even Windows (though you have to adjust some settings to get it to work on Windows).  You can just use a std::string for this, since it's UTF-8.  In fact, you have to, if you want to interoperate with std::domain_error().


Logic error in rendering appropriate borders

For example, let n = 5, and you'd notice starting from the 7th path there will always be one up step that is misplaced to the right by one unit

Resolved.
There was an appending of incorrect classes to the list of classes of the currently selected square.

Resolved.
There was an appending of incorrect classes to the list of classes of the currently selected square.


NPM on windows <10

Without the bash for windows you can't just install npm with apt-get or cURL. Maybe included into the readme a link to download npm

That was a really well put together meme...A+

If you reopen this I will block you because you have no idea how long that took


Resolution diamond corresponds to the wrong date

Current behavior
I resolved this just now with resolution date/time of Sep 4 10:22 AM, but the diamond shows up on August 28.

Expected/desired behavior
The diamond should show on the day the resolution was known, and potentially the community prediction should be cut off at that time? @SylvainChevalier can you confirm desired behavior here.

I see two issues:

The "actual close time" failed to be set to the resolution time.
The diamond doesn't match any of the question's dates.

Something is still weird @hlibowski like this one I just resolved with resolution time today (Sep 13) 4PM but it appears as though the diamond is on Aug 12. This is on dev:

Also there it says resolved Nov 25, 2025? That's extra weird.

Wait a minute.... when I went back to dev.metaculus.com/questions/7596 just now the resolution diamond had moved! Now it looks right. :fearful:

@lsabor can you look at this, it looks like the resolution to datetime conversion is probably wrong (?)

It seems like the resolution diamond has appeared on top of the most recent CP. Simple fix - on it

I've fixed this issue, so closing.
There is another issue, probably not blocking, where the "resolved at" time for group questions is messy. It might require (at least for now) adding actual_resolve_time to the Post object.


merge initial jenkins config file for flocker jobs

In order for the new ci-platform to build jobs using the master branch, the build.yaml and job.config.j2 needs to be merged into master.
the jenkins [setup_ClusterHQ-flocker] (http://ci-live.clusterhq.com:8080/job/setup_ClusterHQ-flocker/) job consumes the build.yaml from within the branch.
This job executes a jinja2 parser consuming the job.groovy.j2 that generates a jobs.groovy file which is then consumed by the jenkins job dsl plugin.
This plugin then builds all the folders/jobs/views for that particular branch.
(build.yam + jobs.groovy.j2 ).render.py -> jobs.groovy | jenkins_job_dsl_plugin
this PR also updates the Dockerfile used in some of the jobs to consume a clusterhq/fpm- docker image, which is built by jenkins with the latest OS patches every 24 hours. These docker images already contain most of the development tools required to build/package flocker.

Thanks for addressing my comments. While this is very new, and hence difficult to review confidently, I have seen Jenkins in operation with these files. Given that the changes which could affect the existing system are simple, I intend to accept the PR.
Please merge when all tests are green. Thanks.

This comes with no documentation. Perhaps that can be a follow-up but I'd really like to be able to set up a Jenkins instance with the configuration in this branch.

this PR also updates the Dockerfile used in some of the jobs to consume a clusterhq/fpm- docker image, which is built by jenkins with the latest OS patches every 24 hours. These docker images already contain most of the development tools required to build/package flocker.

Can this instead be a separate issue?

I'm concerned that I don't really understand this yet. I would like to have a look at it tomorrow pre-merge if you don't mind.

@adamtheturtle this is a MVP,
will hapilly walk through the code with you as planned.
There's a lot more than those two files


[PE-D][Tester E] Edit person with student number

No one has a student number of 1, although it states there exists one in the address book.

Labels: type.FunctionalityBug severity.Low
original: sarjinius/ped#2

The new student ID s/A1234567X is probably duplicated with an existing one.
Student ID of 1 is not a valid Student ID, potentially wrong error thrown.

STUDENT_ID is being fuzzy matched. Should be exact match.


Move gRPC methods to unique concrete type

ref: https://github.com/osmosis-labs/osmosis/pull/1136/files#r834706380
We should define a dedicated Querier type that we define all our gRPC query methods on to provide a cleaner abstraction from the state-machine and the query layer.
e.g. https://github.com/cosmos/cosmos-sdk/blob/master/x/staking/keeper/grpc_query.go#L16-L19
Happy to tackle this if there's agreement :)

This seems great to me! The API bloat queries cause has definitely been bothering me!


[WIP][#35] first draft implementation of Path/FilePath functions and API

design API of the Path/FilePath for ticket #35

looking like a nice start for this brave new world
need to document the design choices, and limitation properly


Failing to read scenario file

StormLib in ReleaseUS compile mode (if not others) fails to read scenario file from attached volcanis.scm map. It is one of the blizzard maps but it seems to open fine in, at least, one of the older Ladik's MPQ editor version (<IP_ADDRESS>5)/older StormLib (8.20) versions.
volcanis.zip
e.g. following code prints "problem reading file":
#include <StormLib.h>
#include <cstdint>
#include <iostream>
#include <fstream>
#include <vector>

int main()
{
    HANDLE hMpq = NULL;
    bool opened = SFileOpenArchive(L"C:\\volcanis.scm", NULL, STREAM_FLAG_READ_ONLY, &hMpq);
    size_t bytesRead = 0;
    HANDLE openFile = NULL;
    if ( SFileOpenFileEx(hMpq, "staredit\\scenario.chk", SFILE_OPEN_FROM_MPQ, &openFile) )
    {
        size_t fileSize = (size_t)SFileGetFileSize(openFile, NULL);
        auto fileData = std::vector<std::uint8_t>(fileSize);

        if ( !SFileReadFile(openFile, (LPVOID)&fileData[0], (DWORD)fileSize, (LPDWORD)(&bytesRead), NULL) )
            std::cout << "problem reading file" << std::endl;

        SFileCloseFile(openFile);
    }

    if ( opened )
        SFileCloseArchive(hMpq);

    return 0;
}

Starcraft: Brood War v 1.16 is also unable to read the file. Is there any game version that can actually read it?

I believe it's unable to open in StarCraft cause there's a blizzard_2.cnd (blizzard certified) file and the file was resaved, I'll see if I can have this MPQ reproduced without the .cnd file...

This is not true.
In the game:

SFileOpenFileEx(hMapMpq, "staredit\scenario.chk", ..., &hFile) succeeds
SFileReadFile(hFile, pbBuffer, 0x24517, &dwBytesRead, NULL) never returns and throws a message box with error about file corruption.

Check this:

Open the map in MPQ Editor and look to MPQ Tables (Ctrl+T)
Go to scenario.chk and look at the file sectors.
The first byte is the compression type
You can see that first 4 file sectors are compressed with compression 0x01 (Huffmann)
5th and subsequent sectors are compressed with 0x03, which would be ZLIB+Huffmann. However, this compression is not allowed in Starcraft, because the entry for ZLIB is not present in the data decompression table:

.rdata:15050B4C dcmp_table      dd 40h                  ; DATA XREF: storm_551+15F↑o
.rdata:15050B50                 dd offset Decompress_ADPCM_mono
.rdata:15050B54                 dd 80h
.rdata:15050B58                 dd offset Decompress_ADPCM_stereo
.rdata:15050B5C                 dd 1
.rdata:15050B60                 dd offset Decompress_huff
.rdata:15050B64                 dd 8
.rdata:15050B68                 dd offset Decompress_PKLIB

Normally, I would say that someone tried to trick StormLib into a compression that is not supported by the game, but then the game should be able to load the file.
TLDR: This is some sort of f.u.c.k.u.p. file.

Yeah I agree this is an invalid MPQ, I misdiagnosed my issue as a read error (since other tools were still opening it for whatever reason) but it was a write error; this invalid MPQ was created when I (unintentionally) passed HUFFMAN | ZLIB to SFileWriteFile; I consider this my own error and not stormlib's (HUFFMAN shouldn't have been applied to a scenario file in the first place), but maybe generating an invalid MPQ or lacking validation for this flag combo is still not the right behavior; this issue can be closed if you want, otherwise below/attached is my reproduction of the MPQ getting corrupted.
(2)Volcanis.zip
#include "StormLib.h"
#include <cstdint>
#include <iostream>
#include <fstream>
#include <vector>

int main()
{
    HANDLE hMpq = NULL;
    HANDLE openFile = NULL;
    if ( !SFileOpenArchive(L"C:\\(2)Volcanis.scm", NULL, 0, &hMpq) )
        return 1;
    else if ( !SFileOpenFileEx(hMpq, "staredit\\scenario.chk", SFILE_OPEN_FROM_MPQ, &openFile) )
        return 2;

    size_t bytesRead = 0;
    size_t fileSize = (size_t)SFileGetFileSize(openFile, NULL);
    auto fileData = std::vector<std::uint8_t>(fileSize);
    if ( !SFileReadFile(openFile, (LPVOID)&fileData[0], (DWORD)fileSize, (LPDWORD)(&bytesRead), NULL) )
        return 3;
    else if ( bytesRead != fileSize )
        return 4;
    else if ( !SFileCloseFile(openFile) )
        return 5;

    openFile = NULL;
    if ( !SFileCreateFile(hMpq, "staredit\\scenario.chk", 0, (DWORD)fileData.size(), 0, MPQ_FILE_COMPRESS | MPQ_FILE_REPLACEEXISTING, &openFile) )
        return 6;
    else if ( !SFileWriteFile(openFile, &fileData[0], (DWORD)fileData.size(), MPQ_COMPRESSION_HUFFMANN | MPQ_COMPRESSION_ZLIB) )
        return GetLastError();
    else if ( !SFileFinishFile(openFile) )
        return 8;
    else if ( !SFileCloseArchive(hMpq) )
        return 9;

    return 0; // Success, but MPQ is corrupted
}

Ok then. My first suspicion was correct then. Closing the issue.

Fixed. New version of MPQ Editor is up.


Refactor tests/end_to_end/test_repo_manager.py and enable tests

Assumes #1166 has been merged.
The end-to-end tests for repo manager were disabled as part of the sql repo update. These should be reinstated.

Fixed in #1198


Add support for jmx metrics and prometheus exporter

I guess this would replace #140 and #135. Please think about how these changes can be tested - if we're adding resource manifests for external services, we need to deploy them too in the CI to make sure everything works as expected.

Added the extra services to the CI, but not sure if that's the best way.

I dont have time to really review but just wanted to mention that this should all be optional and disabled by default.

I realized I was trying to modify the test.sh to work on repeated runs when cleanup is skipped. But, two things would stop it even before it gets to the Prometheus items

kubectl create namespace would fail if namespace exists
kubectl create secret would fail if secret exists in namespace
I can do a follow up PR to fix these.

I realized I was trying to modify the test.sh to work on repeated runs when cleanup is skipped.

By default, the namespace is random. If you specify a namespace, you're supposed to clean it up manually. But we certainly could ignore the existing namespace and the secret, since it does make sense to avoid using a random namespace when debugging issues locally. Thanks for being patient with all the issues here.

@kempspo I applied some minor changes and opened #183, preserving your authorship of commits. I hope you don't mind.
I rewrote the tests in Python. I think they didn't work, the test container might have completed successfully even if the commands inside failed. Took me a bit to get it right, that's why I know it works now. BTW I removed the release label from the service monitor and instead added prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false when deploying the operator, so it can grab any service monitor.

BTW I could have pushed my changes here if you've opened the PR from any other branch than main, since it's protected.

@kempspo I applied some minor changes and opened #183, preserving your authorship of commits. I hope you don't mind.
I rewrote the tests in Python. I think they didn't work, the test container might have completed successfully even if the commands inside failed. Took me a bit to get it right, that's why I know it works now. BTW I removed the release label from the service monitor and instead added prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false when deploying the operator, so it can grab any service monitor.

No worries. Those changes make sense to me.
Yeah I realized just now that I've been working on main 😅


Grid prototypes are bound incorrectly (still)

@Hazelfire has figured out that the grid.bindToSelf function is actually bound and therefore all subsequent clones are also bound. Gonna check this out myself but all help appreciated.

Hey, I personally don't have a chance right now, but @Hazelfire can you test this using your previous method (that you figured this out with) to see if its fixed?

Here's a bot that shows the bug
return class RedBot extends Bot {
    main(grid) {

        let newgrid = grid.take(grid.get(0,0),'red');

        console.log("should be red");
        console.log(newgrid.get(0,0));
        console.log("should be red");
        console.log(newgrid.state[0][0]);
        console.log("should be neutral");
        console.log(grid.get(0,0));
        // Get all hexs
        let all = grid.all()

        // Filter for those that are empty
        let empty = all.filter(grid.is_empty)

        // Use lodash to get a random one
        return _.sample(empty)
    }
}

newgrid.state and newgrid.get give different results. This issue is still not fixed, I'll look into it

Ok, I think I worked it out. I have a strange feeling that newgrid.get is double bound (god, I hate JS).
What's happening is when grid.get is called, it's target function is already bound, and adding a this argument to the already bound target function does nothing. So when binding, we need to work out whether it is already bound, and if not, we'll bind it, if it is, then we change it's bound object.

Fixed the issue, still not sure how to build it though, npm run build just runs a development server.


Fd cube on SURF Research Cloud

Several services in one yaml file in the SURF settings

Several services in one yaml file in the SURF settings

Several services in one yaml file in the SURF settings

approved


Feature/style post/179060208

Purpose
A description of the purpose for this pull request and weather it is to provide new functionality or fix a bug
Solution
A descrition of what you have done to implement/fix the above mentioned feature or bug
with @
Change summary

A detailed list of bulleted
changes that go into detail about
the specifics of the changes
to the codebase

Steps to Verify

A numbered list of steps
To verify that this feature/bug
is now working

Additional details / screenshot

Any suplmenental pictures or material

should be in liberty30 branch


UDS+ Lab Observation inherits US Core BMI constraints

Inferno is applying the BMI constraints from US Core onto the UDS+ Lab Observation resource when using the specific BMI LOINC code:

None of these constraints are listed in the UDS+ IG, so they should not be a part of this Inferno test (or they should be added to the UDS+ IG if they are, in fact, supposed to be there).

These constraints are from base FHIR, not from US Core, but given the error messages I'm a little confused too.
There are certain "magic" codes which require that a certain profile be conformed to, one of which is BMI: http://hl7.org/fhir/R4/observation.html#core

If implementations use this Resource when expressing the profile-specific concepts as structured data, they SHALL conform to the following profiles:

Then the Vital Signs link below that brings you to the full set of magic Observation profiles, including height, weight, blood pressure, etc.
If the observation in question here has the  LOINC code 39156-5 then the core BMI profile applies: http://hl7.org/fhir/R4/bmi.html
What's confusing to me though is the 5th error message in your image though, it seems to be saying that code is required but not found.  Does this Observation have that code?

This specific test case used LOINC 59574-4, which is BMI percentile.
I'll also call out that the 6th issue from that image also conflicts with the UDS+ IG, as the UDS+ IG requires a precision of year.
Here's the exact JSON I pasted into the test kit:
{
"category": [
{
"coding": [
{
"code": "laboratory",
"system": "http://terminology.hl7.org/CodeSystem/observation-category"
}
]
}
],
"code": {
"coding": [
{
"code": "59574-4",
"system": "http://loinc.org"
}
]
},
"effectiveDateTime": "2024",
"encounter": {
"reference": "Encounter/.EnOjlM8xz8qXKvWe7KNIm1.NFqAxrC.qOE8hLDdv0Q"
},
"id": ".pJN7QgAnqJnNjftcIMjD6wdQDvJidtYhzbs-PstaNc",
"issued": "2024-04-04T17:54:56Z",
"meta": {
"profile": [
"http://fhir.org/guides/hrsa/uds-plus/StructureDefinition/uds-plus-lab-observation"
]
},
"performer": [
{
"reference": "Practitioner/eHRprVirvy4WAGyBeAFdS6w3"
}
],
"resourceType": "Observation",
"status": "final",
"subject": {
"reference": "Patient/drhDdFRZmc-RgYtuxa-qutruINh5F788w8p8HakKaso"
},
"valueString": "22"
}


Add simple password to staging server

Discussion began in #1303
Adding a simple password would continue to provide wide access to those we might call on to look at staging but would prevent indexing/crawling of the staging site.

I think this is up to @fujinaga

Ok. Please go ahead: medium priority?


Update to bundle-lib 0.2.15 to fix net-policy issue

Fixes Bug 1613280 - only create netpolicy if one already exists

I used the following dep version:
$ dep version
dep:
 version     : v0.5.0
 build date  : 2018-07-26
 git hash    : 224a564
 go version  : go1.10.3
 go compiler : gc
 platform    : linux/amd64
 features    : ImportDuringSolve=false

Coverage increased (+0.07%) to 41.003% when pulling aaf728c824d7b3308f2a655da82a9956f3f7c542 on 3.11-network-policy into 8aa1e33cc359df0266d918b393913f74dbcc40cd on release-1.3.

https://github.com/openshift/ansible-service-broker/blob/release-1.3/apb/templates/broker-auth.clusterrole.yaml#L20 for 1.3, if the same works for master.


Minor Fix Knowledge

I worked on ... because ... this was needed for knowledge center search index... this is needed to handled in a better way hanlding it more better way .. (will take a look)

Type of change:

[x] Bug fix
[x] Improvement

@07Himank can you fix style change


Artifact Plots in Readable formats

Is your enhancement request related to a problem? Please describe
In the current implementation, when users artifact plots it is not in a human readable format. A user needs to take additional steps to view the plots.
Describe the solution you'd like
Have artifact plots stored in a readable format like PNGs. This will eliminate the need to take additional steps when sharing plots to be made human readable.

this ticket needs a lot more detail but I think it'll end up being a new example in the documentation. I'm gonna move it to the docs milestone for now
we can already log PNGs as artifacts so if they're asking for something else we'll have to figure that out

Our customer's use case requires a collection of plots to be logged with each feature like so:   {feature_name: png_bytes}. They use pickle.dumps() for serialization and logging with rubicon. This requires extra effort to structure the dictionaries to make sure that the same concept  (e.g. SHAP feature importance) are logged together . Additionally, extra effort is spent on serializing and unserializing the plots for viewing.
They are not utilizing the current logging feature for Rubicon since it requires the effort to create a file for the png, then log the file, and finally delete the file. For our customer's optimization they are using in storing the PNGSs memory strings and using the data_bytes argument.   If they were to log each as an individual artifact, then they could have 100+ feature importance artifacts as opposed to just one to log and then retrieve.


Fixed broken link from "chart.md" to "charts.md"

Found and fixed this broken link in the docs.

This change is

I signed it!

Thanks! I'll watch for the linuxfoundation bot to pass, then merge. Sometimes (I am told), you have to force-push the same commit again to get that to go green after signing.

linuxfoundation cla signed.  I'll give it a bit and then try force pushing.

Thanks! Merged!


Application don't want start without google apps.

Are Google apps essential for eRouska?  I tried to run it on Lineage Os. Besides eRouška I don't know any application which can't run without GApps (yes, google maps) even Waze can, so why eRouška? Yes, I miss notification with Mattermost, but it can run and work.

Exposure Notifications API is part of Google Play services. Plus it checks for Google Play Protect certification, so it probably don't work if you sideload Google Play Services to uncertified devices like newest Huawei devices.

@vitstradal I use Lineage OS as well and I have Google apps installed. You can install them https://wiki.lineageos.org/gapps.html

@vandus Thanks for comment. I use Lineage OS because it doesn't contain Google apps. I only wander how it comes that only eRouska needs Google apps. Every other app I need can run without gaps. Even Google's Waze.
I understand that set of users who don't  trust Google AND kind of trust Czech government is not so big. Probably only me.

Sure, I kind of thought it was the reason. Well, yeah, as David said, the app is based on an API that Google developed and the way to use it is by using Google apps.

Yes, it doesn't work without GApps, we can't do anything about it, if we want to use Google/Apple API. Same reason why it doesn't work on new Huawei phones.


Add an API to $fh.forms Client SDK to remove a section

Motivation
https://issues.jboss.org/browse/RHMAP-9290
Changes

added removeSection() function along with new utility function getFieldsInSection().
back-compatibility adjustments
tests

@jhellar

LGTM, just commits could be squashed.


Fix json breakage in slack notifiiction.

The issue is that [json.loads() is called upstream] (https://github.com/GoogleCloudPlatform/forseti-security/pull/762/files#diff-35e1e2967889bcc1d9049e770706aee7R164), which then causes a problem when json.loads() is called here again.
Best way to make sure this doesn't regress in the future is to test this functionally, when we have an in-memory database.  Will open an issue for this.

Fixes issue #848


Some requests becomes very slow(several seconds)

We made a test on kubernetes and etcd: 4500 nodes, each has one pod. Then from the log of API server, some requests becomes very slow(seconds), and others are normal:
I0331 18:20:57.395788       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (114.958349ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:21434]
I0331 18:20:57.397911       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (111.765933ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:17112]
I0331 18:20:57.398033       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (122.495104ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:49117]
I0331 18:20:57.398760       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (122.507883ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:21683]
I0331 18:20:57.398988       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (114.352425ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:22163]
I0331 18:20:57.399574       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (114.201966ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:21821]
I0331 18:20:57.399955       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (114.618983ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:21148]
I0331 18:20:57.401170       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (128.034216ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:22215]
I0331 18:20:57.402067       6 handlers.go:144] , GET /api/v1/services: (2.107599062s) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:44284]
I0331 18:20:57.403535       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (112.886076ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:20543]
I0331 18:20:57.405869       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (145.922009ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:17561]
I0331 18:20:57.405895       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (2.472018267s) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:12967]
I0331 18:20:57.406131       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (113.295554ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:21904]
I0331 18:20:57.406217       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (125.41683ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:15228]
I0331 18:20:57.406274       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (118.959124ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:32937]
I0331 18:20:57.406311       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (114.947446ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:35304]
I0331 18:20:57.406341       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (116.553634ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:39378]
I0331 18:20:57.406461       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (120.794757ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:16322]
I0331 18:20:57.406886       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (121.67343ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:18948]
I0331 18:20:57.406990       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (146.058874ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:15068]
I0331 18:20:57.407187       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (2.474173989s) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:2995]
I0331 18:20:57.407295       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (115.19927ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:17204]
I0331 18:20:57.407333       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (115.41544ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:54553]
I0331 18:20:57.407474       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (121.624376ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:64090]
I0331 18:20:57.407657       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (146.841849ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:19616]
I0331 18:20:57.407689       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (117.484453ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:36729]
I0331 18:20:57.407806       6 handlers.go:144] , PUT /api/v1/nodes/<IP_ADDRESS>/status: (122.410125ms) 200 [[kubelet/v<IP_ADDRESS> (linux/amd64) kubernetes/unknown] <IP_ADDRESS>:51271]

As the log demonstrate, one GET and one PUT requests is very slow, but others are normal. It's really strange.
BTW, there are some "broken pipe" log in etcd's log, what's the meaning of this log?
81814 2016-03-31 12:17:30.714777 W | etcdhttp: error writing event (write tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:60910: write: broken pipe)
81815 2016-03-31 12:17:32.378037 W | etcdhttp: error writing event (write tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:54154: write: broken pipe)
81816 2016-03-31 12:17:34.574282 W | etcdhttp: error writing event (write tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:33120: write: broken pipe)
81817 2016-03-31 12:17:47.079529 W | etcdhttp: error writing event (write tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:57699: write: broken pipe)
81818 2016-03-31 12:17:47.648702 W | etcdhttp: error writing event (write tcp <IP_ADDRESS>:2379-><IP_ADDRESS>:33729: write: broken pipe)

Do you know how many items will that get return? And what size is the returned response?
Broken pipe means the client has already closed the connection. Most of the time it is a client side issue.
Moreover, please check https://github.com/coreos/etcd/blob/master/Documentation/reporting_bugs.md#reporting-bugs before reporting bugs.
We can look into this if you can reproduce the issue in an isolated environment. Or you can open an issue in kubernetes.

@mqliang I chatted with some people working on k8s. It is probably just because k8s trying to return large response to user. Please verify the response type and size.

@xiang90 Thanks. We are now adding more logs to API server, hoping we could reproduce it and find out what on earth caused the strange fluctuation.
Close this. Will open a new one if we confirm that the fluctuation is from etcd.


fix: inherit stdio pipes from parent process

This change fixes color output from child processes, allows the deploy shell commands to request user input, and makes temporary progress output show up (e.g. messages like "Updating CloudFormation stack (33s)").

@onebytegone with you to cut a release and notify internal users of the upgrade


connection to ATEM crashing after reboot (Companion Pi version 3.1 and latest beta)

Hi, I have also an ATEM connection issue.
Latest Blackmagic Switcher Software 9.2.1, latest stable Companion 3.1 running on a PI 4 with fast SD card and I also tried latest beta 3.2.
Issue:
I get a connection with ATEM Extreme ISO, when I setup a new connection. But if I want to use an ATEM connection from the backup, it works only until I reboot. After the reboot the module tries to connect and reconnect and reconnect and then crashes (error message: crashed, connection config unavailable).
It worked totally fine with companion 2.4.2. Also using ATEM Software Controll.app on Mac works like a charm, so probably no network configuration or firewall issue. I am running Companion on Companion Pi via Ethernet, Streamdeck XL is directly connected with this PI. Latest Mac OS.
EDIT: If I add another ATEM connection to the same ATEM Switcher, this one comes up stable after a reboot (not used on any buttons). Seems one of the existing button actions or macros or triggers is responsible for this module crash!!!
I want to re-use my existing profiles from @davidjoshuaford...
Logfile:
Here is my logfile (companion 3.1 or 3.2):
Date,Module,Type,Log
2023-10-05T15:27:11.153Z,Instance/ModuleHost,info,"Starting connection: ""atem""(vskApziC1WMESHmLjLvPC)"
2023-10-05T15:27:15.753Z,Instance/ModuleHost,debug,"Connection ""atem"" started"
2023-10-05T15:27:15.786Z,Instance/Wrapper/atem,info,Connected to a ATEM Mini Extreme ISO
2023-10-05T15:27:15.902Z,Instance/ModuleHost,debug,"Connection ""atem"" stopped"
2023-10-05T15:27:21.296Z,Instance/atem,verbose,"stdout: Starting up module class: la
Sentry disabled
"
2023-10-05T15:27:21.297Z,Instance/ModuleHost,debug,"Registered module client ""vskApziC1WMESHmLjLvPC"""
2023-10-05T15:27:28.354Z,Instance/ModuleHost,warn,"Instance ""atem"" failed to init: Error: Call timed out Error: Call timed out
at t.IpcWrapper.sendWithCb (/opt/companion/main.js:2:2165692)
at Object.init (/opt/companion/main.js:2:7901847)
at l. (/opt/companion/main.js:2:7910834)
at l.emit (node:events:514:28)
at l.emit (node:domain:489:12)
at ChildProcess. (/opt/companion/main.js:2:2153197)
at ChildProcess.emit (node:events:514:28)
at ChildProcess.emit (node:domain:489:12)
at emit (node:internal/child_process:937:14)
at process.processTicksAndRejections (node:internal/process/task_queues:83:21)"
2023-10-05T15:27:28.591Z,Instance/atem,verbose,"stdout: Module-host accepted registration
ThreadedClass (6660) Skipping exit handler registration as no exit handler is registered
"
2023-10-05T15:27:32.339Z,Instance/ModuleHost,info,"Starting connection: ""atem""(vskApziC1WMESHmLjLvPC)"
2023-10-05T15:27:39.319Z,Instance/ModuleHost,debug,"Connection ""atem"" started"
2023-10-05T15:27:39.444Z,Instance/Wrapper/atem,info,Connected to a ATEM Mini Extreme ISO
Any idea?

Can you (or @davidjoshuaford) email me a copy of the profile to import<EMAIL_ADDRESS>This isnt happening with a blank config, so it is probably in some way related to the amount of actions/feedbacks in use in that profile? Trying to recreate that myself will be rather time consuming and not guaranteed to reproduce the issue

Hi Julian, I'll email you. I'm not aware of an atem problem on import. The vlc connection does crash since the latest updates (the work around is to generate a new vlc connection, then import the vlc buttons and patch them to the new connection). I know vlc may be beyond your purview though.

thanks. Im suspecting its some combination of the lower performance of the pi causing things to take slightly too long. not sure though. with the profile it should be a quick and easy test to try and reproduce it though
it looks like a vlc fix was pushed to beta ~2 weeks ago for some upgrade script issues. perhaps that could be what you were seeing? I can take a quick look at it, its probably a small fix

ok, I've just tried importing that config onto a pi4.
My first thought was how slow it was doing that import. That took much longer than I would expect.
Once it finished, I could see it stuck as crashed. Disabling and reenabling it 'fixed' it. I am pretty sure we do retry crashed modules eventually, so it would probably recover by itself eventually.
But even enabling the module was slow, with higher cpu than I would expect for longer than I would expect.
Im going to see if I can figure out why it is taking so long and so much cpu on the companion side

Thanks for you efforts!!!
Julian Waller @.***> schrieb am Do. 5. Okt. 2023 um
22:18:

ok, I've just tried importing that config onto a pi4.
My first thought was how slow it was doing that import. That took much
longer than I would expect.
Once it finished, I could see it stuck as crashed. Disabling and
reenabling it 'fixed' it. I am pretty sure we do retry crashed modules
eventually, so it would probably recover by itself eventually.
But even enabling the module was slow, with higher cpu than I would expect
for longer than I would expect.
Im going to see if I can figure out why it is taking so long and so much
cpu on the companion side
—
Reply to this email directly, view it on GitHub
https://github.com/bitfocus/companion-module-bmd-atem/issues/264#issuecomment-1749584737,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF673T5DIECO6TY7WPBD6CTX54IZVAVCNFSM6AAAAAA5UTCFRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZGU4DINZTG4
.
You are receiving this because you authored the thread.Message ID:
@.***>

and fyi: The workaround David describes above worked here too. I was able to create a new ATEM connection, then import the atem buttons and patch them to the new connection (quite some work on 50 pages, a mapping for the full import would be nice too). Then everything works fine UNTIL I reboot the companion pi, then also this "new" atem connection keeps crashing... maybe this helps finding the root cause...

Hi Julian,
did you find any solution which improves Raspi4 speed, should I load latest
beta, did you include any update that might help me?
Kind regards, Michael
Julian Waller @.***> schrieb am Do. 5. Okt. 2023 um
22:18:

ok, I've just tried importing that config onto a pi4.
My first thought was how slow it was doing that import. That took much
longer than I would expect.
Once it finished, I could see it stuck as crashed. Disabling and
reenabling it 'fixed' it. I am pretty sure we do retry crashed modules
eventually, so it would probably recover by itself eventually.
But even enabling the module was slow, with higher cpu than I would expect
for longer than I would expect.
Im going to see if I can figure out why it is taking so long and so much
cpu on the companion side
—
Reply to this email directly, view it on GitHub
https://github.com/bitfocus/companion-module-bmd-atem/issues/264#issuecomment-1749584737,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF673T5DIECO6TY7WPBD6CTX54IZVAVCNFSM6AAAAAA5UTCFRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZGU4DINZTG4
.
You are receiving this because you authored the thread.Message ID:
@.***>

I'm having the same issue.

I have pushed some changes to beta, but I have also pushed some new features which may counteract the improvement and could have some instability

Hi Julian,
with the latest beta of companion I can confirm Atem connection is working
and performance on a raspberry pi back to normal. Thanks for your support,
I guess this ticket can be closed!
Kind regards, Michael
Julian Waller @.***> schrieb am Do. 19. Okt. 2023 um
00:30:

I have pushed some changes to beta, but I have also pushed some new
features which may counteract the improvement and could have some
instability
—
Reply to this email directly, view it on GitHub
https://github.com/bitfocus/companion-module-bmd-atem/issues/264#issuecomment-1769500910,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF673T3JNLVJ7GZD6IWHW6TYABJ6XAVCNFSM6AAAAAA5UTCFRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRZGUYDAOJRGA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Unit testing nodes

I have been using this project and it's quite amazing!
Describe the solution you'd like
I would like to test out a combination of nodes.  to do this, I need the ability to push individual messages to nodes and (sort of) invoke multiple cycles in the graph. for instance,

Push a message to one of the nodes
Assert the value in the output
Push message onto one of the other nodes
Assert the value in the output

Describe alternatives you've considered
I have been looking at the unit tests. I can see you create out ts in the test and push it to the node/graph and check the output in the end. I am able to do the same, however, do you think I can test out using the above workflow pls
sorry, if I missed something.
ta!

I'm somewhat curious what you're using the above workflow for. Is it just so tests fail quicker if an output value is incorrect? You could always add assertions within the node itself before you return, but that will incur unnecessary overhead.
As you alluded to, you can add any intermediate node's output to the result of csp.run by using csp.add_graph_output. Then you can extract all its values from the result dictionary and unit test against those values.

Ok, thanks for more info. This is doable using csp.curve if you just set the times to be in a specific order. In your example:

Let node N = add(A, B)
step 1: stream A has event in a csp.curve at time t
check output of N in node N before returning (could do this by asserting in the node, or printing if its debug)
step 2: stream B has event in a csp.curve at time t+1
same as 3

You could also avoid any modifications to N by just checking all outputs at the end of csp.run. Does this answer your question?

ahh ok, I must be doing something wrong in my current tests then, I will give it another shot, thanks a lot mate. I will try again when I am home