SPUserProfileServiceApp: The WinRM client cannot process the request. A computer policy does not allow the delegation of the user credentials to the target computer because the computer is not trusted.

Details of the scenario you try and problem that is occurring:
Trying to configure a new farm with DSC
I get the error from the SPUserProfileServiceApp resource:
VERBOSE: [SP2016Ent01sp01]:                            [[SPUserProfileServiceApp]UserProfileServiceApp] Executing using
 a provided credential and local PSSession as user sp\_spfrm
Connecting to remote server SP2016Ent01sp01 failed with the following error message : The WinRM client cannot process
the request. A computer policy does not allow the delegation of the user credentials to the target computer because
the computer is not trusted. The identity of the target computer can be verified if you configure the WSMAN service to
use a valid certificate using the following command: winrm set winrm/config/service
'@{CertificateThumbprint="<thumbprint>"}'  Or you can check the Event Viewer for an event that specifies that the
following SPN could not be created: WSMAN/<computerFQDN>. If you find this event, you can manually create the SPN
using setspn.exe .  If the SPN exists, but CredSSP cannot use Kerberos to validate the identity of the target computer
and you still want to allow the delegation of the user credentials to the target computer, use gpedit.msc and look at
the following policy: Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow
Fresh Credentials with NTLM-only Server Authentication.  Verify that it is enabled and configured with an SPN
appropriate for the target computer. For example, for a target computer name "myserver.domain.com", the SPN can be one
of the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com. Try the request again after these changes. For more
information, see the about_Remote_Troubleshooting Help topic.
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:) [], CimException
    + FullyQualifiedErrorId : -2144108124,PSSessionOpenFailed
    + PSComputerName        : SP2016Ent01sp01

After the configuration is applied the user profile service is in place, but I need to understand what this error means and I have to get rid of it if possible.
All the other service applications and other resources are created without any errors.
I don't use any certificate for credentials: I use PSDscAllowPlainTextPassword = $True instad.
The DSC configuration that is using the resource:
The resource looks like this:
            SPUserProfileServiceApp UserProfileServiceApp
            {
                Name                    = "User Profile Service Application"
                ApplicationPool         = "SharePoint Services App Pool"
                MySiteHostLocation      = "http://$SPSiteCollectionHostName/sites/my"
                ProfileDBName           = "SP_UserProfiles"
                SocialDBName            = "SP_Social"
                SyncDBName              = "SP_ProfileSync"
                EnableNetBIOS           = $false
                FarmAccount             = $SPFarmAccountCredential
                PsDscRunAsCredential    = $SPInstallAccountCredential
                DependsOn               = @("[SPServiceAppPool]SharePointServicesAppPool","[SPSite]MySite")
            }

Version of the Operating System and PowerShell the DSC Target Node is running:
Windows Server 2016
PS C:\Users\dauser1> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      14393  1613

Version of the DSC module you're using:
<IP_ADDRESS>

Hi @shurick81, can you please more Verbose output and not just the line that is failing? To determine where this is being generated, I am interested in what is going on before and after this error.

@shurick81 Also, can you give us a bit more context related to the deployment context? Are you pushing the configuration from a Push Server (a.k.a. remotely calling Start-DSCConfiguration from a different box) or using Pull Refresh Mode? Is the MOF encrypted at Rest using a CertificateFile and Thumbprint property in the ConfigurationData?

I'll jump in and put some money on where this is coming from. In version 1.x of SharePointDsc there are two resources that still use CredSSP internally to impersonate different contexts - SPUserProfileServiceApp and SPUserProfileSyncService. So my money here is on that being an issue because CredSSP isn't configured on the machine.
There are two options here:

You grab the xCredSSP module and configure the CredSSP settings (the typical scenario here is to allow only each machine to connect to itself as a client/server combo - this alleviates most of the security concerns attached to CredSSP and will get things working with this resource)
You wait for us to release version 2.0 of SharePointDsc where we have completely removed the need for CredSSP and this resource will then be able to work without additional configuration to the system being required.

Does that make sense?

Hello colleagues,
Thank you for all your replies.
Sorry for delay but it took some time to organize clean tests and provide you more details.
First, I run following configuration:
Configuration SPFarm
{
    param(
        $configParameters
    )
    $DomainName = $configParameters.DomainName;

    Import-DscResource -ModuleName PSDesiredStateConfiguration
    Import-DSCResource -ModuleName xNetworking
    Import-DSCResource -ModuleName xSQLServer -Name xSQLServerAlias
    Import-DscResource -ModuleName xCredSSP
    Import-DSCResource -ModuleName SharePointDSC

    Node $AllNodes.NodeName
    {

        LocalConfigurationManager
        {
            RebootNodeIfNeeded = $true;
        }

        xCredSSP CredSSPServer
        {
            Ensure  = "Present"
            Role    = "Server"
        }

        xCredSSP CredSSPClient
        {
            Ensure = "Present";
            Role = "Client";
            DelegateComputers = "*.$DomainName"
        }

    }
}

Here is the result I managed to grab:

Then after reboot I see that configuration is pending and running existing:

Then I apply following configuration:
Configuration SPFarm
{
    param(
        $configParameters,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPPassphraseCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPFarmAccountCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPInstallAccountCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPWebAppPoolAccountCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPServicesAccountCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPSearchServiceAccountCredential,
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $SPCrawlerAccountCredential,
        [Boolean]$GranularApplying = $false,
        [Boolean]$SearchTopologyGranule = $false
    )
    $DomainName = $configParameters.DomainName;
    $SPSiteCollectionHostName = $configParameters.SPSiteCollectionHostName;

    $shortDomainName = $DomainName.Substring( 0, $DomainName.IndexOf( "." ) );
    $webAppHostName = "SPWA_01.$DomainName";
    $DBServer = $configParameters.SPDatabaseServer;
    if ( !$DBServer -or ( $DBServer -eq "" ) )
    {
        $DBServer = $null;
        $configParameters.Machines | ? { $_.Roles -contains "SQL" } | % { if ( !$DBServer ) { $DBServer = $_.Name } }
    }

    Import-DscResource -ModuleName PSDesiredStateConfiguration
    Import-DSCResource -ModuleName xNetworking
    Import-DSCResource -ModuleName xSQLServer -Name xSQLServerAlias
    Import-DscResource -ModuleName xCredSSP
    Import-DSCResource -ModuleName SharePointDSC

    Node $AllNodes.NodeName
    {

        xHostsFile WAHostEntry
        {
            HostName  =  $webAppHostName
            IPAddress = "<IP_ADDRESS>"
            Ensure    = "Present"
        }

        xHostsFile SiteHostEntry
        {
            HostName  = $SPSiteCollectionHostName
            IPAddress = "<IP_ADDRESS>"
            Ensure    = "Present"
        }

        xSQLServerAlias SPDBAlias
        {
            Ensure      = 'Present'
            Name        = $configParameters.SPDatabaseAlias
            ServerName  = $DBServer
        }

        SPFarm Farm
        {
            Ensure                    = "Present"
            DatabaseServer            = $configParameters.SPDatabaseAlias
            FarmConfigDatabaseName    = "SP_Config"
            AdminContentDatabaseName  = "SP_AdminContent"
            Passphrase                = $SPPassphraseCredential
            FarmAccount               = $SPFarmAccountCredential
            RunCentralAdmin           = $true
            CentralAdministrationPort = 50555
            ServerRole                = "SingleServerFarm"
            PsDscRunAsCredential      = $SPInstallAccountCredential
            DependsOn                 "[xSQLServerAlias]SPDBAlias"
        }

        SPManagedAccount ApplicationWebPoolAccount
        {
            AccountName             = $SPWebAppPoolAccountCredential.UserName
            Account                 = $SPWebAppPoolAccountCredential
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = @( "[SPFarm]Farm" )
        }

        SPWebApplication RootWebApp
        {
            Name                    = "RootWebApp"
            ApplicationPool         = "All Web Application"
            ApplicationPoolAccount  = $SPWebAppPoolAccountCredential.UserName
            Url                     = "http://$webAppHostName"
            DatabaseName            = "SP_Content_01"
            AuthenticationMethod    = "NTLM"
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = "[SPManagedAccount]ApplicationWebPoolAccount"
        }

        SPCacheAccounts CacheAccounts
        {
            WebAppUrl               = "http://$webAppHostName"
            SuperUserAlias          = "$shortDomainName\$($configParameters.SPOCSuperUser)"
            SuperReaderAlias        = "$shortDomainName\$($configParameters.SPOCSuperReader)"
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = "[SPWebApplication]RootWebApp"
        }

        SPWebAppPolicy RootWebAppPolicy
        {
            WebAppUrl               = "http://$webAppHostName"
            MembersToInclude        = @(
                MSFT_SPWebPolicyPermissions {
                    Username        = $SPInstallAccountCredential.UserName
                    PermissionLevel = "Full Control"
                    IdentityType    = "Claims"
                }
            )
            SetCacheAccountsPolicy = $true
            PsDscRunAsCredential   = $SPInstallAccountCredential
            DependsOn              = "[SPCacheAccounts]CacheAccounts"
        }

        SPSite RootPathSite
        {
            Url                     = "http://$webAppHostName"
            OwnerAlias              = $SPInstallAccountCredential.UserName
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = "[SPWebApplication]RootWebApp"
        }

        SPSite RootHostSite
        {
            Url                         = "http://$SPSiteCollectionHostName"
            OwnerAlias                  = $SPInstallAccountCredential.UserName
            Template                    = "STS#0"
            HostHeaderWebApplication    = "http://$webAppHostName"
            PsDscRunAsCredential        = $SPInstallAccountCredential
            DependsOn                   = "[SPSite]RootPathSite"
        }

        SPManagedAccount SharePointServicesPoolAccount
        {
            AccountName             = $SPServicesAccountCredential.UserName
            Account                 = $SPServicesAccountCredential
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = "[SPFarm]Farm"
        }

        SPServiceAppPool SharePointServicesAppPool
        {
            Name                    = "SharePoint Services App Pool"
            ServiceAccount          = $SPServicesAccountCredential.UserName
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = "[SPManagedAccount]SharePointServicesPoolAccount"
        }

        SPSite MySite
        {
            Url                         = "http://$SPSiteCollectionHostName/sites/my"
            OwnerAlias                  = $SPInstallAccountCredential.UserName
            Template                    = "SPSMSITEHOST#0"
            HostHeaderWebApplication    = "http://$webAppHostName"
            PsDscRunAsCredential        = $SPInstallAccountCredential
            DependsOn                   = "[SPSite]RootPathSite"
        }

        SPUserProfileServiceApp UserProfileServiceApp
        {
            Name                    = "User Profile Service Application"
            ApplicationPool         = "SharePoint Services App Pool"
            MySiteHostLocation      = "http://$SPSiteCollectionHostName/sites/my"
            ProfileDBName           = "SP_UserProfiles"
            SocialDBName            = "SP_Social"
            SyncDBName              = "SP_ProfileSync"
            EnableNetBIOS           = $false
            FarmAccount             = $SPFarmAccountCredential
            PsDscRunAsCredential    = $SPInstallAccountCredential
            DependsOn               = @("[SPServiceAppPool]SharePointServicesAppPool","[SPSite]MySite")
        }

    }
}

Here is the full verbose output:
dscprofile.txt
@BrianFarnhill , when may I expect the 2.0 version?
When it comes to using xCredSSP, do I have to use certificates? I currently just use PSDscAllowPlainTextPassword = $True
Just for the reference, the SharePoint version I use:

SharePoint 2016
August 2017 Public Update for SharePoint 2016
Language Packs:
ru-ru
sv-se
nb-no

I also tried with SP 2013 with the same error.

Just in case, I'm telling that my intention is to be able to run the configuration not only via Start-DscConfiguration but also with Set-AzureRmVmDscExtension.

We are implementing one last change for v2.0. I expect we would be ready for release somewhere next month (no guarantees :-)).
Can you test the following code on the target server:
$farm = Get-Credential
$s = New-PSSession -ComputerName [FQDN of Target Server] -Authentication CredSSP -Credential $farm
Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.SharePoint.PowerShell -ea 0 }
Invoke-Command -Session $s -ScriptBlock { get-spfarm }
Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
Get-PSSession | Remove-PSSession

A preview of what v2.0 will be like is available at: https://github.com/powershell/sharepointdsc/tree/preview-2.0

@shurick81 The question of "do i need to use certificates for xCredSSP" isn't actually a question specific to that module, rather to DSC in general. Any time you have a Configuration in PowerShell that has a PSCredential used in it anywhere (as a property value, or as the PsDscRunAsCredential value) you will need to use a certificate to encrypt the passwords in the MOF files, or you will need to tell it to use plain text passwords. So that being said, I actually don't think you need to use credentials for those resources in xCredSSP so you should be fine.
The one thing here that has me interested as well is the xCredSSP stuff requiring a reboot every time. I believe this was a bug that got introduced and never fixed. Can I suggest you specifically try version 1.0.1 of xCredSSP - its what I use in my builds and it seems to work fine without that issue.

@ykuijs, here is the output of the test you suggested:
PS C:\Windows\system32> $farm = Get-Credential
>> $s = New-PSSession -ComputerName SP2016Ent01sp01 -Authentication CredSSP -Credential $farm
>> Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.SharePoint.PowerShell -ea 0 }
>> Invoke-Command -Session $s -ScriptBlock { get-spfarm }
>> Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
>> Get-PSSession | Remove-PSSession

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
New-PSSession : [SP2016Ent01sp01] Connecting to remote server SP2016Ent01sp01 failed with the following error message
: The WinRM client cannot process the request. A computer policy does not allow the delegation of the user credentials
to the target computer because the computer is not trusted. The identity of the target computer can be verified if you
configure the WSMAN service to use a valid certificate using the following command: winrm set winrm/config/service
'@{CertificateThumbprint="<thumbprint>"}'  Or you can check the Event Viewer for an event that specifies that the
following SPN could not be created: WSMAN/<computerFQDN>. If you find this event, you can manually create the SPN
using setspn.exe .  If the SPN exists, but CredSSP cannot use Kerberos to validate the identity of the target computer
and you still want to allow the delegation of the user credentials to the target computer, use gpedit.msc and look at
the following policy: Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow
Fresh Credentials with NTLM-only Server Authentication.  Verify that it is enabled and configured with an SPN
appropriate for the target computer. For example, for a target computer name "myserver.domain.com", the SPN can be one
of the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com. Try the request again after these changes. For more
information, see the about_Remote_Troubleshooting Help topic.
At line:2 char:6
+ $s = New-PSSession -ComputerName SP2016Ent01sp01 -Authentication Cred ...
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108124,PSSessionOpenFailed
Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:3 char:25
+ Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.Shar ...
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:4 char:25
+ Invoke-Command -Session $s -ScriptBlock { get-spfarm }
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:5 char:25
+ Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

Might it be that the xCredSSP resource didn't work properly?

@BrianFarnhill , So what you are saying is that using CredSSP does not require using certificates and I can continue using PSDscAllowPlainTextPassword unless I need to encrypt the credentials?
Sorry if my report was confusing but it was not like it reboots every time. It only reboots twice. Probably because there are two resources and each of them requires DSC manager for reboot?

@shurick81 You only need to use encryption on a resource that has a PSCredential object assigned to one of its properties, and if you look at xCredSSP you don't specifically need to use these anywhere (the properties are just Ensure, Role, and DelegateComputers - none of which are a credential). If you use the PsDscRunAsCredential on the xCredSSP (or any) resource then yes you will need to deal with encryption by either allowing plain text passwords or doing it with a certificate - but you shouldn't need to use this property with xCredSSP as the local system account (that DSC executes as when you don't use PsDscRunAsCredential) has the required access to do the work that xCredSSP does.
Does that make sense?

Ok, maybe I just puzzled myself, but what I tried to get my head around is why it doesn't work for me even if I apply xCredSSP parameters. As you can see, I still get error with SPUserProfileServiceApp resource. WIth this resource, I do use PsDscRunAsCredential but I don't apply any encryption. I use PSDscAllowPlainTextPassword. Can this be a root cause?

@shurick81 No, the encryption there isn't related to this at all. Based on the output you have there the CredSSP settings aren't applied correctly (as per the description in the error there). There are some things to try for this:

Try configuring it with xCredSSP. My usual configuration for a dev environment looks something like this:
 xCredSSP CredSSPServer
 {
     Ensure = "Present"
     Role = "Server"
 }
 xCredSSP CredSSPClient
 {
     Ensure = "Present"
     Role = "Client"
     DelegateComputers = "*.$domainName"
 }

Now the reason I say this is suitable for a dev environment is because of the DelegateComputers property - in this case I say that any computer in the same domain (where $domainName is set to something like "contoso.local" or whatever the FQDN of the domain is) can connect to this computer via CredSSP. For a more secure implementation I would limit this to something that would filter it to just the SharePoint servers - or for a more secure version than that make sure that each computer has only its own computer name there so you can only do local CredSSP connections.

If you can't do this via DSC you can always configure CredSSP manually. There is some good doco on how to use the Enable-WSManCredSSP cmdlet for this, make sure you do both client and server on each node and again the same details about what you put in for delegate computers applies here still.

Do it via group policy - there are settings for the CredSSP stuff you can manage via GPO's instead of doing it manually. Now if you have done the first two options and they aren't working I would be inclined to believe there might actually be GPO's in place that are actually denying the use of CredSSP which would be causing your issue. But if you can't get the first two working have a chat to whoever manages the group policies in your environment to see if anything has been put in place already on these settings

Hope that helps you get started.

Hi Brian,
So isn't the xCredSSP configuration that I already apply is not correct?
I quoted it above:
        xCredSSP CredSSPServer
        {
            Ensure  = "Present"
            Role    = "Server"
        }

        xCredSSP CredSSPClient
        {
            Ensure = "Present";
            Role = "Client";
            DelegateComputers = "*.$DomainName"
        }

You were also wondering why DSC manager restarts the computer when applying it. Might it be that this resource was never applied correctly? What is the best way to check this?

In my test code, you didn't use the FQDN of the server (so servername.domain.com) but just the servername instead. Since you didn't post your data file, I don't know for sure, but I expect that you also aren't using the FQDNs in there.
So the issue you are running into is that in xCredSSP you specified the domain name, but during the connection, you are using the server name. These do not match, so CredSSP won't work.
Please check if you are using the FQDN in your data file and try my test code again with the FQDN.

One more detail is that even if the configuration ends up with an error, the profile service application is in fact created. Furthermore, when I run the very same configuration second time on the farm, it completes with no error (probably it doesn't apply any change since all the resources were already applied on the first iteration).

@ykuijs , sorry, I read your message with delay. Here is the output when I use FQDN:
PS C:\Windows\system32> $farm = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\Windows\system32> $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authentication CredSSP -Credential $farm

New-PSSession : [SP201601sp01.sp.local] Connecting to remote server SP201601sp01.sp.local failed with the following
error message : The WinRM client cannot process the request. A computer policy does not allow the delegation of the
user credentials to the target computer because the computer is not trusted. The identity of the target computer can
be verified if you configure the WSMAN service to use a valid certificate using the following command: winrm set
winrm/config/service '@{CertificateThumbprint="<thumbprint>"}'  Or you can check the Event Viewer for an event that
specifies that the following SPN could not be created: WSMAN/<computerFQDN>. If you find this event, you can manually
create the SPN using setspn.exe .  If the SPN exists, but CredSSP cannot use Kerberos to validate the identity of the
target computer and you still want to allow the delegation of the user credentials to the target computer, use
gpedit.msc and look at the following policy: Computer Configuration -> Administrative Templates -> System ->
Credentials Delegation -> Allow Fresh Credentials with NTLM-only Server Authentication.  Verify that it is enabled and
configured with an SPN appropriate for the target computer. For example, for a target computer name
"myserver.domain.com", the SPN can be one of the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com. Try the
request again after these changes. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:6
+ $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authenticatio ...
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108124,PSSessionOpenFailed
PS C:\Windows\system32> Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.SharePoint.PowerShell -ea 0 }
Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:1 char:25
+ Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.Shar ...
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

PS C:\Windows\system32> Invoke-Command -Session $s -ScriptBlock { get-spfarm }
Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:1 char:25
+ Invoke-Command -Session $s -ScriptBlock { get-spfarm }
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

PS C:\Windows\system32> Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
Invoke-Command : Cannot validate argument on parameter 'Session'. The argument is null or empty. Provide an argument
that is not null or empty, and then try the command again.
At line:1 char:25
+ Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
+                         ~~
    + CategoryInfo          : InvalidData: (:) [Invoke-Command], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeCommandCommand

PS C:\Windows\system32> Get-PSSession | Remove-PSSession
PS C:\Windows\system32>

You can test the CredSSP Settings with "Get-WSManCredSSP" - this should return the currently applied CredSSP settings for the current machine.

Hi Brian,
Here is the output of the command you suggested:
PS C:\Windows\system32> Get-WSManCredSSP
The machine is configured to allow delegating fresh credentials to the following target(s): wsman/*.sp.local
This computer is configured to receive credentials from a remote client computer.
PS C:\Windows\system32>

@shurick81 OK so I'm seeing two conflicting messages here. In the original error message I see that it says that the computer is not trusted, but then in the output of that command says it should tag anything in the sp.local domain as being trusted, so I'm not quite sure how it can be doing that as one of them has to be wrong.
Can you try changing the delegate computer property to just "*" when you set the settings, as to allow anything to connect to it. I just want to rule out the possibility that it somehow is filtering the server wrongly.

Hello again. Sorry for a long delay one more time but then again, I decided to put some more efforts before I disturb you.
I struggled with changing this settings on existing machine: every time I called Get-WSManCredSSP, I got
wsman/*.sp.local so I ended up with creating a few new boxes from the scratch with different settings.
So here is what I got:
new machine with both AD and SharePoint installed:
PS C:\Users\_spadm> Get-WSManCredSSP
The machine is configured to allow delegating fresh credentials to the following target(s): wsman/*
This computer is configured to receive credentials from a remote client computer.

PS C:\Users\_spadm> $farm = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\Users\_spadm> $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authentication CredSSP -Credential $farm
New-PSSession : [SP201601sp01.sp.local] Connecting to remote server SP201601sp01.sp.local failed with the following
error message : The WinRM client cannot process the request. A computer policy does not allow the delegation of the
user credentials to the target computer because the computer is not trusted. The identity of the target computer can
be verified if you configure the WSMAN service to use a valid certificate using the following command: winrm set
winrm/config/service '@{CertificateThumbprint="<thumbprint>"}'  Or you can check the Event Viewer for an event that
specifies that the following SPN could not be created: WSMAN/<computerFQDN>. If you find this event, you can manually
create the SPN using setspn.exe .  If the SPN exists, but CredSSP cannot use Kerberos to validate the identity of the
target computer and you still want to allow the delegation of the user credentials to the target computer, use
gpedit.msc and look at the following policy: Computer Configuration -> Administrative Templates -> System ->
Credentials Delegation -> Allow Fresh Credentials with NTLM-only Server Authentication.  Verify that it is enabled and
configured with an SPN appropriate for the target computer. For example, for a target computer name
"myserver.domain.com", the SPN can be one of the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com. Try the
request again after these changes. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:6
+ $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authenticatio ...
+      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2144108124,PSSessionOpenFailed

Then I installed machine that is separate from DC:
PS C:\Users\_spadm> Get-WSManCredSSP
The machine is configured to allow delegating fresh credentials to the following target(s): wsman/*
This computer is configured to receive credentials from a remote client computer.
PS C:\Users\_spadm> $farm = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\Users\_spadm> $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authentication CredSSP -Credential $farm
PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.SharePoint.PowerShell -ea 0 }
PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { get-spfarm }

Name                                                    Status                    PSComputerName
----                                                    ------                    --------------
SP_Config                                               Online                    SP201601sp01.sp.local

PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
PS C:\Users\_spadm> Get-PSSession | Remove-PSSession

And finally this is what I get when running on the machine with a proper *.sp.local domain:
PS C:\Users\_spadm> Get-WSManCredSSP
The machine is configured to allow delegating fresh credentials to the following target(s): wsman/*.sp.local
This computer is configured to receive credentials from a remote client computer.
PS C:\Users\_spadm> $farm = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\Users\_spadm> $s = New-PSSession -ComputerName SP201601sp01.sp.local -Authentication CredSSP -Credential $farm
PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { add-pssnapin Microsoft.SharePoint.PowerShell -ea 0 }
PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { get-spfarm }

Name                                                    Status                    PSComputerName
----                                                    ------                    --------------
SP_Config                                               Online                    SP201601sp01.sp.local

PS C:\Users\_spadm> Invoke-Command -Session $s -ScriptBlock { get-spcontentdatabase }
PS C:\Users\_spadm> Get-PSSession | Remove-PSSession

So the issue seems to only appear for single box installations (AD + SharePoint)
Then I checked that it even works when I use short server name:
$s = New-PSSession -ComputerName SP201601sp01 -Authentication CredSSP -Credential $farm
I will update you when I check how it works with profile service.

OK that's good to know - one thing worth highlighting though is that installing SharePoint on a domain controller isn't a supported configuration for production environments. The quote below is from https://technet.microsoft.com/en-us/library/ee667264.aspx

SharePoint 2013 does not support installation on to a domain controller in a production environment. Additionally, SharePoint 2013 does not support installation on to a domain controller when using the sandbox service in developer, test, or demo environments.

Since SharePointDsc is something we have built for use in production environments, we can't account for behavior like that. Now that being said, once we ship v2.0 and we don't have a dependency on CredSSP it all "should" work on the single server setup - but in reality it's not a scenario we will formally support.
So let us know how you go, but I suspect it will work fine when running on servers that aren't also domain controllers. If that's the case, we can close this off as "not supported" - but if its still not working on a server that isn't also a domain controller then we can keep investigating with you.

This is very good to know! Was you previous environment also using a single server with AD and SharePoint combined?
I guess: If you try to start a session on your domain controller to your domain controller, the same error appears? So on your DC, run:
$farm = Get-Credential
$s = New-PSSession -ComputerName <dcname>.sp.local -Authentication CredSSP -Credential $farm

Since this Powershell/CredSSP/Domain Controller related, we will close the issue. Let us know if you have any other questions/suggestions!


Fix test teardown

We weren't stopping the correct set of applications as well as
forgetting to unload meck. I've also changed the test generators so that
they execute all of the provided assertions.
Fix #687

+1 once Travis is green

+1
make eunit apps=mango,mem3 suites=mem3_sync_security_test passes now and it failed before.

All green.


Update style.css

I have changed the font style of text in the "Our Accomplishment " section and also use media queries to enhance the UI for the small sizes also.
Closes
Closes #723
Fix #724
Screenshots

Checklist:

[x] I have linked the PR to the correct issue.
[x] I have read the Contribution Guidelines
[x] I have built and tested the changes, and they do not break or show any errors.

@akshathar-navada How to fix this failed check?

There will be 3 checkpoints when u open a PR, that has to be ticked!


signuppopup should go away after 'login' or 'signup' is clicked (with route change)

As of right now, clicking anywhere outside of signuppopup makes it disappear - however, taking action such as clicking 'signup' or 'login' currently does not send the modal away. This should not be the case.
Initial investigation into this issue lead me to http://stackoverflow.com/questions/23762323/is-there-a-way-to-automatically-close-angular-ui-bootstrap-modal-when-route-chan I attempted to implement the top solution by updating the .run statement in my app.js file. This did not work.
Next, I tried adding ng-click="closeModal()" to each URL in signuppopup.html, as in Sign up to save your mix! This also did not work, despite the fact that NavCtrl is the controller for that file, and that the closeModal function in NavCtrl appears to be written correctly according to the documentation provided by angular.
At this point I am stepping back from this issue to try fixing it again later. Submitting here to document my progress and also to separate it out as a different issue from issue #43.

No longer relevant as login/save functionality has been removed.


Support networking

This will be a meta-issue for tracking the path towards a functional net package in the standard library.
As of Jul 24, the net package can be imported and compiled, but all methods are stubbed out to return errors.
The net package will depend on a functional Xen network device driver and some kind of TCP/IP stack.
Device driver
Package atman/net implements a network device driver for Xen. It can currently be imported and used to read and write network packets, as shown in this example.
TCP/IP stack
Package atman/net/ip implements a TCP/IP stack. It currently only contains basic types for decoding network packets.
Going forward, we may incorporate gonet into the project for this component.

Right now (997c34c4be19738674b3299dda59b067fab1e431) src/net/net_atman.go is just a stub.

The following may be of interest to explore:

picoTCP
mirage-tcp

Also @justincormack @avsm might have some thoughts on this. NetBSD TCP/IP stack would be a good fit, I am not able to find an appropriate link right now, besides from anything obvious on NetBSD itself...
I've worked with uIP and lwIP previously, but would rather recommend picoTCP over it. The authors, @danielinux and @maximevince, might be interested in this also.

Thanks for the pointers, @errordeveloper!
I think, though, that we do want to build or incorporate a Go TCP/IP stack into the project.
The goal of Atman is not only to run Go programs on a hypervisor, but also to do it in Go, and leverage the things that come along with that (e.g., cross-compiling).
I think bringing in an existing TCP/IP stack would also be pretty large undertaking, because there's currently no C, cgo, or even libc in the project.
The more promising way forward using existing tools, at the moment, is using gonet. Some folks at GopherCon had some promising experiments combining it with Atman.

@sputnik13 Good questions. I've updated the issue description to better describe the current state of things.

I think bringing in an existing TCP/IP stack would also be pretty large undertaking, because there's currently no C, cgo, or even libc in the project.

@bernerdschaefer  just FYI, the requirements in terms of libC of a stack like picoTCP is very limited, as it's originally designed to run on bare-metal embedded applications, and the porting/cross compiling impact is very low.
Also, at a first glance, the protocols implementation in gonet seems a bit basic/rudimental at the moment, if compared to picoTCP.
However, not crossing with external C dependencies seems a valid point in your case.

At the risk of gross oversimplification, to have full networking capabilities, the protocols required are
L2 - Ethernet, ARP, NDP
L3 - IPv4/IPv6
L4 - TCP, UDP, ICMP
All other application level protocols can then be built on top of these and there's value in providing a solid core set of protocols and requiring use of libraries for others.
Also, yes I'm taking a very narrow IP over Ethernet view here since that's what the vast majority of use cases are outside of embedded contexts.
Strictly speaking DHCP isn't required but from an "OS" standpoint, especially in the world of clouds, lacking DHCP as a core functionality makes the "OS" less usable, so it would be nice to have DHCP support built in.

@sputnik13 I am not referring to "application layer" protocols. I am talking mostly about TCP. There are many implementations out there, but I haven't seen many that offers good support for timestamps, SACKs, or with a more than mediocre congestion control mechanism. Implementing just RFC791 is not enough in 2016, while gonet seems to support just a subset of that. Moreover, it seems to have no multicast sockets, no IPv6 support, no routing and no IP layer filtering. Please note that I am not criticizing the project as whole, I reckon it's a just very young. I believe it has potential to grow in the good direction especially if it can gain a few more contributors.

@danielinux thanks for the clarification.  I agree all of those are needed for a robust implementation, but I'm kind of interested in just seeing things make progress to a point where real applications can run, even if performance inhibited because congestion control algorithms are subpar, etc.
I assume that would help to drive more interest to the project and bring more people in, which in turn will lead to better implementations of things, all assuming that @bernerdschaefer wants external contributors of course.
Also, I agree keeping everything strictly go is an attractive proposition.

There's been some progress on this over the past few months, and I'm ready to close this out.
I'm not sure when AtmanOS will get a fully integrated net stack, such that you could just net.Listen and everything works. Rather than block on this, I've moved forward with out-of-tree networking where it's easier to experiment.
atmanos/anet is now home to all the code needed to do networking under AtmanOS, with an adapter for google/netstack to provide the upper layers.
There's an HTTP server example in the atmanos/example repo!

Thanks!
On Sat, 28 Oct 2017, 6:11 pm Bernerd Schaefer<EMAIL_ADDRESS>wrote:

There's been some progress on this over the past few months, and I'm ready
to close this out.
I'm not sure when AtmanOS will get a fully integrated net stack, such that
you could just net.Listen and everything works. Rather than block on
this, I've moved forward with out-of-tree networking where it's easier to
experiment.
atmanos/anet https://github.com/atmanos/anet is now home to all the
code needed to do networking under AtmanOS, with an adapter for
google/netstack https://github.com/google/netstack to provide the upper
layers.
There's an HTTP server example in the atmanos/example
https://github.com/atmanos/example/tree/master/netstack repo!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/atmanos/atmanos/issues/9#issuecomment-340205962, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPWS4_1kGPf5-IRRE7jAsyok2K9Vlpeks5sw2BagaJpZM4HP3E2
.


[BUG] Breaking on Opus without proper error reporting on UI level

Describe the bug
Filling out this field will help us investigate the issue efficiently. Providing detailed information allows us to set the appropriate priority. We appreciate your cooperation.
The Claude 3 (Opus) model is currently unavailable due to a throughput limitation. The system is unable to process the request at this time.
Request:
It would be helpful if the actual error message ("The provided model doesn't support on-demand throughput") was displayed directly in the user interface, rather than the generic "An error occurred while responding." This would provide users with more specific information about the issue without requiring them to use web inspection tools for debugging.
To Reproduce
Filling out this field will help us investigate the issue efficiently. Providing detailed information allows us to set the appropriate priority. We appreciate your cooperation.
Steps to reproduce the behavior:

Go to Home screen
Open WebInseptor
open network tab
select WebScoket filter
focus back on the main screen
Click on "Claude 3 (Opus)" (select)
Type "Hi" and press enter
See error: "An error occurred while responding."
check the WebSocket message flow in inspector
find last mesage "Failed to run stream handler: An error occurred (ValidationException) when calling the ConverseStream operation: The provided model doesn't support on-demand throughput."

Screenshots

If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

@wzr1337 Thank you for your suggestion. We'd be appreciated if you could create a PR to improve UI when you have time.

HI, I also get the same error "An error occurred while responding." when after first deployment. Please help to fix that. Thanks.


Question: Include external html view, without compose container ?

When using compose to include the external html view.html segment:

    <compose view="./view.html"></compose>

the compose creates an element around the included view, breaking the css style and more css required to fix that.
Is it possible to include the external html directly, as innerHTML of the element, and how?

Try this

    <compose view="./view.html" containerless></compose>

I actually tried exactly that, before posting question and the result was that the container, with the included view.html probably, is removed, no content. I was not sure if it possibly a bug ?

We'll take a look at it.

Hi Rob,
sorry, it happens with the custom element only, perhaps only when inline view used.
Please find the gist https://gist.run/?id=8914354f532d1daba33f095fa8bb7e79
Thanks and regards,
srdjan


"Image was NaN% different from saved snapshot with undefined different pixels."

Hi,
I get "Image was NaN% different from saved snapshot with undefined different pixels." error with following settings:
Windows 10 / vscode
Cypress version: 13.7.2 (same error with 13.2.0)
@simonsmith/cypress-image-snapshot: 9.0.2
cypress-image-snapshot is the only plugin I'm using (cf: https://github.com/simonsmith/cypress-image-snapshot/issues/26)
// cypress.config.ts
import { defineConfig } from 'cypress';
import { addMatchImageSnapshotPlugin } from '@simonsmith/cypress-image-snapshot/plugin';

module.exports = defineConfig({
   ...
   e2e: {
    experimentalStudio: true,
    experimentalRunAllSpecs: true,
    setupNodeEvents(on) {
      addMatchImageSnapshotPlugin(on);
      on('after:screenshot', (details) => {
        console.log(details);
      });
    },
  },
});

commands.ts
import { addMatchImageSnapshotCommand } from '@simonsmith/cypress-image-snapshot/command';
addMatchImageSnapshotCommand();

tsconfig.ts
{
  "extends": "../tsconfig.json",
  "compilerOptions": {
    "target": "es5",
    "lib": ["es5", "dom"],
    "types": ["cypress", "node", "@simonsmith/cypress-image-snapshot/types"]
  },
  "include": ["**/*.ts"]
}

package.json
  "scripts": {
    "cyopen": "cypress open --env updateSnapshots=true",
  },

TestFile.cy.ts
...
cy.getBySel('myComponent').matchImageSnapshot('snap-test');
...

I have /cypress/screenshots/snap-test.png created with
"Image was NaN% different from saved snapshot with undefined different pixels."

I have the same results with script
"cypress open",

Hmm, I wonder if this was introduced by #51
Can you try 9.0.1 and see if it is still an issue?

Just tried with 9.0.0 and 9.0.1, same issue.

What is the relevancy of the prop "e2eSpecDir"?

That would relate to this part - https://github.com/simonsmith/cypress-image-snapshot?tab=readme-ov-file#snapshot-paths

@simonsmith I found out that if @badeball/cypress-cucumber-preprocessor installed with cypress-image-snapshot.   Error occured (Image was NaN% ...)
Any suggestion for this?
My cypress config:
import { defineConfig } from 'cypress'
import createBundler from "@bahmutov/cypress-esbuild-preprocessor";
import { addCucumberPreprocessorPlugin } from "@badeball/cypress-cucumber-preprocessor";
import createEsbuildPlugin from "@badeball/cypress-cucumber-preprocessor/esbuild";
import { addMatchImageSnapshotPlugin } from '@simonsmith/cypress-image-snapshot/plugin'

export default defineConfig({
  video: false,
  e2e: {
    specPattern: "**/*.feature",
    "baseUrl": "https://local-ditto.devel.kakao.com/",
    env: {
      "ALLOW_SNAPSHOT": process.env.ALLOW_SNAPSHOT === undefined ? false : true
    },
    trashAssetsBeforeRuns: false,
    async setupNodeEvents(on, config) {
      addMatchImageSnapshotPlugin(on)
      await addCucumberPreprocessorPlugin(on, config);
      on(
        "file:preprocessor",
         createBundler({
           plugins: [createEsbuildPlugin(config)],
        })
       );
      return config
    }
  },
})

@eddie0329 Sounds like #50


Investigate public CI

Problem:
External contributors can not interact with our CI system, beyond ✅  / ❌  and even that requires a Shopify employee to kick off the run.  Employees have to remember to run the test suite and external contributors have to wait till an employee provides them with failing test status.
Proposal:
Investigate adding a public CI component. I believe circleci or travis is free to open source projects. We also should consider if this is enough to remove Shopify CI or if we want to run both. If we have to keep internal CI the external CI will need to be easy to maintain, (this isn't trivial)
note: Please do not sink more than an hour or so into this task with reporting back. This might be a yak shave or worse.

Circle-CI might be a good option for this.
helm/charts uses Circle-CI for testing and it works pretty well.
Travis-CI also works well.
Given this is an open source project there would also be no cost for either option.
I would also be happy to volunteer some time to help getting this setup.

Thanks for offering to look into this and sorry for the slow response. We've got some internal discussion going on and think we have a simpler solution than building a new external pipeline. I'll update here once we have some concrete progress.

Closing for now, will revaluate if the amount of external contributions grows.


Error installing fastlane: signet requires Ruby version >= 2.4.0

Why can't I install fastlane?
Unable to install fastlane due to ruby 2.3.3, I can't update ruby to newest versions.
Docker commands that you execute

docker run --name my_jenkins_container jenkins/jenkins:lts
docker container exec -u 0 -i -t my_jenkins_container bash

Inside the container:

apt-get update && apt-get install
echo yes| apt-get install ruby-full
echo yes | apt-get install build-essential
gem install fastlane -NV

Actual result
ERROR:  Error installing fastlane:
signet requires Ruby version >= 2.4.0.

Output of docker version
Client: Docker Engine - Community
Version:           19.03.5
API version:       1.40
Go version:        go1.12.12
Git commit:        633a0ea
Built:             Wed Nov 13 07:22:34 2019
OS/Arch:           darwin/amd64
Experimental:      false
Server: Docker Engine - Community
Engine:
Version:          19.03.5
API version:      1.40 (minimum version 1.12)
Go version:       go1.12.12
Git commit:       633a0ea
Built:            Wed Nov 13 07:29:19 2019
OS/Arch:          linux/amd64
Experimental:     false
containerd:
Version:          v1.2.10
GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version:          1.0.0-rc8+dev
GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version:          0.18.0
GitCommit:        fec3683
Other relevant information

@slide I'm looking for these Jenkins images that are based on Debian Buster for similar reasons as CarlitosDroid. Can you provide a link to them?

So, do you have what you need?

I'll be able to test it tomorrow. Not sure if the change to :slim will require more adaptions.
What other tags in jenkins/jenkins use Debian Buster?

It looks like slim is the only one right now. That will change once some PR's are merged and the main docker image will be Buster instead of Stretch.

Good to know. Thanks for the info :)

@slide The :slim image works well for us. Needed some adjustments, but nothing major :+1:


v0.5.0 release artifacts contain build path and html

The zip files in the release artifacts for v0.5.0 contain the build path in the zip, i.e. docs/build/default/*.json as well as an index.html.  These are intended to be flat zips containing the just data *.json and *.geojson.

Artifacts have been updated so they are 1-folder deep and only contain json and geojson now.


Removal of accent removal / accented letter conversion from latin scripts

At the moment it is not possible to use libpostal to clean addresses in a lot of European languages, as it automatically converts all accented letters to either their unaccented counterparts or into a combination of non-accented ascii letters (example: ä -> a or ä->ae). To my knowledge, it is somewhat allowed to do the latter transformation in German language (Jürgen -> Juergen), but it is an absolute no-no in any human-facing text for example in the Scandinavian languages and French, or my native Finnish. (Example: Pitkämäentie <> Pitkaemaeentie). Also, most free or commercially available geocoding services do not understand this translitteration and thus they won't find anything based on the street name.
It seems the transliteration is buried very deep in the code, perhaps also in the NLP training and there is no option to switch it off at present.
Would it be a big effort to 1) make the transliteration optional or 2) (better) remove it completely and retrain the entire dataset using the proper accents?

As it turns out, you're in luck.
In the parser-data branch of libpostal, the branch I'm primarily developing in at the moment, I've made umlaut transliteration language-specific in expand_address (normalizations like ü => ue only happen if the detected/specified language is German), and added a new transliterator called "latin-ascii-simple" which only transliterates punctuation, various types of hyphens, HTML entities, etc.
This approach will allow the parser to make only minimal modifications/sanitizations to the input string, possibly without even needing to lowercase the string used in the output.
Transliteration isn't buried very deep. It's a simple API, may even be able to be factored into its own library at some point. The tricky part is ensuring that the parser works just as well on e.g. Finnish text typed without accent characters (I'm aware that in many languages they're actually different letters of the alphabet, just referring to the Unicode designation) as with proper spelling. The current parser model is still trained on transliterated words, so it won't help to switch branches, but in the new model, the strategy will be to train on up to 3 different versions of every string: the original, correct local-language version (e.g. München), a version with Latin-ASCII transliteration (Muenchen, only for German, not any of the Scandinavian or Turkic languages), and a version with accents stripped (Munchen) so every spelling will at least be seen by the model and have a similar weight. The point of doing all of this is so that the model is robust enough to handle strings typed by non-native speaker (the "ignorant American spelling"), or by native speakers using a foreign keyboard, or for records from some database where the accents may have been stripped by an automated process, etc.
The new transliterators are not going to be in master for a little while, and the parser has to be fully retrained to take advantage of them, which takes about a full week end-to-end.
For people running their own geocoders, I usually suggest that strings be run through libpostal normalization at index time so the normalized strings returned from the parser can be found, but obviously that's not a possibility when using a third-party geocoder.

This is great news! I will definitely give it a try once the training data is available.
And I completely agree that it is a good idea to train the system both with accents and with accents stripped, as a lot of legacy data may have been entered in the 7bit ASCII times or may have been translitterated to that earlier.

Hey @jheimone1, libpostal 1.0 has been released into master, and no longer transliterates
As mentioned, there's a new transliterator which only makes simple modifications to the input (HTML entity normalization, NFC unicode normalization). Latin-ASCII transliteration is no longer used at runtime. Instead, addresses are transliterated to multiple forms during training so the parser has to deal with all the variants rather than normalizing to a single variant (which previously was not even correct in cases like Finnish, Turkish, etc.) in both places.
In expand_address, there are also language-specific transliterators for German and the Scandinavian languages, which would not even apply in the case of Finnish.


langchain RAG

as alternate to #95 #26 or #33
https://templates.langchain.com/?integration_name=rag-chroma-multi-modal

I think #126 is the way to go.


Document "Copy link to stream" mobile app feature

In the mobile app, stream links can be copied via the stream long-press menu.
The stream long-press menu can be accessed via the list of streams, the stream name in the "inbox" (unreads) view, or the top bar in a stream narrow.
We should document this feature on https://zulip.com/help/link-to-a-message-or-conversation#get-a-link-to-a-specific-stream.
Instructions for accessing the long-press menu should be added using macros in templates/zerver/help/include for reusability.
Related issues: https://github.com/zulip/zulip/issues/22147 (PR: #22158), #22196

@zulipbot claim

Fixed by #22158
@zulipbot abandon

@zulipbot claim
On Mon, Jun 20, 2022, 12:44 PM David C Priest @.***>
wrote:

Fixed by #22158 https://github.com/zulip/zulip/pull/22158
@zulipbot https://github.com/zulipbot abandon
—
Reply to this email directly, view it on GitHub
https://github.com/zulip/zulip/issues/22197#issuecomment-1160658846, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR4FAWNFXULRZQG2NLQDKDVQCNWXANCNFSM5YAL75FQ
.


[API] $.import - assignment to the $ object

It is expected the following code to be working:
importTest.xsjs:
$.import("test", "calculator");
let calculator = $.test.calculator;

$.response.setBody("Result is: " + calculator.add(5, 3));

However an error message is returned after executing the importTest.xsjs

TypeError: Cannot read property "calculator" from undefined

Sample project attached - test.zip.
Related to: https://github.com/SAP/xsk/issues/18

Currently the workaround is to use the following syntax instead:
let calculator = $.import("test", "calculator");

$.response.setBody("Result is: " + calculator.add(5, 3));

Progress tracked in: https://github.com/SAP/xsk/issues/534
Fixed with: https://github.com/SAP/xsk/pull/558


Cannot use native objects

In vue components,like this use:

  created(){
    console.log(document)
  }

result:  500 | Internal Server Error
How do I solve

document is only available in browsers. Since Nuxt uses server side rendering (SSR), it doesn't recognise this property and throws an error.
You can solve this by using the mounted() method instead.

@stephan281094
I tried, still won't do,but still thank you

In that case, do you have any error message/ stack trace?

http://ssr.vuejs.org/en/universal.html


fix: fixed default theme for createTailwindConfig

#271  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bob-obringer and the rest of your teammates on  Graphite


AppCenter.Start throws exception "Specified cast is not valid"

In one of my WPF apps I get the exception "Specified cast is not valid" when calling AppCenter.Start().
The same error comes if I try with AppCenter.Configure().
I have a different WPF app that does not throw any exception (however data is missing in AppCenter for that one.)
I am using an Organization for both apps. I am only using AppCenter for diagnostics, not for building or deploying apps.
Why do I get this exception and how can I fix it?

Now I'm seeing events in App Center. For some reason it only works with the installed version of the app, not when debugging in Visual Studio.


chore(main): release 1.4.0

:rocket: New release
1.4.0 (2023-06-21)
:bug: Bug Fixes

deprecated Twitter post format (#67) (8968732)

:sparkles: Features

add forgejo icon (#69) (468682a)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/hugo-sid/hugo-blog-awesome/releases/tag/v1.4.0 :sunflower:


Add a filter shape compensation

ATA data uses a 4-tap per coarse channel pfb that leaves a pretty distinct filter shape that leaves a non-uniform noise floor. Add a noise-floor linearizer / filter shape compensation method.
Some sane approaches off the cuff:

Pass through the known filter shape and subtract out the expected shape to get a uniform noise floor
Estimate / fit some curve or parameter to the actual noise floor shape, and remove that. This is adaptive which might make some folks cringe

Check in with Wael to see if he has specific preferences

Fixed in #69


Regression Testing

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/413
Originally created by @valexieva on 2019-02-25 00:32:30

Problem to solve
(Summarize the problem we are trying to solve in the form of we need to do [A], so that [B] can [C])

Currently no systematic way to detect regressions. We need to have a set of automated test currently in place for every function that one can use in the CLI, so that we ensure that users can trust the bare minimum functionality we have works. (we pushed a name change to master => meltano discover all doesn't work, Cannot list available plugins. Error: 'models')

Target audience
(For whom are we doing this? Include a persona)

For anyone who might be using the CLI + ourselves

Further details
(Include use cases, benefits, and/or goals)

Test commands execute and return expected values
Test individual functions for validity of inputs and outputs

Proposal
(How are we going to solve the problem? Try to include the user journey)

Add well-defined test cases to each feature and bug. Given the small amount of features we currently have, each command should be executed and should return expected values.

What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)

AC: There is a suit of automated test that run weekly with clear ownership and bugs get fixed by next release
Success Metrics: Any failure is caught before release through reg tests; No more issues with regards to reg tests as this should be part of ongoing software development

Links / references
Please note that this was taken from GitLab, to be changed accordingly
@jschatz1 - please write more detailed proposal of specific reg test
FYI: @mbergeron @derek-knox @bencodezen @cpetren

View 19 previous comments from the original issue on GitLab


chore(main): [bot] release stellio-context-broker:0.1.1

:robot: I have created a release beep boop
0.1.1 (2023-05-02)
Miscellaneous Chores

stellio-context-broker/artifacthub-images: Update images in 'Chart.yaml' (#18) (b01d3d2)

This PR was generated with Release Please. See documentation.

:robot: I have diffed this beep boop
"/$namespace/$kind/$name.yaml" for normal resources
"/$namespace/HelmRelease/$name/$namespace/$kind/$name.yaml" for HelmReleases <- this is recursive
'null' means it's either cluster-scoped or it's in the default namespace for the HelmRelease
charts/stellio-context-broker/values.yaml has no changes
charts/stellio-context-broker/ci/artifacthub-values.yaml has no changes


Fix IE9 console issue

This is a similar commit trying to solve IE9 console issue addressed once by #185
Also, the other commit fix a lint issue introduced by #205

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@539dab5). Click here to learn what that means.
The diff coverage is 0%.

@@           Coverage Diff           @@
##             master   #259   +/-   ##
=======================================
  Coverage          ?    82%
=======================================
  Files             ?      5
  Lines             ?    150
  Branches          ?      0
=======================================
  Hits              ?    123
  Misses            ?     27
  Partials          ?      0

Impacted Files
Coverage Δ

src/defaults.js
75% <ø> (ø)

src/core.js
73.33% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 539dab5...2e26dab. Read the comment docs.


Remote opencd

I use Raspberry PI as programmer for NRF 52832 chips.
Now I installed Zephyr on my PC and switched my project to this platform.
I see openocd is supported by Zephyr but it seems to me only for local use.
Is there a way to automate deployment with west tool using opencd listening on remote 4444 port of my Raspberry?
Thanks.

I don't know what this would take, sorry. I personally do not use openocd very much. Can you explain what you would do with openocd to get this done without using west?

I've switched my work-related account to @mbolivar-nordic (https://github.com/zephyrproject-rtos/zephyr/pull/23530).

Sorry, at the moment I can't give you exact command to execute.
Using command line I do it in 2 steps but using Eclipse it works automatically in remote mode.
I should investigate deeply how it works behind the scene. I'll post the instructions when I understand it. Sorry for the delay.

There's no rush. Thank you for your response.

We have no plans to implement this functionality. Please send a Pull Request if you have interest in this becoming part of the tree.


fix dates collisions

There is an issue where If you have events where one ends when another is started, you will get different containers.
Example (in minutes):
[
    {start: 540, end: 550},
    {start: 550, end: 560},
    {start: 560, end: 570},
]

Right now. They are 3 different event containers.
After the fix, They are nested.
[
    {
        start: 540,
        end: 550,
        rows: [
            {
                start: 550,
                end: 560,
                leaves: [
                    {
                        start: 570,
                        end: 580
                    }
                ]
            }
    },
]

Just like in google calendar.

Hi, thanks for the PR. Maybe I misunderstood you, but to me it looks like Google Calendar treats them as different containers?

I checked that. You are right with those dates, but If event ends when another start AND this event is under 30minutes They would be nested.

You are right with those dates. But I checked some possibilities. As far as I understand.
If a difference between eventA.start and eventB.start is under 30 minutes They are a container with row. If We have also with eventC.start difference between eventA.start under 30 minutes it can become leaf.

Check events at 7pm

Ah, I see! I don't mind a change as long as the current examples still work.

@tobiasandersen
I understand You, but please consider some changes, because If you have 3 short events (for example 10 minutes) one after another They are quite hard to read.

I change the code. For me, it behaves like in Google Calendar. Please check that. I only tested that manually. Did not write tests.

There was the new release at 23 Mar 2018 which change code structure so I moved my change to the new place.
Examples:
Google Calendar:

v0.19.0 before fix

v0.19.0 after fix

Looking great! 🎉


Remove unused import

Removing an unused import for pre-1.7 users that led to compilation errors.

Coverage remained the same at 40.59% when pulling 1038efecd6d54ba60f9fcf0bf760312b2406121e on MixedMessages:unused_import_1.6 into d3e4c168c015141462183d09ea00fc6e79eafba7 on NYTimes:master.


Documentation: sap.ui.core.IAsyncContentCreation is not mentioned in "Use Asynchronous Loading" of the Best Practices for App Developers

There is no any mention of the recently introduced sap.ui.core.IAsyncContentCreation interface on Use Asynchronous Loading.

To also load all application-specific configuration settings asynchronously, set the async property in the metadata of the manifest.json file to true.

Besides that, with this interface implemented, there is no need to use "async": true in rootView.
Version 1.95.0.

Hi @pubmikeb,
thanks for finding this missing piece.
We are currently working on a similar documentation clean-up.
We will update this issue once the documentation is updated.
BR,
T

Hi @Thodd,
Please, don't forget to update the source code of demo apps and controllers samples as well!

Hi @pubmikeb,
Use Asynchronous Loading has been updated!
Best regards,
Tommy

Use Asynchronous Loading has been updated!

@devtomtom, thanks a lot for the update!
I just want to add a reference to another async-related issue: https://github.com/SAP/openui5/issues/3333.
There a developer can find a great explanation regarding IAsyncContentCreation vs. data-sap-ui-async="true" vs. sap.ui5."async": true vs. routing."async": true.
So, it might be clearer the reason of existence of several similar properties across the framework.


Add ENCODE blacklists

Add ENCODE blacklists for assemblies where available:
https://sites.google.com/site/anshulkundaje/projects/blacklists
These were originally generated relative to UCSC genome builds. The files can be added to a new folder called Annotation/Mappability/. Alternatively, they could be added to the existing Sequence/AbundantSequences/ but the files dont really contain any actual sequence so its probably not appropriate.

GRCh38

https://raw.githubusercontent.com/nf-core/atacseq/master/assets/blacklists/hg38-blacklist.bed
Can be added to "<IGENOME_BASE>/Homo_sapiens/NCBI/GRCh38/Annotation/Mappability/"
May need to rename this to GRCh38-blacklist.bed

hg19

https://raw.githubusercontent.com/nf-core/atacseq/dev/assets/blacklists/hg19-blacklist.bed
Can be added to "<IGENOME_BASE>/Homo_sapiens/UCSC/hg19/Annotation/Mappability/"

mm10

https://raw.githubusercontent.com/nf-core/atacseq/dev/assets/blacklists/mm10-blacklist.bed
Can be added to "<IGENOME_BASE>/Mus_musculus/UCSC/mm10/Annotation/Mappability/"

GRCh37

https://raw.githubusercontent.com/nf-core/atacseq/dev/assets/blacklists/GRCh37-blacklist.bed
Can be added to "<IGENOME_BASE>/Homo_sapiens/Ensembl/GRCh37/Annotation/Mappability/"

GRCm38

https://raw.githubusercontent.com/nf-core/atacseq/dev/assets/blacklists/GRCm38-blacklist.bed
Can be added to "<IGENOME_BASE>/Mus_musculus/Ensembl/GRCm38/Annotation/Mappability/"

mm9

http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/mm9-mouse/mm9-blacklist.bed.gz
Can be added to "<IGENOME_BASE>/Mus_musculus/UCSC/mm9/Annotation/Mappability/"

ce10

http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/ce10-C.elegans/ce10-blacklist.bed.gz
Can be added to "<IGENOME_BASE>/Caenorhabditis_elegans/UCSC/ce10/Annotation/Mappability/"

dm3

http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/dm3-D.melanogaster/dm3-blacklist.bed.gz
Can be added to "<IGENOME_BASE>/Drosophila_melanogaster/UCSC/dm3/Annotation/Mappability/"

These files can be used in conjunction with the liftOver tool to make them available for other assemblies for the same organism, or by changing the chromosome identifiers to map between Ensembl and UCSC ( I have done this already for the latest human and mouse Ensembl genomes). However, for now I think this would be a good start!

I actually have some nice applications for nf-core/eager for these (human) ones too :-)

Just a heads up. The GRCh38 version of the file above is for the NCBI build. Not sure where things are with the Ensembl build but if they are the same assembly then it could be added here too: "<IGENOME_BASE>/Homo_sapiens/Ensembl/GRCh38/Annotation/Mappability/" with some chromosome identifier replacement.


[RMP] Support Offline Batch processing of Recs Generation Pipelines

Problem:
As a user, I would like to run my merlin systems inference pipeline in an offline setting. This will allow me to produce a set of recommendations for all users to be served from a data store, email campaign, etc. I will also be able to conduct rigorous testing and better compare behaviors against other systems, at both operator and system level.
Goal:
To do this I need to be able to run my merlin systems inference graph without using triton or the configs generated for it. It will require a new operator executor class that runs the ops in python instead of tritonserver. The execution should behave exactly as it does in the tritonserver setting, meaning each operator should be provided same inputs, and return same outputs.

Run an Inference operator graph without tritonserver.
Does not require any new user-facing API changes.
Execute the same graph, that would be deployed to tritonserver.
Execute in Python process

Constraints:

Use the same merlin systems graph/ops that were created for inference pipeline, that would run on tritonserver
Swap out the operator executor to python version (non-triton).
Allow for all types of graphs, supporting multiple chains and parallel running of ALL available operators.

TODO:
Core

[x] https://github.com/NVIDIA-Merlin/core/pull/140
[x] https://github.com/NVIDIA-Merlin/core/pull/141
[x] https://github.com/NVIDIA-Merlin/core/pull/143
[x] https://github.com/NVIDIA-Merlin/core/pull/146

Systems

[x] https://github.com/NVIDIA-Merlin/systems/pull/204
[ ] Validate that we can run a systems ensemble on Dask

Issues

[x] #461
[x] #462
[x] #463
[x] https://github.com/NVIDIA-Merlin/Merlin/issues/505
[ ] https://github.com/NVIDIA-Merlin/Merlin/issues/506
[ ] https://github.com/NVIDIA-Merlin/Merlin/issues/507

Example

[ ] #798

Assignees will be Karl / Adam.

This is a prerequisite for cross-FW evaluation

My impression is that batch inference for models is required for cross-FW evaluation, not the full batch inference for a system. The additional steps in the Systems' computation graph (QueryFeast, QueryFaiss, Softmax, filtering, etc) would likely not be required for batch inference on a single Model. Batch inference for the model would have a simpler "training data in -> predictions out" process, which would likely be a step in the Systems graph.
Perhaps we should first build the batch inference functionality (apply nvt transform + use model to predict) including the output format schema, and then that functionality could be shared in cross-FW evaluation and systems-wide batch prediction.

We do have some batch prediction functionality for models already, but it's not quite structured in a way that would make it a reasonable foundation for batch processing of graphs. I think we could massage it in that direction though and try to standardize how batch graph processing works in Merlin Core by taking what exists and refactoring it in the right direction.

@karlhigley do you think we should add an example for it?

I think we should add an example for every new piece of significant functionality (i.e. almost all roadmap issues.)

https://github.com/NVIDIA-Merlin/core/pull/352
https://github.com/NVIDIA-Merlin/systems/pull/376

@jperez999 Could you add appropriate tasks to the list in the description?

Need to be able to swap out certain operators, based on runtime. I.e. when running daskexecutor for offline batch it is not necessary to run the feature store operator unless we are testing against it. You could run a dataset merge operator instead using offline features stored in a parquet file.  Please refer to task list created for further tracking


Remove mutable default arguments

For example reqHistoricalData has a mutable default argument.  I won't list them all here, but a simple PR is required to replace any such cases with immutable default args. Since the code is typed, it will be easy to search for e.g. : List[  and find some of them.
I'm happy to contribute the PR

Good catch.
It looks like most of cases are only parameters for sending data, so technically it's "safe" because the methods never modify the parameters, but still good to be more correct overall.
Feel free to update and add a PR. Just use the next branch for updates.
The data sending API treats empty lists and None as the same value (both just send ""), so modification should be as simple as updating defaults to list[thing] | None = None almost everywhere.


Create Intel Container Image

Is your feature request related to a problem? Please describe.
The only container template we have is docker-ubuntu-gcc-openmpi. There should be an Intel container as well.
Describe the solution you'd like
Create Intel container template based off of https://hub.docker.com/r/intel/oneapi-hpckit

@kgerheiser I spoke with @mark-a-potts last week. He said that he already built a container using Intel oneAPI (I think he installed the compiler via spack). Let's make sure we don't reinvent the wheel.
Also tagging @mer-a-o for her awareness. Maryam will be working on a clang+gfortran container next, and update the existing gnu container so that it works correctly. So far, I have only built the container, not tried to run anything with it.

@climbfuji this is related to that because @mark-a-potts brought it up and I was talking @ulmononian about containers as well.

I currently have a oneapi-based container with spack-stack installed here--docker://noaaepic/ubuntu20.04-spack-stack:latest. That container required manual intervention to build, however. I used spack to install both compilers and MPI because the oneapi-hpckit recipe referenced would not run correctly for me. I am working on a recipe.


fix for vertical scrollable modifiers

The number of months was incorrectly assumed to be constant in addModifier for the VERTICAL_SCROLLABLE orientation. I switched it to use the number of months in visibleDays. I was going to switch it for all orientations, but then decided this would be less error prone (smaller change).
@majapw

Coverage increased (+0.2%) to 84.654% when pulling 4174bd26cff32dde36281faf2e2230081b6d679c on amh-fix-vertical-scrollable-modifiers into 7fb63a3850343874147bd045dbaf0865da7beeea on master.

@amhunt is this a duplicate of #1261?

I've rebased this on top of #1261; so merging this one can close both.


Bug: Type complaining issues with mux-player 1.0.0

Checklist

[x] Verified whether your issue exists in the latest version Mux Element(s) release: npm install @mux/[element_name] or yarn @mux/[element_name].

(NOTE: If the issue is not occurring in the latest version, we may not resolve it.)

[x] Confirmed that this issue hasn't already been filed.

Which Mux Elements/Packages has a bug?
mux-playback , mux-player

Which browser(s) are you using?
N/A - Compile time issue
Which operating system(s) are you using?
MacOS
How are you using Mux Elements?
npm package, in an angular typescript project
Describe the bug

Steps To Reproduce

install mux-player
Attempt to compile angular project after including mux-player element

Link to code example:

The current behavior
Getting a compile time error
Error: node_modules/@mux/playback-core/dist/types/index.d.ts:5:28 - error TS1005: ',' expected.

5 import { StreamTypes, type ValueOf, type PlaybackCore, type MuxMediaProps, type MuxMediaPropsInternal } from './types';
                             ~~~~~~~

Error: node_modules/@mux/playback-core/dist/types/index.d.ts:5:42 - error TS1005: ',' expected.

5 import { StreamTypes, type ValueOf, type PlaybackCore, type MuxMediaProps, type MuxMediaPropsInternal } from './types';
                                           ~~~~~~~~~~~~

Error: node_modules/@mux/playback-core/dist/types/index.d.ts:5:61 - error TS1005: ',' expected.

5 import { StreamTypes, type ValueOf, type PlaybackCore, type MuxMediaProps, type MuxMediaPropsInternal } from './types';
                                                              ~~~~~~~~~~~~~

Error: node_modules/@mux/playback-core/dist/types/index.d.ts:5:81 - error TS1005: ',' expected.

5 import { StreamTypes, type ValueOf, type PlaybackCore, type MuxMediaProps, type MuxMediaPropsInternal } from './types';
                                                                                ~~~~~~~~~~~~~~~~~~~~~

Apologies for the short bug report. I've attempted to clear down node modules and reinstall however that didn't resolve for me either...

thanks for the feedback! it looks like the used bundler doesn't support the inline type keyword.
do you happen to know which bundler this Angular project is using? esbuild, rollup, or something else?

Interestingly, I had the exact same pop up just now in Next.js 11 and 12, after updating from a sub 0 version of mux player to latest. I'm still investigating.

Upgrading typescript from 4.3.x to 4.8.x did the trick over here.

Going to close this and advise that folks on TS 4.3 upgrade to 4.8, I don't think there are major blockers that would prevent people from doing that upgrade


trace memory of MemTables

Signed-off-by: tabokie<EMAIL_ADDRESS>Ref #205
Added a new thread to periodically report internal metrics, including memory used by MemTables.

I ran a benchmark to get a baseline memory usage:
# ./target/release/stress --path /data3/re_bench --time 60 --regions 1000 --write-threads 1 --purge-interval 0 --compact-count 0 --write-without-sync
Memory usage: 8003MiB


[SPARK-21243][Core] Limit no. of map outputs in a shuffle fetch

What changes were proposed in this pull request?
For configurations with external shuffle enabled, we have observed that if a very large no. of blocks are being fetched from a remote host, it puts the NM under extra pressure and can crash it. This change introduces a configuration spark.reducer.maxBlocksInFlightPerAddress , to limit the no. of map outputs being fetched from a given remote address. The changes applied here are applicable for both the scenarios - when external shuffle is enabled as well as disabled.
How was this patch tested?
Ran the job with the default configuration which does not change the existing behavior and ran it with few configurations of lower values -10,20,50,100. The job ran fine and there is no change in the output. (I will update the metrics related to NM in some time.)

Test build #78984 has started for PR 18487 at commit d60a0be.

Test build #78984 has finished for PR 18487 at commit d60a0be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78984/
Test PASSed.

@rxin @cloud-fan Can you review this PR?

Test build #79290 has started for PR 18487 at commit ad11f64.

Test build #79292 has started for PR 18487 at commit 6c12854.

Test build #79290 has finished for PR 18487 at commit ad11f64.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79290/
Test PASSed.

Test build #79292 has finished for PR 18487 at commit 6c12854.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79292/
Test PASSed.

Merged build finished. Test PASSed.

@jiangxb1987 I have made the changes requested. Can you have a look. Thanks.

This LGTM, @dhruve could you rebase it with the master branch please?

Test build #79469 has started for PR 18487 at commit d92c05c.

@jiangxb1987 I have resolved the merge conflicts and reworded the config to make it more clear.

Test build #79469 has finished for PR 18487 at commit d92c05c.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79469/
Test FAILed.

Jenkins, test this please

Test build #79477 has started for PR 18487 at commit d92c05c.

Test build #79477 has finished for PR 18487 at commit d92c05c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79477/
Test PASSed.

cc @cloud-fan

Will this be convered by https://github.com/apache/spark/pull/18388 ? And another concern is how shall we expect users to tune this config? Can users just tune spark.reducer.maxReqsInFlight instead?

@jiangxb1987 Thanks for the review.
@cloud-fan #18388

Test build #79614 has started for PR 18487 at commit e6e5f6b.

Test build #79614 has finished for PR 18487 at commit e6e5f6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79614/
Test PASSed.

maxReqsInFlight and maxBytesInFlight is hard to control the # of blocks in a single request. When # of map is very high, this change can alleviate the pressure of shuffle server.
@dhruve what is the proper value for maxBlocksInFlightPerAddress? I like this pr if there is no performance issue. It will be great if you can post some benchmark.

@jinxing64 I performed few runs to see if we were observing any performance issues with the change. I ran a simple word count job over a random set of text - 3TB. I couldn't get 100's of executors hammering a single NM, however each NM ended up serving approximately a gig of shuffle data (This might be less in magnitude compared to the job that you are running). I ran it for different maxBlocks # - 10, 20 and 50 keeping the executors to be roughly same and didn't notice any performance difference in terms of the running time of the job across different runs.
For a proper value of maxBlocksInFlightPerAddress, my understanding is that we want to set it to a value as high as possible which doesn't max out the NM and this depends a lot on your workloads and how your data ends up being distributed. Setting it to a lower value mitigates the issue.
Since the changes are on the reducer side, I would recommend to run your job against the changes and set it to a low value and have couple of runs to see if you are seeing any performance hit with large no. of connections hitting a node, as I couldn't reproduce it with the 3TB word count.

@jinxing64 We have the default set the int max so that by default there is no performance penalty for users.  We have done some testing as Dhruve mentioned but we don't regularly hit the issue. My plan was to set it to 20 just like mapreduce does as a starting point.  The exact configuration can be configuration dependent though as it depends on how much memory you give to your NM's, how many requests in parallel you think you will have, etc.  We do plan on doing some more testing but it probably won't be for a couple days

retest this please

Test build #79760 has started for PR 18487 at commit e6e5f6b.

+1, pending jenkins build.  if no further comments I'm going to commit this later today.

Test build #79760 has finished for PR 18487 at commit e6e5f6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79760/
Test PASSed.

cherry pick to 2.2 wasn't clean so can you please put up a separate PR against branch 2.2

hm is this a bug fix? if not we shouldn't cherry pick it.

@rxin it's kind of a stability fix(make shuffle service more stable), so I'm ok to backport if the conflict is small.

@cloud-fan replied to your comments.

@tgravescs Thanks for merging this. I have created a PR for 2.2 https://github.com/apache/spark/pull/18691
I had to remove a couple of newer config entries which landed while resolving a merge conflict.


Are you sure can ant execute ListClientExamle ?

java.lang.ClassNotFoundException: com.allendowney.javacs.ListClientExample

build file is wrong

1- replace each "com.allendowney.javacs" with "com.allendowney.thinkdast" in build.xml file
2- build the project using "ant" command with no arguments

you can use this fork https://github.com/Vaiterius/ThinkDataStructures/tree/edit-build-file


logbackextension: Throwable from error log is missing in LogEntry

When using the logbackextension the throwables von log.error(message, throwable) are missing in the logrecorder-LogEntries
In our Company we have tests where we want to assure, that those errors are logged to the appender, which we actually cant do with Version 2.3.0 of the logrecorder
It would be great, if you could extend the LogRecord with this information.
The logback appender for example creates the logmessage bei getting formattedmessage and throwable:
public String doLayout(ILoggingEvent event) {
        if (!isStarted()) {
            return CoreConstants.EMPTY_STRING;
        }
        StringBuilder sb = new StringBuilder();
        long timestamp = event.getTimeStamp();
        sb.append(cachingDateFormatter.format(timestamp));
        sb.append(" [");
        sb.append(event.getThreadName());
        sb.append("] ");
        sb.append(event.getLevel().toString());
        sb.append(" ");
        sb.append(event.getLoggerName());
        sb.append(" - ");
        sb.append(event.getFormattedMessage());
        sb.append(CoreConstants.LINE_SEPARATOR);
        IThrowableProxy tp = event.getThrowableProxy();
        if (tp != null) {
            String stackTrace = tpc.convert(event);
            sb.append(stackTrace);
        }
        return sb.toString();
    }

It  would be great, if you could extend the LogRecord with this message.

Hi @Daniel-80 thank you for your suggestion.
We are looking into providing the original throwable as part of the LogEntry.
This has been on our Radar for some time, but we never got around to it.
Providing the original Throwable should be a more flexible way for actually asserting desired outcomes than having the formatted message, that might change over time.
As soon as this is available I'll update this issue.

@Drakojin added support for Throwables as well as some additional features and changes in #31
I just triggered the release of v2.4.0 (https://github.com/logrecorder/logrecorder/releases/tag/v2.4.0) - please take note of the breaking change described in that release.

Hi @slu-it ,
thank you for your fast reaction and implementation!
In our compony we get the artefacts from maven- central (https://search.maven.org/artifact/io.github.logrecorder/logrecorder). I do not find the new Version 2.4.0 there. Is that an oversight or intentional?

Hi @slu-it ,
the artifact is now available in maven-central in Version 2.4.0. The test of the new version was successful (i.e. we can now also check the throwable)
Thanks again for the quick implementation!


Fix commitment tests

It turned out that the Pedersen commitment tests were disabled. I re-enabled them in https://github.com/mimblewimble/secp256k1-zkp/pull/34 but some of the tests are broken.

That's weird, or perhaps it's never enabled:)
I would propose to give a fix on another PR, and don't mix it with the important #34 . I will take a look now.

Yes, for now I commented the broken tests out in #34 but we should provide a separate PR that fixes and re-enables them

The root cause is:
The implementation of secp256k1_pedersen_commit never use its 1st parameter const secp256k1_context* ctx, but the tests would like to see the different behavior for different ctx:
    CHECK(secp256k1_pedersen_commit(none, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) == 0);
    CHECK(ecount == 1);
    CHECK(secp256k1_pedersen_commit(vrfy, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) == 0);
    CHECK(ecount == 2);
    CHECK(secp256k1_pedersen_commit(sign, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) != 0);
    CHECK(ecount == 2);

Actually, all above 3 functions call will return 1 (i.e. success!), no matter what ctx is passed.
And another point, I'm wondering why we don't use this secp256k1_context* ctx to achieve an optimized ecmult (i.e. secp256k1_ecmult_gen), since we always use the GENERATOR_G for our pedersen commitment. The current implementation use a generic ecmult to be compatible with any generator.
secp256k1_ecmult_const(&bj, blind_gen, sec, 256);

Since this is the upstream bug (on test_commitment_api), I gave the comments on ElementsProject/secp256k1-zkp#23, and hope it's fixed on there, and then we just merge it.
So, no need a PR here in this repo. But we can keep this issue open until we saw it is fixed on upstream.
And one important concern here:  the current Travis-CI test script doesn't have the test step (i.e. run ./test), that's a major risk on our secp256k1-zkp lib. We'd better take a test to enable it and ensure each test has been passed for any future PR. It deserves a new issue and a dedicated PR for that.

BTW, if you really want a fix here, then you can modify it as following:
in commitment/tests_impl.h:
static void test_commitment_api(void) {
...
    int32_t ecount = 2;
    ...
    secp256k1_rand256(blind);
    CHECK(secp256k1_pedersen_commit(none, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) != 0);
    CHECK(ecount == 2);
    CHECK(secp256k1_pedersen_commit(vrfy, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) != 0);
    CHECK(ecount == 2);
    CHECK(secp256k1_pedersen_commit(sign, &commit, blind, val, &secp256k1_generator_const_h, &secp256k1_generator_const_g) != 0);
    CHECK(ecount == 2);

Then the tests should be OK.

Thanks for investigating. Also the test_multiple_generators() test is failing, namely at secp256k1_pedersen_verify_tally

Fixed in https://github.com/mimblewimble/secp256k1-zkp/pull/44


resetOnDisconnect: does not work

Describe the bug
The line resetOnDisconnect: true in evcc.yaml is marked as error. If removed everything works
Steps to reproduce

Add resetOnDisconnect: true as loadpoints parameter in evcc.yaml
evcc does not start anymore and line with resetOnDisconnect: true is marked as error
Worked on previous versions of evcc

Configuration details
loadpoints:
- title: Garage RS82
  charger: wallbox5
  mode: pv
  phases: 0
  mincurrent: 6
  maxcurrent: 32
#  resetOnDisconnect: true

site:
  title: RS82
  meters:
    grid: grid1
    pv:
    - pv2
    battery:
    - battery3

Log details
evcc does not start

What type of operating system are you running?
Linux
Version
evcc version 0.123.2 (170e2cad)

Please read the release notes: https://github.com/evcc-io/evcc/releases/tag/0.123.0