AzureFileCopy@4 to VM with WinRM prerequisites is creating invalid Expired Certificate (1/1/2022)

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: AzureFileCopy@4
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines

If using Azure Pipelines, provide the account name: sword-grc,
team project name: ARM
build definition name: BranchTestPrototype
build number: 60300

Build Link

Agent - Hosted:

If using Hosted agent, provide agent queue name: Azure Pipelines

Issue Description
Self-Certified Certificate created on VM by FileCopy prerequisites has expiry date of 1/1/2022, hence is invalid and so WinRM copy fails with message:
"The remote session query failed for <IP_ADDRESS> with the following error message: The server certificate on the destination computer (<IP_ADDRESS>:5986) has the following errors:
The SSL certificate is expired."

Explanation:
ConfigureWinRM.ps1 has code which erroneously hardwires the expiry date of the certificate as follows:
$serial = Get-Random
    .\makecert -r -pe -n CN=$hostname -b 01/01/2012 -e **01/01/2022** -eku <IP_ADDRESS>.<IP_ADDRESS>.1 -ss my -sr localmachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -# $serial 2>&1 | Out-Null

Task logs
[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]
2022-01-04T17:51:34.2640866Z ##[section]Starting: Copy Files
2022-01-04T17:51:34.2762104Z ==============================================================================
2022-01-04T17:51:34.2762469Z Task         : Azure file copy
2022-01-04T17:51:34.2762789Z Description  : Copy files to Azure Blob Storage or virtual machines
2022-01-04T17:51:34.2763069Z Version      : 4.195.0
2022-01-04T17:51:34.2763324Z Author       : Microsoft Corporation
2022-01-04T17:51:34.2763679Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
2022-01-04T17:51:34.2764090Z ==============================================================================
2022-01-04T17:51:35.6812335Z ##[command]Import-Module -Name C:\Modules\az_6.5.0\Az.Accounts\2.7.0\Az.Accounts.psd1 -Global
2022-01-04T17:51:36.9280390Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2022-01-04T17:51:37.9678495Z ##[command]Clear-AzContext -Scope Process
2022-01-04T17:51:38.6841511Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
2022-01-04T17:51:40.3167982Z ##[command] Set-AzContext -SubscriptionId 61e25dca-a55a-45ee-9a73-80b5b16e99a2 -TenantId ***
2022-01-04T17:51:41.7602777Z ##[command]Import-Module -Name C:\Modules\az_6.5.0\Az.Resources\4.4.0\Az.Resources.psd1 -Global
2022-01-04T17:51:42.8948128Z ##[command]Import-Module -Name C:\Modules\az_6.5.0\Az.Storage\3.12.0\Az.Storage.psd1 -Global
2022-01-04T17:51:43.3003941Z ##[command]Import-Module -Name C:\Modules\az_6.5.0\Az.Compute\4.17.1\Az.Compute.psd1 -Global
2022-01-04T17:51:44.8576897Z ##[command]Import-Module -Name C:\Modules\az_6.5.0\Az.Network\4.12.0\Az.Network.psd1 -Global
2022-01-04T17:51:52.3485872Z ##[command] & "AzCopy\AzCopy.exe" login --service-principal --application-id "" --tenant-id="" --aad-endpoint "https://login.windows.net/"
2022-01-04T17:51:52.3877133Z INFO: If you set an environment variable by using the command line, that variable will be readable in your command line history. Consider clearing variables that contain credentials from your command line history.  To keep variables from appearing in your history, you can use a script to prompt the user for their credentials, and to set the environment variable.
2022-01-04T17:51:52.5378968Z INFO: AzCopy.exe: A newer version 10.13.0 is available to download
2022-01-04T17:51:52.5379412Z
2022-01-04T17:51:52.6670910Z INFO: SPN Auth via secret succeeded.
2022-01-04T17:51:52.7774150Z INFO: AzCopy.exe: A newer version 10.13.0 is available to download
2022-01-04T17:51:52.7776936Z
2022-01-04T17:51:52.7859690Z Uploading files from source path: 'D:\a\1\s' to storage account: 'armdeployfilestorage' in container: '30fbf4d0-2a75-4e2f-a8a7-a77aa4a9d9c7' with blob prefix: ''
2022-01-04T17:51:52.7957640Z ##[command] & "AzCopy\AzCopy.exe" copy "D:\a\1\s" "https://armdeployfilestorage.blob.core.windows.net/30fbf4d0-2a75-4e2f-a8a7-a77aa4a9d9c7"  --log-level=INFO --recursive
2022-01-04T17:51:52.8357960Z INFO: Scanning...
2022-01-04T17:51:52.8359470Z INFO: Authenticating to destination using Azure AD
2022-01-04T17:51:53.0512443Z INFO: AzCopy.exe: A newer version 10.13.0 is available to download
2022-01-04T17:51:53.0513192Z
2022-01-04T17:51:53.0880420Z INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2022-01-04T17:51:53.0948435Z
2022-01-04T17:51:53.0974649Z Job 392de216-ebd4-8b4b-67a2-c3ce235da969 has started
2022-01-04T17:51:53.0976731Z Log file is located at: C:\Users\VssAdministrator.azcopy\392de216-ebd4-8b4b-67a2-c3ce235da969.log
2022-01-04T17:51:53.0978594Z
2022-01-04T17:51:53.1011154Z
2022-01-04T17:51:55.0931441Z 0.0 %, 0 Done, 0 Failed, 83 Pending, 0 Skipped, 83 Total,
2022-01-04T17:51:57.0933617Z 7.5 %, 81 Done, 0 Failed, 2 Pending, 0 Skipped, 83 Total, 2-sec Throughput (Mb/s): 793.5969
2022-01-04T17:51:59.0936567Z 51.1 %, 82 Done, 0 Failed, 1 Pending, 0 Skipped, 83 Total, 2-sec Throughput (Mb/s): 1444.9781
2022-01-04T17:52:01.0946692Z 97.8 %, 82 Done, 0 Failed, 1 Pending, 0 Skipped, 83 Total, 2-sec Throughput (Mb/s): 840.9941
2022-01-04T17:52:01.0950890Z
2022-01-04T17:52:01.0951510Z
2022-01-04T17:52:01.0952232Z Job 392de216-ebd4-8b4b-67a2-c3ce235da969 summary
2022-01-04T17:52:01.0952754Z Elapsed Time (Minutes): 0.1334
2022-01-04T17:52:01.0953213Z Number of File Transfers: 83
2022-01-04T17:52:01.0953703Z Number of Folder Property Transfers: 0
2022-01-04T17:52:01.0954185Z Total Number of Transfers: 83
2022-01-04T17:52:01.0954661Z Number of Transfers Completed: 83
2022-01-04T17:52:01.0955134Z Number of Transfers Failed: 0
2022-01-04T17:52:01.0955574Z Number of Transfers Skipped: 0
2022-01-04T17:52:01.0956543Z TotalBytesTransferred: 770165203
2022-01-04T17:52:01.0958237Z Final Job Status: Completed
2022-01-04T17:52:01.0960772Z
2022-01-04T17:52:01.2398620Z Uploaded files successfully from source path: 'D:\a\1\s' to storage account: 'armdeployfilestorage' in container: '30fbf4d0-2a75-4e2f-a8a7-a77aa4a9d9c7' with blob prefix: ''
2022-01-04T17:52:01.2411793Z ##[command] & "AzCopy\AzCopy.exe" logout
2022-01-04T17:52:01.2708972Z INFO: Logout succeeded.
2022-01-04T17:52:01.4340413Z INFO: AzCopy.exe: A newer version 10.13.0 is available to download
2022-01-04T17:52:01.4343249Z
2022-01-04T17:52:01.4344082Z INFO: AzCopy.exe: A newer version 10.13.0 is available to download
2022-01-04T17:52:01.4345093Z
2022-01-04T17:52:08.1438358Z [Azure Call]Getting the custom script extension 'WinRMCustomScriptExtension' for vm 'swordvmcertific'
2022-01-04T17:52:09.1974444Z [Azure Call]Setting the custom script extension 'WinRMCustomScriptExtension' for vm 'swordvmcertific'
2022-01-04T17:53:11.6823169Z [Azure Call]Set the custom script extension 'WinRMCustomScriptExtension' for vm 'swordvmcertific'
2022-01-04T17:53:11.9397238Z [Azure Call]Getting the status for vm 'swordvmcertific'
2022-01-04T17:53:12.8578801Z [Azure Call]Got the status for vm 'swordvmcertific'
2022-01-04T17:54:52.1830507Z ##[warning]A parameter cannot be found that matches parameter name 'and'.
2022-01-04T17:54:52.2781308Z ##[error]The remote session query failed for <IP_ADDRESS> with the following error message: The server certificate on the destination computer (<IP_ADDRESS>:5986) has the following errors:
The SSL certificate is expired.
2022-01-04T17:54:52.3372826Z ##[section]Finishing: Copy Files
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
2022-01-04T17:54:52.2781308Z ##[error]The remote session query failed for <IP_ADDRESS> with the following error message: The server certificate on the destination computer (<IP_ADDRESS>:5986) has the following errors:
The SSL certificate is expired.

This is our task script for info
- task: AzureFileCopy@4   displayName: Copy Files   inputs:     SourcePath: $(System.DefaultWorkingDirectory)     azureSubscription: Sword GRC Paid Subscription (61e25dca-a55a-45ee-9a73-80b5b16e99a2)     Destination: AzureVMs     storage: armdeployfilestorage     resourceGroup: ${{ parameters.ResourceGroup }}     MachineNames: ${{ parameters.VmName }}     vmsAdminUserName: ${{ parameters.VmUserName }}     vmsAdminPassword: ${{ parameters.VmPassword }}     TargetPath: d:\install     enableCopyPrerequisites: true

We are also encountering the same issue with AzureFileCopy version 3.
This has a high impact as we are currently unable to use our DevOps pipelines to deploy.
Is anyone able to suggest any temporary work arounds?

We are also encountering the same issue with AzureFileCopy version 3. This has a high impact as we are currently unable to use our DevOps pipelines to deploy.
Is anyone able to suggest any temporary work arounds?

I found that pre-creating the self-cert certificate enables the AzureFileCopy to continue without making the new invalid certificate. You have to be careful to get the certificate spec right - this worked for me, must be run on a clean VM which has not already attempted to install WinRM scripting:
$script = "New-SelfSignedCertificate -DnsName $(IpAddress), $(IpAddress) -TextExtension @('<IP_ADDRESS>={text}<IP_ADDRESS>.<IP_ADDRESS>.1') -KeyAlgorithm RSA -KeyLength 1024 -KeySpec Signature" ... az vm run-command invoke  --command-id RunPowerShellScript ....vm auth...  --scripts $script

Thanks for the suggestion Marcus.
Setting up clean VMs for our environments may be a problem as we have other software on the machines that is not scripted, so the process could be time consuming.
Hopefully we can find a solution for a VM with an existing certificate.
So far we have tried deleting the certificate and generating a new one using the makecrt command seen in the PowerShell script. After doing that we are getting the deployment error:
Encountered an internal error in the SSL library.

I had the same error, internal error, which seems to be the WinRM script rerunning. In our case we can wipe it all and start again. There is a way to uninstall the WinRM script completely but I didn’t have to try that. Hope Microsoft pick this up quickly, hope you find a work around.
Get Outlook for iOShttps://aka.ms/o0ukef

From: aaroncoderepo @.>
Sent: Wednesday, January 5, 2022 9:42:32 PM
To: microsoft/azure-pipelines-tasks @.>
Cc: Marcus Richards @.>; Author @.>
Subject: Re: [microsoft/azure-pipelines-tasks] @.*** to VM with WinRM prerequisites is creating invalid Expired Certificate (1/1/2022) (Issue #15656)
Thanks for the suggestion Marcus.
Setting up clean VMs for our environments may be a problem as we have other software on the machines that is not scripted, so the process could be time consuming.
Hopefully we can find a solution for a VM with an existing certificate.
So far we have tried deleting the certificate and generating a new one using the makecrt command seen in the PowerShell script. After doing that we are getting the deployment error:
Encountered an internal error in the SSL library.
—
Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F15656%23issuecomment-1006101265&data=04|01||538854066b4d4f05e50008d9d094480a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637770157539703047|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=8bsuckvOk46nlNUX4COW2qqX8sNKczzX81rcz1P7GTs%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIHET2AAJK6CLEDHBTQDOFTUUS3MRANCNFSM5LIWPPNA&data=04|01||538854066b4d4f05e50008d9d094480a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637770157539703047|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=wMM1AjTJVq5P3S9C8NAN9xUjl0Kk9nuJnlpQVMBEt9k%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04|01||538854066b4d4f05e50008d9d094480a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637770157539703047|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=7yZBk%2FQiq3HUyy09r%2BRtILyjumt8390MylgKvXiEpIQ%3D&reserved=0 or Androidhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04|01||538854066b4d4f05e50008d9d094480a|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637770157539703047|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=pYP9CrxiWscGbDRHQjhQ5zIHSw6iAnX5B4eIuRXUYAA%3D&reserved=0.
You are receiving this because you authored the thread.Message ID: @.***>

Also having the same issue with V3 and the AzureFileCopy step and unable to use any of the suggested workarounds. This is a critical disruption for our pipelines, can this be escalated?

We managed to generate a new certificate running the cmd below.
Then configured WinRM to use the new certificate thumbprint by deleting the existing WinRM https configuration, and creating a new one.
Code was sourced from the following repo, and dates modified:
https://stackoverflow.com/questions/34897260/how-to-configure-winrm-for-azure-virtual-machine-in-resource-manager-portal
https://github.com/microsoft/azure-pipelines-extensions/blob/master/TaskModules/powershell/WinRM/WinRM-Http-Https-With-Makecert/ConfigureWinRM.ps1
Cmd Prompt
$hostname = "<FQDN>"

.\makecert -r -pe -n CN=$hostname -b 01/01/2022 -e 01/01/2032 -eku <IP_ADDRESS>.<IP_ADDRESS>.1 -ss my -sr localmachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12

PowerShell
PS C:\Users\<user>> $hostname = "<FQDN>"

PS C:\Users\<user>> $thumbprint = (Get-ChildItem cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint

PS C:\Users\<user>> function Delete-WinRMListener
{
    $config = winrm enumerate winrm/config/listener
    foreach($conf in $config)
    {
        if($conf.Contains("HTTPS"))
        {
            Write-Verbose -Verbose "HTTPS is already configured. Deleting the exisiting configuration."

            winrm delete winrm/config/Listener?Address=*+Transport=HTTPS
            break
        }
    }
}

PS C:\Users\<user>> Delete-WinRMListener
VERBOSE: HTTPS is already configured. Deleting the exisiting configuration.

PS C:\Users\<user>> $WinrmCreate= "winrm create --% winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname=`"$hostName`";CertificateThumbprint=`"$thumbPrint`"}"
    invoke-expression $WinrmCreate
    winrm set winrm/config/service/auth '@{Basic="true"}'

ResourceCreated
    Address = http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous
    ReferenceParameters
        ResourceURI = http://schemas.microsoft.com/wbem/wsman/1/config/listener
        SelectorSet
            Selector: Address = *, Transport = HTTPS

Auth
    Basic = true
    Kerberos = true
    Negotiate = true
    Certificate = false
    CredSSP = false
    CbtHardeningLevel = Relaxed

We have run this on our test environment and it seems to work.
Not sure if this is a good idea however, and there would be more elegant ways of accomplishing the same thing.
Possibly just modifying and running this PowerShell script:
https://github.com/microsoft/azure-pipelines-extensions/blob/master/TaskModules/powershell/WinRM/WinRM-Http-Https-With-Makecert/ConfigureWinRM.ps1

anyone else try updating the script for another 10 years? We need this to work.

The fix will need to consider VMs with expiring/expired certificates:

ensure that the certificate is deleted, and a new certificate is created (when the current certificate is about to expire or has expired)
remove the old WinRM https config, recreate with the new certificate thumbprint

Seriously, how long does it take to review such a simple pull request.

We are able to Repro issue  in our local machine .Now we are updating code changes.

Got the same issue, waiting for the fix

Hi All,  Could you please run the below work around for Temporary fix. Soon we will  raise the PR.
The below script has to be run in  Target Machine.
Download makecert.exe from " https://aka.ms/vstsmakecertexe" and Need to  save the file in PowerShell location.
Please Make Sure run three scripts in one ISE window:

Create New Cert:

$hostname = hostname
$thumbprint = (Get-ChildItem cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint
if(-not $thumbprint)
{
if(-not (Test-Path -Path .\makecert.exe))
{
throw "File not found: makecert.exe"
}
Write-Host "Hello world1"
.\makecert -r -pe -n CN=$hostname -b (Get-Date).ToString('MM/dd/yyyy') -e (Get-Date).Addyears(10).ToString('MM/dd/yyyy') -eku <IP_ADDRESS>.<IP_ADDRESS>.1 -ss my -sr localmachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12
$thumbprint=(Get-ChildItem cert:\Localmachine\my | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint
if(-not $thumbprint)
{
throw "Failed to create the test certificate."
}
}

Delete WinRM Listner:

$hostname = hostname
$thumbprint = (Get-ChildItem cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint
function Delete-WinRMListener
{
$config = winrm enumerate winrm/config/listener
foreach($conf in $config)
{
if($conf.Contains("HTTPS"))
{
Write-Verbose -Verbose "HTTPS is already configured. Deleting the exisiting configuration."
winrm delete winrm/config/Listener?Address=*+Transport=HTTPS
break
}
}
}
Delete-WinRMListener

WinRM set with new cert

$WinrmCreate= "winrm create --% winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="$hostName";CertificateThumbprint="$thumbPrint"}"
invoke-expression $WinrmCreate
winrm set winrm/config/service/auth '@{Basic="true"}'
Thanks.

I was able to test this and confirmed it worked @v-ibshaik . Here is the updated powershell script and instructions, make sure to use this on the target VM. This is a workaround, I've also created a Severity A support ticket with Microsoft and their Product Team is pushing a patch into place. Until then, use this:
1: download makecert put in same directory as powershell script: https://aka.ms/vstsmakecertexe
2: copy paste script into ADMIN powershell ISE
3: Change the below variable
4: Open up certificates.msc and delete your personal store, expired certificate on the TARGET VM
5: Run script
$hostname = ''
$thumbprint = (Get-ChildItem cert:\LocalMachine\My | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint
if(-not $thumbprint)
{
if(-not (Test-Path -Path .\makecert.exe))
{
throw "File not found: makecert.exe"
}
.\makecert -r -pe -n CN=$hostname -b (Get-Date).ToString('MM/dd/yyyy') -e (Get-Date).Addyears(10).ToString('MM/dd/yyyy') -eku <IP_ADDRESS>.<IP_ADDRESS>.1 -ss my -sr localmachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12
$thumbprint=(Get-ChildItem cert:\Localmachine\my | Where-Object { $_.Subject -eq "CN=" + $hostname } | Select-Object -Last 1).Thumbprint
if(-not $thumbprint)
{
    throw "Failed to create the test certificate."
}

}
Delete all existing win rm listener
function Delete-WinRMListener
{
$config = winrm enumerate winrm/config/listener
foreach($conf in $config)
{
if($conf.Contains("HTTPS"))
{
Write-Verbose -Verbose "HTTPS is already configured. Deleting the exisiting configuration."
winrm delete winrm/config/Listener?Address=*+Transport=HTTPS
break
}
}
}
Delete-WinRMListener
Recreate WinRM listener with new certificate created above
$WinrmCreate= "winrm create --% winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="$hostName";CertificateThumbprint="$thumbPrint"}"
invoke-expression $WinrmCreate
winrm set winrm/config/service/auth '@{Basic="true"}'

Hi All,
We have fixed the issue.
Thanks.

great news thanks. Has it been deployed to the production environment?
Get Outlook for iOShttps://aka.ms/o0ukef

From: v-ibshaik @.>
Sent: Tuesday, January 18, 2022 8:56:36 AM
To: microsoft/azure-pipelines-tasks @.>
Cc: Marcus Richards @.>; Author @.>
Subject: Re: [microsoft/azure-pipelines-tasks] @.*** to VM with WinRM prerequisites is creating invalid Expired Certificate (1/1/2022) (Issue #15656)
Hi All,
We have fixed the issue.
Thanks.
—
Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F15656%23issuecomment-1015197035&data=04|01||5ba52a16d07e467fb1d908d9da606f69|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637780929989326113|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=NU4%2F95FxgA4Ty6455KZBkJNYTsC2ex4T7GKHDN8Hdu4%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIHET2ENGN7MRSABY3TIKN3UWUTMJANCNFSM5LIWPPNA&data=04|01||5ba52a16d07e467fb1d908d9da606f69|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637780929989326113|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=4e9BUuVjo95OYk6fxJhxzs2wGxEkriw9uvQgBVEzwIc%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04|01||5ba52a16d07e467fb1d908d9da606f69|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637780929989326113|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=%2F4GJs9ZznXQxxt5NBotI%2Fr8nZIL2%2BgcTJyfobj%2BxkuY%3D&reserved=0 or Androidhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04|01||5ba52a16d07e467fb1d908d9da606f69|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637780929989326113|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=7dpOKNbNE4f1UnC20NMMA3dYzi0DF8%2Byza9lLBNFb68%3D&reserved=0.
You are receiving this because you authored the thread.Message ID: @.***>

Hello  marcusrichards,
Yes, We have fixed the issue by updating the Cert Valid dates.
Thanks.

Good news, thank you.
Get Outlook for iOShttps://aka.ms/o0ukef

From: v-ibshaik @.>
Sent: Tuesday, January 18, 2022 1:37:50 PM
To: microsoft/azure-pipelines-tasks @.>
Cc: Marcus Richards @.>; Author @.>
Subject: Re: [microsoft/azure-pipelines-tasks] @.*** to VM with WinRM prerequisites is creating invalid Expired Certificate (1/1/2022) (Issue #15656)
Hello marcusrichards,
Yes, We have fixed the issue by updating the Cert Valid dates.
Thanks.
—
Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-tasks%2Fissues%2F15656%23issuecomment-1015420034&data=04|01||855d7113937040c8606008d9da87b8fe|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637781098729391408|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=acZZYIYNmo3F5lCG5saKxWFSt%2B8YYGo5CsYa8di3MCk%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIHET2EIFMJVIPFCBRC3D23UWVUK5ANCNFSM5LIWPPNA&data=04|01||855d7113937040c8606008d9da87b8fe|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637781098729391408|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=IkeS%2Ffnd%2F9%2F4aIhqnIfybNuTLH3wd%2BFPjvCZspz0RtA%3D&reserved=0.
Triage notifications on the go with GitHub Mobile for iOShttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04|01||855d7113937040c8606008d9da87b8fe|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637781098729391408|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=ce7UFbDUFqNkcTvqmoy4bYjdxyI%2F0WI6%2Boa0ZS45VVw%3D&reserved=0 or Androidhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04|01||855d7113937040c8606008d9da87b8fe|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637781098729391408|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=wJArEmrnXjLNO7xR70cHJ2GxOVSLWkTPqimW49xg260%3D&reserved=0.
You are receiving this because you authored the thread.Message ID: @.***>

Hello v-ibshaik,
I am confused here.  Does this mean if we are using the AzureFileCopy task in an AzureDevops pipeline, should error be fixed or do we have to do anything on the target servers to update the certs still?

I'm not seeing any fix on my release pipelines.

Me neither..

Hi mrsquish,
We don't have any code changes in azure-pipeline-tasks and basically we have updated the code changes  in ==> azure-quickstart-templates/master/demos/vm-winrm-windows/ConfigureWinRM.ps1
Could you please verify the task .
Thanks.

Thanks v-ibshaik.
So, I assumed I was supposed to run the ConfigureWinRM.ps1 script on the VM, which I did. Using the DNS name as the hostname? After that I ran the release pipeline, this time getting the dreaded "Operation 'PUT Extension' is not allowed on VM extension 'WinRMCustomScriptExtension' since it is marked for deletion.". Tried, restarting the VM (no change). I suspected the restart didn't work, so tried restarting from windows. Now the VM is completely unresponsive, can't start it, can't stop it, can't connect to it. Portal thinks it's running, but my guess is it isn't.

Ok, so I was able to resurrect the VM with a restart --force.
After doing so, the WinRMCustomScript extension had been removed, so I thought, good to go.
Ran the Release pipeline again, and in short, it wouldn't finish (end up stopping it after the AzureCopy task had been running for an hour).
The last two lines from the log was :
[debug]Set-AzVMCustomScriptExtension -ResourceGroupName  -VMName  -Name WinRMCustomScriptExtension -FileUri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/demos/vm-winrm-windows/ConfigureWinRM.ps1 https://raw.githubusercontent.com/Azure/azure-quickstart-templates/501dc7d24537e820df7c80bce51aba9674233b2b/201-vm-winrm-windows/makecert.exe  -Run ConfigureWinRM.ps1 -Argument .australiaeast.cloudapp.azure.com -Location australiaeast -ErrorAction Stop -Verbose
[debug]Performing the operation "Set" on target "CustomScriptExtension".
Could it be possible the argument isn't being passed properly and it's waiting for user input?

Hello mrsquish,

Issue got fixed ,No need to do any update  in Target machine. could you please Try to connect new Target machine from your Task.

In your case you have work around with  your target machine. for this you need to  delete the cert ,run the Delete-WinRMListener and clear the cache.

Thanks.

In case anyone stumbles on this thread, I had a Windows 2012 R2 VM in Azure created maybe in 2016, with an Azure file copy task in Azure DevOps that started failing due to certificates that had expired in 2021. Updating the task version from 1.* to 4.* and running the release again had no effect on the certificate. I ended up following the script from @meyemh30to delete and recreate the certificate. I also had to manually download AzCopy and add the path to the system environment variables. I also had to adjust my source path to add "**" to the end to make sure the source folder name wasn't created and used as the destination. My specific challenges may have been unique to my combination, but I'm back up and running again.

Hi, we are still experiencing this issue.
The above comments say we don't need to run any scripts or fixes, is this the case?
If not can you advise the correct steps?
Thanks

@richRubie  This issue is not closed, it can only be fixed by running the script I posted.


fix: typo

Description

Testing required outside of automated testing?

[ ] Not Applicable

Screenshots (if appropriate):

[ ] Not Applicable

Rollback / Rollforward Procedure

[ ] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 1.151.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update NuGet packages

How often will NuGet packages be updated? There are a lot of recent changes that would be nice to be able to grab from there. For now I am just building the project and using those dlls, but pushing the newer stuff with NuGet would be nice.

Yes that is planned fairly soon. I need to run some more tests and then I will release the new version. I'm on vacation currently so it won't be this and probably also not next week.


fix: #6 file tree issues

Fixes file tree rendering and adds:

debug logger
replit config
prettier config in package.json
hidden files and config through localStorage
debounce function
globalThis types (wo importing @types/node)
VITE_HOST config env var (for access in Cloud IDE's)

Please merge before I add a yaml config file for VSLite with a UI.
Thanks


Draft HYCOM.org recipe

HYCOM from HYCOM.org
I've drafted up recipes for the most recent experiment on HYCOM.org. When I try to run it with http access to NetCDF files, I get a 403 Forbidden, despite being able to wget the same URL.
I also tried to build a recipe using OpenDAP to access. That appeared to crash the Jupyter kernel when I tested it.
Here's a notebook of my experiments: https://gist.github.com/abkfenris/ca45610ce7ef5ec703ba3c751de682c7
I suspect errors might be due to underlying issues with too many connections to the server, or for too large a file, which are probably upstream issues.
<!-- If you have created an issue for this dataset, please reference it here.  If not, you can include the relevant information from the issue template  https://github.com/pangeo-forge/staged-recipes/issues/new/choose in this PR --!>

Sane spelling apparently helps...

/run recipe-test recipe_run_id=1

/run recipe-test recipe_run_id=2

I'm largely expecting things to explode, most likely in new and unique ways.

I'm largely expecting things to explode, most likely in new and unique ways.

Yes that's what happened! But it's on me. Pasting a snippet of backend logs here for our reference:
Traceback (most recent call last):
...
    print(f"\n***BAKERY_IMAGES[pangeo_forge_version] = ***\n")
KeyError: '0.8.2'

Looks like pangeo-forge-recipes==0.8.2 is so new that the backend actually isn't configured to support it yet. The shortcut here would be to specify an older version; but that could lead to all sorts of problems of its own.
The right way to handle this is for me to add 0.8.2 support on the backend and then re-run this test. I won't be able to get to that tonight, but will prioritize it by the end of the week (ideally tomorrow, maybe Friday). I'll check back on this thread as soon as that's ready. Thanks for being such a cutting edge early adopter, Alex! And for your patience as we iron out these details.

Well, I am also expecting my code, or the Hycom server to freak out.
And I tried to adopt things and try silly stuff way too early. https://github.com/gulfofmaine/OISST_intake There I tried to setup a mini reference bakery with Github Actions for OISST. Things are much more polished now, especially with the bot!

The backend should now support pangeo-forge-recipes==0.8.2 (as evidenced by https://github.com/pangeo-forge/staged-recipes/pull/107#issuecomment-1058481448, where a test recipe was successfully executed using this version).
I am going to re-run the recipe tests now and we'll see what happens.

/run recipe-test recipe_run_id=1

/run recipe-test recipe_run_id=2

I'm getting completely blank pages from both of the bot links, is that expected?

@abkfenris note that we are still waiting for the frontend linked in the above two comments to come online, so sadly logs are not available in real time right now. I will monitor them on the backend and paste any key details (i.e. tracebacks) to this thread.

Ah, ok.
I went down a rabbit hole exploring the Github Checks API last night and thinking about if the backend could ship logs to there, not that you need any ideas....

if the backend could ship logs to there

Cool idea, anything's possible! Let's see how they look on the frontend, since it's already configured. I'm told that'll be up today.
So for hycom-glby008-expt-93 we're getting ClientResponseError: 403, message='Forbidden' on http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ts3z.nc.

 click to expand full traceback
Task 'cache_input[1]': Exception encountered during task execution!
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 861, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/prefect/utilities/executors.py", line 323, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/registrar/flow.py", line 69, in wrapper
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 151, in cache_input
    config.storage_config.cache.cache_file(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 166, in cache_file
    _copy_btw_filesystems(input_opener, target_opener)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py", line 43, in _copy_btw_filesystems
    data = source.read(BLOCK_SIZE)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 566, in read
    return super().read(length)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/spec.py", line 1565, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/caching.py", line 377, in _fetch
    self.cache = self.fetcher(start, bend)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/fsspec/implementations/http.py", line 605, in async_fetch_range
    r.raise_for_status()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ts3z.nc')

But it looks like my browser allows me to download that file. I wonder if it's a User Agent  problem or similar; i.e. maybe the server doesn't like that the request is originating from a Dask worker? Hmm.. 🤔
As for hycom-glby008-expt-93-opendap, do I understand correctly that the output of that recipe is intended to be the same as hycom-glby008-expt-93, and the only difference is that it's sourcing the inputs from a different endpoint? If so, my suggestion is that we focus on debugging hycom-glby008-expt-93 first, and then fall back to -opendap if we hit an impasse with that. (FWIW, -opendap also failed, but it looks like it was an infra issue--dead worker--not a recipe issue.)

So yes, both recipes are supposed to be for the same data.
I found in the sandbox the OpenDAP recipe could progress a little further before it failed, whereas the regular recipe was getting those 403s also. Similarly xarray also had issues while trying to open directly from a NetCDF, but it was able to use the OpenDAP service.
So for OpenDAP, if subset_inputs={"time": 1, "depth": 1, "lon": 500, "lat": 500}, meant that it was trying to download and cache the entire time range at once (even if it was dividing the lat and lon by 500 each), that probably blew the memory in the sandbox.

Also setup_logging("DEBUG") is very useful because it will actually print out the datasets as it opens them.

I found in the sandbox the OpenDAP recipe could progress a little further before it failed, whereas the regular recipe was getting those 403s also. Similarly xarray also had issues while trying to open directly from a NetCDF, but it was able to use the OpenDAP service.

Awesome. So I revise my earlier suggestion. Let's still focus on just one of these for now, but based on this insight let's make that one hycom-glby008-expt-93-opendap.

I am trying now with subset_inputs={"time": 9473, "depth": 10}. Seems to be working a bit.

Since this looks like it may work, if we get this (or similar) pushed to the PR, I can re-run a test of hycom-glby008-expt-93-opendap on Pangeo Cloud.

Ya, it's big, let alone the other ones to get a more complete time range, but HYCOM's servers have been a pain that I've fought for a while, and both Chelle and Rich were all over the idea of liberating the data for more efficient access.
I'll work on some tweaks. I was thinking that I would open it initially in xarray to find out the length of the time dimension (since this is the dataset they are updating daily), and use that to inform the subset size.

The full length of the time dimension is 9473. And the full dataset size is 60 TB! 😱

I'm happy to do all sorts of guinea pigging to avoid getting handed paper bags full of hard drives in dark alleys. And to reduce the number of other people who get to have that experience.

I will run the new version now, but noting that I think we'll have to subset further.
In general (and, note to self, this should be better documented...) we should shoot for subsets to be <= 500 MB each.
If this math is correct (is it?)
In [1]: (6e13/9473) \  # 60 TB divided by 9473 time steps
    ...: /10 \  # divided by 10 depth steps
    ...: /1e6  # in megabytes
Out[0]:<PHONE_NUMBER>778106

then we're still a little large on the subsets. But it's a good experiment and cheap to run either way so I'll do that now.

/run recipe-test recipe_run_id=3

It's 40 depths total, and I've set it to target one depth per chunk, so we could subset it down by 40 to get it closer to 150 MB each. Or 20 for 2 chunks per subset for 315.

I just timed how long it takes to load one timestep / single vertical slice (~344 MB): 34 s. So we are getting about 10 MB/s throughput from the opendap server.
If we exceute the recipe in serial it will take 70 days to run. Not sure if we can expect any parallel scaling of i/o from the opendap server to bring that down.

20 for 2 chunks per subset for 315

This seems like a good compromise.
... in the meantime I am updating @pangeo-forge-bot. It turns out they did not have netcdf4 installed, so the part where you open the dap_url with xarray to get the time length confused them... I'll have that fixed momentarily.

Ok @pangeo-forge-bot has netcdf4 now so I'm going to re-try the recipe test with the current subsetting and we'll see if the workers die or not...

/run recipe-test recipe_run_id=3

The Dask dashboard is available at http://dask-jovyan-ecd7bc20-5.pangeo-forge-columbia-staging-bakery:8787/status

Would be amazing if we could open this port to the outside.

For some reason, the logs at https://pangeo-forge.org/dashboard/recipe-run/3 are truncated and missing the error encountered in the most recent test. @jhamman and I are coordinating to resolve the logging issue. In the meantime, here's the relevant error pasted from the backend:

Traceback
Task 'store_chunk[71]': Exception encountered during task execution!
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('http://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 861, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/prefect/utilities/executors.py", line 323, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/registrar/flow.py", line 69, in wrapper
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 523, in store_chunk
    with open_chunk(chunk_key, config=config) as ds_chunk:
  File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 339, in open_chunk
    dsets = [stack.enter_context(open_input(input_key, config=config)) for input_key in inputs]
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 339, in <listcomp>
    dsets = [stack.enter_context(open_input(input_key, config=config)) for input_key in inputs]
  File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 303, in open_input
    with xr.open_dataset(f, **kw) as ds:
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py", line 495, in open_dataset
    backend_ds = backend.open_dataset(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 550, in open_dataset
    store = NetCDF4DataStore.open(
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 379, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 327, in __init__
    self.format = self.ds.data_model
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 388, in ds
    return self._acquire()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 382, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/srv/conda/envs/notebook/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "src/netCDF4/_netCDF4.pyx", line 2307, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'http://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0'

So this looks like a problem when a worker is trying to get time_length dynamically from the http endpoint.
@abkfenris, can we try skipping that part in favor of hardcoding time_length = 9473 for now?

Ok, I just hard coded it for now, though that hiccup looks more like an OpenDAP/THREDDS server issue to me.

/run recipe-test recipe_run_id=4

I'm guessing that the opendap server is not happy at us throwing a bunch of simultaneous huge requests at it. That's a place where having limits on task concurrency would really help.

looks more like an OpenDAP/THREDDS server issue to me.

Yes you are correct. And logs at https://pangeo-forge.org/dashboard/recipe-run/4 (better formatting coming soon) confirm this. My mistake for reading too quickly (I thought for a second the dynamic time_length assignment was the only place we were using the dap_url, but now I realize that's not right at all!)
So in terms of debugging, this does not appear to be an issue with the worker image itself, because I can open the dap_url with xarray within a local container based on that image, e.g.
$ docker pull pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2
$ docker run -it pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2 /bin/bash

jovyan@e41dbb22a2f7:~$ ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.30.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import xarray as xr

In [2]: dap_url = 'http://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0'

In [3]: with xr.open_dataset(dap_url, drop_variables=["tau"]) as ds:
   ...:     print(ds)
   ...:
<xarray.Dataset>
Dimensions:            (lat: 4251, lon: 4500, time: 9473, depth: 40)
Coordinates:
  * lat                (lat) float64 -80.0 -79.96 -79.92 ... 89.92 89.96 90.0
  * lon                (lon) float64 0.0 0.07996 0.16 0.24 ... 359.8 359.8 359.9
  * time               (time) datetime64[ns] 2018-12-04T12:00:00 ... 2022-03-...
  * depth              (depth) float64 0.0 2.0 4.0 6.0 ... 3e+03 4e+03 5e+03
Data variables:
    surf_el            (time, lat, lon) float32 ...
    water_temp         (time, depth, lat, lon) float32 ...
    water_temp_bottom  (time, lat, lon) float32 ...
    salinity           (time, depth, lat, lon) float32 ...
    salinity_bottom    (time, lat, lon) float32 ...
    water_u            (time, depth, lat, lon) float32 ...
    water_u_bottom     (time, lat, lon) float32 ...
    water_v            (time, depth, lat, lon) float32 ...
    water_v_bottom     (time, lat, lon) float32 ...
Attributes:
    classification_level:      UNCLASSIFIED
    distribution_statement:    Approved for public release. Distribution unli...
    downgrade_date:            not applicable
    classification_authority:  not applicable
    institution:               Fleet Numerical Meteorology and Oceanography C...
    source:                    HYCOM archive file
    history:                   archv2ncdf2d
    comment:                   p-grid
    field_type:                instantaneous
    Conventions:               CF-1.6 NAVO_netcdf_v1.1

So place where this could be going wrong include:

Serialization/deserialization of the recipe before it gets to the worker
Worker network connection issues with the source file server
...

I'll think a bit more about how to debug this, and certainly open to suggestions. I'm sure we'll figure it out before too long.

I'm guessing that the opendap server is not happy at us throwing a bunch of simultaneous huge requests at it. That's a place where having limits on task concurrency would really help.

Yes this makes sense, given that we aren't seeing the I/O error when running this in serial (I tried it in the sandbox and it seemed fine).
@rabernat, any specific ideas on where/how to limit concurrency? I can see if Prefect will allow us to configure this for the flow.

Are you hosting your own Prefect server or using cloud? For cloud it's based on tags on tasks, and a matching server side limit.
I think I remember that they weren't going to bring concurrency limiting to their own server, which was one of the things that had me looking closer at Dagster for my own services.

For cloud it's based on tags on tasks, and a matching server side limit.

Alex, this looks perfect! Only I am not seeing in the docs anything about a matching server side limit. (And yes, we're on Prefect Cloud.) The section on Execution Behavior seems to suggest that if we set the tags correctly, then we should be good to go

Task tag limits are checked whenever a task run attempts to enter a Running state in Prefect Cloud. If there are no concurrency slots available for any one of your Task's tags, the Task will instead enter a Queued state. The same Python process that is attempting running your Task will then attempt to re-enter a Running state every 30 seconds (this value is configurable via config.cloud.queue_interval in Prefect Configuration). Additionally, if that process ever fails, Prefect Cloud will create a new runner every 10 minutes, which will then attempt to rerun your task on the specified queue interval. This process will repeat until all requested concurrency slots become available.

I found the placeholder for concurrency, but before getting there, I'm not sure where annotations can be defined to get passed down that far in the first place.
https://github.com/pangeo-forge/pangeo-forge-recipes/blob/a515458d6e42033174e367b9e68075d639b9a29a/pangeo_forge_recipes/executors/prefect.py#L8-L17
Maybe a spot in the meta.yaml schema for executor specific configurations? 'Prefect: tag all tasks with hycom'
On that page it has the server side limit per tag.

On that page it has the server side limit per tag.

Gotcha.

I found the placeholder for concurrency, but before getting there, I'm not sure where annotations can be defined to get passed down that far in the first place.

Maybe this becomes an optional attribute of XarrayZarrRecipe (possibly inherited from BaseRecipe or via a mixin).
from pangeo_forge_recipes.executors import ConcurrencyLimit  # this object doesn't exist yet

recipe = XarrayZarrRecipe(
    ...,
    concurrency_limits=[ConcurrencyLimit(stage_name="store_chunk", limit=10), ...],
)

Then, when the stages are instantiated by the recipe compiler, anything in recipe.concurrency_limits gets assigned to the corresponding stage's .annotation attribute.

Maybe then a configuration like above gets translated into a Prefect tag of something like "max-concurrency-10". And we configure Prefect Cloud with a set of allowable concurrency tags, e.g. "max-concurrency-10", "max-concurrency-20" etc. We could either require ConcurrencyLimit.limit to be exactly one of those values, or just round it to the closest one in this block you found.

The problem is that it is a global limit on tag concurrency, not a per-flow limit.
So tasks across all flows that have a "max-concurrency-10" tag would all be in the same queue, which probably would not make sense based on whatever their limit is (though HYCOM does say a 10 connection limit to their FTP server but doesn't specify for other services, but I did find they have fail2ban running...).
What about including a key in the ConcurrencyLimit, so:

recipe = XarrayZarrRecipe(
    ...,
    concurrency_limits=[ConcurrencyLimit(stage_name="store_chunk", limit=10, key='hycom'), ...],
)

Then during compilation that becomes a concurrency-hycom-10 tag, and the bakery can add it the limit when the flow gets registered.
from prefect import Client

client = Client()

# set a concurrency limit of 10 on the 'database' tag
client.update_task_tag_limit("concurrency-hycom-10", 10)

I found the placeholder for concurrency, but before getting there, I'm not sure where annotations can be defined to get passed down that far in the first place.

Correct. Concurrency limits are not currently supported, but they are on the todo list (see #45). How this is implemented is something we will have to discuss as part of the broader refactor that is coming soon. In the meantime I think we will have to put this recipe on pause for a few weeks.
Fetching so much data over opendap is going to be painful no matter what. We will also need to improve our story around retrying failed tasks.
Sorry @abkfenris - you picked a hard recipe to start on! 😆

I knew it might be a bit of a long shot right now when I got started, especially with the issues that I've encountered with this dataset before.
So even if it's unlikely that we can really do much more now, I tried changing tracks back to debugging the http based implementation while its fresh in my mind, and found a few more issues with that.
First I tried sniffing the traffic with Charles Proxy, Proxyman, and VCRpy, and none of them could capture traffic when xarray tried to open the http://data.hycom.org/... urls directly, but I was able to capture traffic with VCRpy when I used fsspec. I still got the 403 Forbidden from that, but at least I had some evidence.
import xarray as xr
import fsspec
import vcr

url = "http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc"

with vcr.use_cassette("fsspec-h5netcdf.yaml"):
    with fsspec.open(url) as f:
        ds = xr.open_dataset(f, drop_variables=["tau"], engine="h5netcdf")
        print(ds)
        print(ds.info())

➜ python test.py
Traceback (most recent call last):
  File "/Users/akerney/GMRI/staged-recipes/recipes/hycom-glby008-expt-93/test.py", line 10, in <module>
    ds = xr.open_dataset(f, drop_variables=["tau"], engine="h5netcdf")
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/xarray/backends/api.py", line 495, in open_dataset
    backend_ds = backend.open_dataset(
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 387, in open_dataset
    store = H5NetCDFStore.open(
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/xarray/backends/h5netcdf_.py", line 155, in open
    magic_number = read_magic_number_from_file(filename)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/xarray/core/utils.py", line 653, in read_magic_number_from_file
    magic_number = filename_or_obj.read(count)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/implementations/http.py", line 566, in read
    return super().read(length)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/spec.py", line 1544, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/caching.py", line 377, in _fetch
    self.cache = self.fetcher(start, bend)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/fsspec/implementations/http.py", line 605, in async_fetch_range
    r.raise_for_status()
  File "/usr/local/miniconda3/envs/pangeo-forge/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc')

VCR capture

interactions:
- request:
    body: null
    headers:
      Accept-Encoding:
      - identity
    method: HEAD
    uri: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
  response:
    body:
      string: ''
    headers:
      Accept-Ranges:
      - bytes
      Content-Length:
      - '38330744'
      Content-Type:
      - application/x-netcdf
      Date:
      - Fri, 04 Mar 2022 14:28:19 GMT
      Last-Modified:
      - Wed, 05 Dec 2018 03:42:43 GMT
      Server:
      - Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
      X-Content-Type-Options:
      - nosniff
      - nosniff
      X-Frame-Options:
      - SAMEORIGIN
      - SAMEORIGIN
      X-XSS-Protection:
      - 1;
      - 1;
    status:
      code: 200
      message: OK
    url: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
- request:
    body: null
    headers:
      Range:
      - bytes=0-5242887
    method: GET
    uri: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
  response:
    body:
      string: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

        <html><head>

        <title>403 Forbidden</title>

        </head><body>

        <h1>Forbidden</h1>

        <p>You don''t have permission to access /datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc

        on this server.</p>

        </body></html>

        '
    headers:
      Content-Length:
      - '287'
      Content-Type:
      - text/html; charset=iso-8859-1
      Date:
      - Fri, 04 Mar 2022 14:28:19 GMT
      Server:
      - Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
      X-Content-Type-Options:
      - nosniff
      X-Frame-Options:
      - SAMEORIGIN
      X-XSS-Protection:
      - 1;
    status:
      code: 403
      message: Forbidden
    url: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
version: 1

Looking at the capture, I'm guessing there is either an issue with the (lack of) headers, or the request size range. I tried setting block_size=10000 in fsspec to limit things, but got the same error.
To get at headers, I sniffed curl's request.
➜ curl http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc --output hycom_test.nc --proxy <IP_ADDRESS>:9090
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 36.5M  100 36.5M    0     0  10.1M      0  0:00:03  0:00:03 --:--:-- 10.1M

GET /datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc HTTP/1.1
Host: data.hycom.org
User-Agent: curl/7.77.0
Accept: */*
Connection: Keep-Alive

Though even passing all those header kwargs through didn't work.
import xarray as xr
import fsspec
import vcr

url = "http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc"

with vcr.use_cassette("fsspec-h5netcdf-client-kwargs-all.yaml"):
    with fsspec.open(
        url,
        client_kwargs={
            "headers": {
                "User-Agent": "curl/7.77.0",
                "Accept": "*/*",
                "Host": "data.hycom.org",
                "Connection": "Keep-Alive",
            }
        },
    ) as f:
        ds = xr.open_dataset(f, drop_variables=["tau"], engine="h5netcdf")
        print(ds)
        print(ds.info())

VCR Capture

interactions:
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - identity
      Connection:
      - Keep-Alive
      Host:
      - data.hycom.org
      User-Agent:
      - curl/7.77.0
    method: HEAD
    uri: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
  response:
    body:
      string: ''
    headers:
      Accept-Ranges:
      - bytes
      Connection:
      - Keep-Alive
      Content-Length:
      - '38330744'
      Content-Type:
      - application/x-netcdf
      Date:
      - Fri, 04 Mar 2022 15:22:39 GMT
      Keep-Alive:
      - timeout=5, max=100
      Last-Modified:
      - Wed, 05 Dec 2018 03:42:43 GMT
      Server:
      - Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
      X-Content-Type-Options:
      - nosniff
      - nosniff
      X-Frame-Options:
      - SAMEORIGIN
      - SAMEORIGIN
      X-XSS-Protection:
      - 1;
      - 1;
    status:
      code: 200
      message: OK
    url: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Connection:
      - Keep-Alive
      Host:
      - data.hycom.org
      Range:
      - bytes=0-5242887
      User-Agent:
      - curl/7.77.0
    method: GET
    uri: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
  response:
    body:
      string: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

        <html><head>

        <title>403 Forbidden</title>

        </head><body>

        <h1>Forbidden</h1>

        <p>You don''t have permission to access /datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc

        on this server.</p>

        </body></html>

        '
    headers:
      Connection:
      - Keep-Alive
      Content-Length:
      - '287'
      Content-Type:
      - text/html; charset=iso-8859-1
      Date:
      - Fri, 04 Mar 2022 15:22:39 GMT
      Keep-Alive:
      - timeout=5, max=99
      Server:
      - Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_qos/11.70
      X-Content-Type-Options:
      - nosniff
      X-Frame-Options:
      - SAMEORIGIN
      X-XSS-Protection:
      - 1;
    status:
      code: 403
      message: Forbidden
    url: http://data.hycom.org/datasets/GLBy0.08/expt_93.0/data/hindcasts/2018/hycom_glby_930_2018120412_t000_ssh.nc
version: 1

So 403 with all the headers cloned. 🤷
During this digging I found two other issues.

These are NetCDF 3s
Blindly enumerating over files may trigger fail2ban when trying missing files. It might take some creative code to explore the server, and what files are currently available and filter down dates appropriately.

I believe @rsignell-usgs has had some communication with HYCOM folks about getting their data more accessible. Maybe it time to explore those options further?


OAK-10334: Node.addMixin() may overwrite existing mixins

Test reproducing issue with jcr:mixinTypes

It looks like NodeDelegate.removeMixin has some check to see if the to-be-removed mixin is actually part of the current mixin list - and that check is IIUC done with the actual session permission. So in the write-but-no-read-permission case that would probably fail there.

That's correct.

But if addMixin now becomes more tolerant, then wouldn't removeMixin also need to be more tolerant?

Yes, I agree, but I rather handle this in a separate issue. The impact is a bit different. Removing a mixin would fail and not result in an unintentional change like in the addMixin() case.

Follow up issue regarding Node.removeMixin(String): OAK-10425

All tests are successful, except DocumentStoreIndexerIT. This is a known issue: https://issues.apache.org/jira/browse/OAK-10413


Split up into individually-importable components

Originally done as part of the add-tests branch, publishing it will make diffs of that branch easier.

hold off on this

OK this is good to go now.


feat(eth): Implement alternative polling mode

Add the option to use a polling strategy instead of a interrupt pin for the SPI EthDriver.
The old api was left intact, while a new EthDriver::new_spi_with_event_source was added.

Not 100% sold on the big cfg include/exclude statement and happy over any suggestions.

Not 100% sold on the big cfg include/exclude statement and happy over any suggestions.

Yeah i also don't like it, though that's what we get when upstream introduces new stuff on patch level releases :dagger:
Long term we may need to come up with some macro magic or some other alternative. Only having NAND style logic with "any" and "not" is a bit limiting ;D
I think the rest itself looks fine. With what board / phy chip did you test this stuff here?

@ivmarkov anything you don't like here otherwise i will merge it?

Give me a day or so to take a look...

Yeah maybe we can get away with not introducing the SpiEventSource struct, as its more or less just used as a glorified config struct with an exlusive option either poll_period or with pin. We could maybe model the internal init_spi to take an Option and than that way, we can use either the new spi_with_event_source constructor just taking a poll_period_ms and it its case setting the event_source pin as a None value.
So still having two different constructors, so we don't break current usage and only changing the internal init_spi. Or is there a special foreseeable usage for an SpiEventSource struct the way it is currently modelt @stabl-gjn ?

The only nit I have is that I don't think EventSource should be a struct. It looks to me more of a candidate for an enum.
But that's not the end of the world, so I'll merge as-is.
Thanks!

Yeah i added the extra struct/enum because i though it would make sense to unify the constructors in the future (probably next breaking change), instead of having two options (just thought that would be nicer instead of having 3-4 constructors).
I also first tought about making it an enum, but to not need to make it generic over the input pin (and having to specify that type when creating a polling config because it can not be inferred) and not wanting to publicly expose the raw gpio pin number as variable ( EventSource::Interrupt(gpio_int)).
So i went with the struct instead to hide the implementation details but i am always open for suggestions :)
Thanks for managing this so quickly

This branch broke the master branch for me (I do not have ethernet enabled).
I included a tentative bugfix here: https://github.com/esp-rs/esp-idf-svc/pull/457


[SUGGESTION] Hide Inactive

Please add the option not to show inactive effects when they have a duration.

Would that not make it inconvenient to re-enable them? I'm wondering what your use case is for this.

I have many active effects that are activated with the use of an item, upon receiving a condition, etc.
Very few require the players to activate them manually, so their screen will be cleaner if they don't see inactive effects.
The main reason i'm using this module (and maybe more people too) is so that players don't forget to deactivate effects they have active for awhile and not to activate effects.


Refactor document component

Review app deployed to https://teaching-vacancies-review-pr-3124.london.cloudapps.digital


Postgrex is erroring on throttle/buggy rate limiter

This may have to do with Process.sleep issues.
[info] Postgrex.Protocol (#PID<0.570.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.908.0> exited

I might be one to solve this, I need to do so in a few days, unless someone can solve it...

Hi

I may just remove the delays. They apparently cause a problem with Cowboy.

Hi

Hello.

With regards to this I have added proper debouncing to the rate limiters a long time ago.


[SPARK-12469][CORE][RFC/WIP] Add Consistent Accumulators for Spark

This is an initial PR illustrating one of the possible approaches for providing consistent accumulators in Spark.
Tasks executed on Spark workers are unable to modify values from the driver, and accumulators are the one exception for this. Accumulators in Spark are implemented in such a way that when a stage is recomputed (say for cache eviction) the accumulator will be updated a second time. This makes accumulators inside of transformations more difficult to use for things like counting invalid records (one of the primary potential use cases of collecting side information during a transformation). However in some cases this counting during re-evaluation is exactly the behaviour we want (say in tracking total execution time for a particular function). Spark would benefit from a version of accumulators which did not double count even if stages were re-executed.
Motivating example:
val parseTime = sc.accumulator(0L)
val parseFailures = sc.accumulator(0L)
val parsedData = sc.textFile(...).flatMap { line =>
  val start = System.currentTimeMillis()
  val parsed = Try(parse(line))
  if (parsed.isFailure) parseFailures += 1
  parseTime += System.currentTimeMillis() - start
  parsed.toOption
}
parsedData.cache()

val resultA = parsedData.map(...).filter(...).count()

// some intervening code.  Almost anything could happen here -- some of parsedData may
// get kicked out of the cache, or an executor where data was cached might get lost

val resultB = parsedData.filter(...).map(...).flatMap(...).count()

// now we look at the accumulators

Here we would want parseFailures to only have been added to once for every line which failed to parse.  Unfortunately, the current Spark accumulator API doesn’t support the current parseFailures use case since if some data had been evicted its possible that it will be double counted.
See the full design document at https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing
cc @squito

Test build #49723 has started for PR 10841 at commit 3f96429.

Test build #49723 has finished for PR 10841 at commit 3f96429.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49723/
Test FAILed.

Test build #49802 has started for PR 10841 at commit 0915634.

Test build #49802 has finished for PR 10841 at commit 0915634.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49802/
Test FAILed.

Test build #49807 has started for PR 10841 at commit 99acdff.

Test build #49807 has finished for PR 10841 at commit 99acdff.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49807/
Test FAILed.

Merged build finished. Test FAILed.

Test build #49821 has started for PR 10841 at commit ce60621.

Test build #49821 has finished for PR 10841 at commit ce60621.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49821/
Test FAILed.

Build finished. Test FAILed.

In docs, you need to remove the following sentence

For accumulator updates performed inside actions only, Spark guarantees that each task’s update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task’s update may be applied more than once if tasks or job stages are re-executed.

That's a good point @codingcat , although the old behavior for regular accumulators is the same I'll add a note about the new API option.

Test build #49849 has started for PR 10841 at commit 5ff6b6a.

Test build #49849 has finished for PR 10841 at commit 5ff6b6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49849/
Test FAILed.

Merged build finished. Test FAILed.

Test build #49870 has started for PR 10841 at commit cdfd0be.

Test build #49873 has started for PR 10841 at commit f4100bc.

Test build #49870 has finished for PR 10841 at commit cdfd0be.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49870/
Test FAILed.

Test build #49873 has finished for PR 10841 at commit f4100bc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49873/
Test PASSed.

ping @andrewor14 if you have some time to look at this :)

Test build #50010 has started for PR 10841 at commit 495c8b6.

Test build #50010 has finished for PR 10841 at commit 495c8b6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50010/
Test FAILed.

seems unrelated, jenkins retest this please.

Test build #50035 has started for PR 10841 at commit 495c8b6.

Test build #50035 has finished for PR 10841 at commit 495c8b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50035/
Test FAILed.

Test build #50278 has started for PR 10841 at commit 523ba43.

Test build #50278 has finished for PR 10841 at commit 523ba43.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50278/
Test FAILed.

Test build #50284 has started for PR 10841 at commit 26db2c6.

Test build #50284 has finished for PR 10841 at commit 26db2c6.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50284/
Test FAILed.

Test build #50285 has started for PR 10841 at commit 3e014c7.

Test build #50291 has started for PR 10841 at commit 5524bc7.

Test build #50285 has finished for PR 10841 at commit 3e014c7.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50285/
Test FAILed.

Test build #50295 has started for PR 10841 at commit dabe8cf.

Test build #50291 has finished for PR 10841 at commit 5524bc7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50291/
Test FAILed.

Merged build finished. Test FAILed.

Test build #50295 has finished for PR 10841 at commit dabe8cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50295/
Test PASSed.

Merged build finished. Test PASSed.

I've updated this against master with @andrewor14's Accumulator changes. cc @squito

Test build #50309 has started for PR 10841 at commit 59c211a.

Test build #50309 has finished for PR 10841 at commit 59c211a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50309/
Test PASSed.

Merged build finished. Test PASSed.

@holdenk Thanks for spending the time to put forth the detailed design. I agree 100% with the problem and have thought a great deal about how to fix it myself. However, I personally found the proposed API kind of clunky. It's not super clear what "consistent" means, especially to someone new to Spark. I think it would be cleaner to just implement this in the existing classes themselves and maybe expose only a flag.
It seems that on a high level we want to associate an accumulated value with a particular computation of an RDD. Can we store something like a map from RDD ID to a list of accumulated values in each accumulator? Then if the user calls acc.value we just sum all of them up. If the user wants an accumulated value associated with a particular RDD or even a particular computation then they can also do that. E.g.
class AccumulableInfo[R, T](...) {
  private val valuesByRDD = mutable.HashMap[Int, ArrayBuffer[R]]
  def value: R = valuesByRDD.values.reduce(mergeThemAll)
  def valueByRDD(rddId: Int): R = valuesByRDD(rddId).reduce(mergeThemAll)
}

Even this may solve only part of the problem, where the same accumulator used across different RDDs, but maybe it's a better first step. I may be missing something but I don't see the need to introduce all these new classes; the existing set is confusing enough and in the past we've made the mistake of exposing all of them to the user.
Also, I believe @rxin has some thoughts on this.

By the way, there's a section on the design doc attached to https://issues.apache.org/jira/browse/SPARK-10620 that describes the same problem by classifying accumulators into different use cases. I think it's simpler to think about this problem in terms of these use cases.

Thanks for taking a look @andrewor14 :) This PR (and the JIRA & design doc) are all very much focused on enabling the data property use case for accumulators.
I introduced the GenericAccumulable as a parent to Accumulable to try and maintain source compatibility between versions and also limit any cost to users of "regular" accumulators - and since we are coming up on Spark 2.0 that might not be worth the overhead so I'm more than happy to simplify that part away. Or if we change the current accumulator base class in such a way it meets the needs while keeping source compatibility all the better.
For the naming of the Consistent Accumulators (or naming of the flag) totally open to ideas - I just didn't have anything else come to mind.
So I'm assuming the clunky part of the API your referring to is the "withAccumulator" part of the transformations - which I do feel is pretty clunky myself. My initial draft attempted to avoid using this (I first tried Implementation Option 1 from the design doc ( https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing ) but when multiple transformations are chained together the TaskContext ends up having the RDD id of the inner most RDD. I'd really love to avoid explicitly having the user explicitly pass this information in and if there is a permanent way to do that I'm happy to change this around to give that a shot.
I like the idea of the map of RDD id to accumulated value, since an entire RDD might not be computed in the same task though I think we will need to keep track of the value by RDD ID & Partition ID (or at least keep track of value and bitmask of partitions accumulated per RDD ID).
cc @squito who did the initial accumulator work and we've gone over the design doc some together.

Test build #50607 has started for PR 10841 at commit a507f74.

Test build #50607 has finished for PR 10841 at commit a507f74.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50607/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50619/
Test FAILed.

Merged build finished. Test FAILed.

looks like an internal jenkins issue, jenkins retest this please.

Test build #50622 has started for PR 10841 at commit 69c5c94.

Test build #50622 has finished for PR 10841 at commit 69c5c94.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50622/
Test FAILed.

Merged build finished. Test FAILed.

looks like an internal jenkins issue, jenkins retest this please.

Test build #50654 has started for PR 10841 at commit 69c5c94.

Test build #50654 has finished for PR 10841 at commit 69c5c94.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50654/
Test FAILed.

Merged build finished. Test FAILed.

jenkins retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50756/
Test FAILed.

Test build #50811 has started for PR 10841 at commit 133a358.

Test build #50811 has finished for PR 10841 at commit 133a358.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50811/
Test PASSed.

@andrewor14 I changed it around a bit so we don't need to have the user explicitly pass the rdd/partition information to the consistent accumulator. It's still got the separate class since we need to keep track of some extra information for consistent accumulators. Think this is closer to the correct direction?

Test build #50842 has started for PR 10841 at commit 91c4f0e.

Test build #50842 has finished for PR 10841 at commit 91c4f0e.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50842/
Test FAILed.

Test build #50868 has started for PR 10841 at commit 5a7bd6f.

Test build #50868 has finished for PR 10841 at commit 5a7bd6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50868/
Test FAILed.

I'm going to close this and re-open one with the "full auto" type approach suggested.


KeyError::

Hello, when i am trying to processes this git, i get the following error
Keyerror: '\x1f'
i am sharing the screen shot.

I am closing this due to inactivity


Various issue fixes

Resolves #4015
Resolves #4013
Resolves #3859
Resolves #3791
Resolves #4039
Resolves #4038
Resolves #4037
Resolves #4029

Will merge just before next snapshot.


Downloading results failed for 60bd3746f83fe5520bad31c3

https://run.biosimulations.org/simulations/60bd3746f83fe5520bad31c3
{"error":[{"status":"500","title":"Internal Server Error","meta":{"time":1623184041076,"url":"/results/60bd3746f83fe5520bad31c3/download"}}]}

This is caused by the file no being available. 2 things to do

Figure out why the file is not uploaded
Catch the error and return a 404


Normalize naming of path related types and functions

Most important changes are:

document.ValuePath renamed to document.Path
expr.FieldSelector renamed to expr.Path

The rest is essentially renaming constructors, tests, etc.

Codecov Report

Merging #294 into master will increase coverage by 0.03%.
The diff coverage is 83.78%.

@@            Coverage Diff             @@
##           master     #294      +/-   ##
==========================================
+ Coverage   61.75%   61.79%   +0.03%
==========================================
  Files          71       71
  Lines        6587     6586       -1
==========================================
+ Hits         4068     4070       +2
+ Misses       1997     1994       -3
  Partials      522      522

Impacted Files
Coverage Δ

database/table.go
56.70% <ø> (ø)

document/document.go
75.93% <ø> (ø)

sql/parser/update.go
87.30% <ø> (ø)

sql/planner/sort.go
15.27% <0.00%> (ø)

sql/planner/tree.go
37.57% <ø> (ø)

sql/query/create.go
64.28% <ø> (ø)

sql/query/expr/path.go
41.17% <40.00%> (+2.28%)
:arrow_up:

sql/parser/parser.go
83.50% <75.00%> (ø)

database/config.go
67.85% <91.66%> (ø)

sql/parser/expr.go
75.79% <100.00%> (ø)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 514564f...6b40a9e. Read the comment docs.


Input Panel(s) for Custom Commands

So following on from the brilliant new Custom Command feature, if you think it's feasible and/or have the time/desire to do so, it would be great to be able to define inputs into custom commands.
 In my fork, I've add issues functionality that allows you to do commands like git issues new -t "Issue Title" -b "Issue body"
As it stands, there (appears to be) no way to provide dynamic input to the Custom Commands feature, that would allow me to provide strings to the -t flag, -b flag (or any others).
Possible Solution
At the moment, the Custom Command syntax is as follows:
    {
        "caption": "Git: <Command Name for Palette>",
        "command": "git_custom",
        "args": {
            "cmd": "<custom command>",
            "output": <"panel"/"view"/null>,
            "async": <true/false>
        }
    }

My thoughts would be to add an extra optional array field to args called inputs (assuming this is doable - I've not proved it out yet)
The new syntax would be:
    {
        "caption": "Git: <Command Name for Palette>",
        "command": "git_custom",
        "args": {
            "cmd": "<custom command>",
            "inputs": [
                {"flag": "<custom flag e.g. -t>", "type": "<raw/quoted>"},
                {"flag": "<custom flag e.g. -b>", "type": "<raw/quoted>"}
            ],
            "output": <"panel"/"view"/null>,
            "async": <true/false>
        }
    }

So you'd get a list of inputs, each of which defines a flag, and the type of info to be collected. That is to say - when type is raw, you pass the collected input straight through without modification, and when type is quoted, you put the collected user input into quotes.
In terms of collecting the input - you would pop an input panel with the flag as the caption. If you were really keen, you could add say an optional help field to the inputs objects, that allows you to provide extra info on the flag, so the caption can include not just the flag, but info on what the flag expects as input.
So to illustrate, my new custom command would look like this:
    {
        "caption": "Git: New Issue",
        "command": "git_custom",
        "args": {
            "cmd": "issues new",
            "inputs": [
                {"flag": "-t", "type": "quoted"},
                {"flag": "-b", "type": "quoted"}
            ],
            "output": "panel",
            "async": true
        }
    }

Upon selecting that command from the palette, you would be presented with an input panel with the caption -t and would enter the new github issue's title. When you pressed enter, you would then be presented with an input panel with the caption -b and would enter the new github issue's body.
That would bring you to the end of the inputs, so you would pass out the following to the git executable:
issues new -t "Title I typed" -b "Body I typed"
One possible further extension to that could be doing something like adding a field that lets you choose whether your input should provide an input panel or a view for entry (like with the commit message), but I'd be perfectly happy even without that :)

+1 for feature that enables passing the current buffer, and selected line number to custom command, so I can do something like...
{
    "caption": "Git: Log Selection",
    "command": "git_custom",
    "args": {
        "cmd": "log -L<selected-from-line>,<selected-to-line>:<filename> --oneline",
        "output": "view",
        "async": false
    }
}

With SublimeGit Diff syntax highlight of course, and then track change history like a boss!

+1


Tempel-CLI Update

Update tempel-cli-py for additional modules

argparse - for parsing argument
pyperclip - for clipboard usage

pyperclip module included

@nsiregar  Emm, please keep the filename stay as tempel, without extension.

done rename tempel.py to tempel. sorry just habit using file extensions.

@tuanpembual @nsiregar LGTM

:+1: :-1:

@nsiregar  can you please add comment each step?
just like old script. :)
It will help new contributor.
And @herpiko what should lang we use for help;etc (in code)?
It must consistent, Bahasa or English.
I prefer using Bahasa.

i've updated some docstrings in tempel


Move all project configuration to pyproject.oml

"One file to rule them all"
Everything is now in pyproject.toml

Looking good.
Should the install test workflow you added try to import the library after installation?


Enable all images on Container Service automatically

Short Description
Have to manually enable all images, is there a way to do this automatically in container_service_images role?
https://groups.google.com/g/xnat_discussion/c/Xcwp_bWcHf8/m/okBgAmDPBwAJ
Evidence/Steps to Reproduce
No response
Acceptance Criteria/Expected Behaviour
No response
Details
No response
Resolution
No response

Me and @p-j-smith spent some time trying to figure this out, but couldn't find a quick solution so prioritised other areas


x/pkgsite: consider surfacing fork status and activity details vs upstream

While in some cases a fork may actually be the better, more maintained, more adopted version, when the parent does have more activity than the fork, it would probably be helpful to show the parent in the Links or elsewhere in the header section of the pkg page
Somewhat related also would be to rank it lower in search results
Example:
https://pkg.go.dev/search?q=fortio&m=
shows
https://pkg.go.dev/github.com/lemonlinger/fortio
and
https://pkg.go.dev/github.com/CrowdStrike/fortio
and
https://pkg.go.dev/github.com/MUzairS15/fortio
all stale forks with minor to no updates and not the original
https://pkg.go.dev/fortio.org/fortio
which isn't at all on the list (only sub packages are somehow)
Screenshot

What did you expect to see?

The tricky part is the source of the information is proxy.golang.org, which doesn't have sufficient info to tell which is a fork.
In theory, the pkgsite may try to check the source repository (heuristically) and query hosting sites (if they offer apis). But I wonder if it's better to investigate we could get better signals from proxy to improve search ranking instead.

Thanks for having looked! I think it may be related tangentially related to #36952 as that one would also need to pull meta information for github and not just the proxy (or the proxy could as it's pulling from github in the end?)


Reference Mpv.NET for prerequisites and licensing

I have updated the prerequisites and licensing section in Mpv.NET. To reduce the need of updating these 2 sections in your project, could you please reference/link to the Mpv.NET project instead?

done


docs: rename alter materialized view to update

Alter implies a DDL sql ALTER MATERIALIZED VIEW statement, the code sample is purely an API
update example.

Thank you!


[HUDI-3625] [RFC-60] Optimized storage layout for Cloud Object Stores

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
RFC for supporting a new storage layout that is optimized for cloud object stores like Amazon S3.
Verify this pull request
This pull request is just an RFC with no code changes that need verification.
Committer checklist

[x] Has a corresponding JIRA in PR title & commit

[x] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Created this gist to track existing usages of FS based listing


Lose connection after allowing access

Expected Behavior
Hi, I want to use the code provided here to generate a pickle file.
Actual Behavior
after I follow all the steps and hit the final "allow" button, the local host I am routed to fails, and the pickle is never produced
I am losing connection after I hit allow.
Steps to Reproduce the Problem

Specifications

Python version (python --version)
OS (Mac/Linux/Windows)

I ran into the same problem........


Use let whenever possible and a few other tweaks

Dunno how I overlooked that ignoreNil :) Thanks for cleaning up!

No worries : ) Thanks for releasing the project, I've found it to be very useful. RAC is a bit bewildering to start with, so it's great to have a working project to experiment with.

Glad you like it, I'm hoping to be able to put some more example stuff in their soon!


corelib/collection_length_test dart2js generated output takes a long time in checked mode on IE

This is regularly timing out, e.g.,
http://build.chromium.org/p/client.dart/builders/dart2js-ie9-win7-all-5-6/builds/4497/steps/dart2js tests --checked failures/logs/stdio
I will mark it in the status file

Checked mode is a goner


Optionally scale egress bandwidth with CPU.

Add support to isolators/port_mapping for optionally scaling egress bandwidth with CPU and with minimum and maximum limits.

Let's squash the extra follow-on commits into 1, I can do it prior to merging but it's best if you get used to keeping your commits organized on your end.